一個filebeat實例 設置多topic設置

 

方法1：一實例多topic：

https://discuss.elastic.co/t/filebeat-5-0-output-to-kafka-multiple-topics/67934

 

The document_type per prospector becomes the event field type. That's why the filter won't match.

Instead of conditionals consider using the format string like:


filebeat.prospectors:
- ...
  document_type: myapp_applog
- ...
  document_type: myapp_applog_stats
- ...
  document_type: myapp_elblog

output.kafka:
  topic: '%{[type]}' # use document_type to set topic


btw. the topic field in conditionals also supports format strings.

 

 

cat filebeat.yml

output.kafka:
  enabled: true
  hosts: ["192.168.111.107:9092","192.168.111.108:9092","192.168.111.109:9092"]
  topic: '%{[type]}'


filebeat.prospectors:
#------------------------------ Log prospector --------------------------------
- paths:
    - /data/logs/financial-manager/server0.log
  fields:
    app_id: financial-manager
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}\s(20|21|22|23|[0-1]\d):[0-5]\d:[0-5]\d\.'
  multiline.negate: true
  multiline.match: after
  document_type: log-log4j

- paths:
    - /data/logs/user-service/*.log
  fields:
    app_id: user-service
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}\s(20|21|22|23|[0-1]\d):[0-5]\d:[0-5]\d\.'
  multiline.negate: true
  multiline.match: after
  document_type: log-iis

- paths:
    - /data/logs/financial-manager/access_log.log
  fields:
    app_id: financial-manager
  document_type: log-undertow

 

 

方法二：多filebeat實例，每個filebeat實例設置一個topic

運行多個filebeat實例，每個實例對應一個配置文件

cat filebeat.yml

output.kafka:
  enabled: true
  hosts: ["192.168.111.107:9092","192.168.111.108:9092","192.168.111.109:9092"]
  topic: log-log4j


filebeat.prospectors:
#------------------------------ Log prospector --------------------------------
- paths:
    - /data/logs/subscribe-consumer/server0.log
  fields:
    app_id: subscribe-consumer 

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}\s(20|21|22|23|[0-1]\d):[0-5]\d:[0-5]\d\.'
  multiline.negate: true
  multiline.match: after

 

cat filebeat2.yml

output.kafka:
  enabled: true
  hosts: ["192.168.111.107:9092","192.168.111.108:9092","192.168.111.109:9092"]
  topic: log-tomcat


filebeat.prospectors:
#------------------------------ Log prospector --------------------------------
- paths:
    - /data/logs/subscribe-consumer/server0.log
  fields:
    app_id: subscribe-consumer 

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}\s(20|21|22|23|[0-1]\d):[0-5]\d:[0-5]\d\.'
  multiline.negate: true
  multiline.match: after