Fluentd直接傳輸日志給kafka

官方文檔地址：https://docs.fluentd.org/output/kafka

td-agent版本自帶包含out_kafka2插件，不用再安裝了，可以直接使用。

若是使用的是Fluentd，則需要安裝這個插件：

$ fluent-gem install fluent-plugin-kafka

配置示例

<match pattern>
  @type kafka2

  # list of seed brokers
  brokers <broker1_host>:<broker1_port>,<broker2_host>:<broker2_port>
  use_event_time true

  # buffer settings
  <buffer topic>
    @type file
    path /var/log/td-agent/buffer/td
    flush_interval 3s
  </buffer>

  # data type settings
  <format>
    @type json
  </format>

  # topic settings
  topic_key topic
  default_topic messages

  # producer settings
  required_acks -1
  compression_codec gzip
</match>

參數說明

• @type：必填，kafka2
• brokers：kafka連接地址，默認是localhost:9092
• topic_key：目標主題的字段名,默認是topic，必須設置buffer chunk key，示例如下：

topic_key category
<buffer category> # topic_key should be included in buffer chunk key
  # ...
</buffer>

• default_topic：要寫入目標的topic，默認nil，topic_key未設置的話則使用這個參數
• 指令，可用參數有json, ltsv和其他格式化程序插件，用法如下：

<format>
  @type json
</format>

• use_event_time：fluentd事件發送到kafka的時間，默認false，也就是當前時間
• required_acks：每個請求所需的ACK數，默認-1
• compression_codec: 生產者用來壓縮消息的編解碼器，默認nil，可選參數有gzip, snappy(如果使用snappy,需要使用命令td-agent-gem安裝snappy)
• @log_level：可選，日志等級，參數有fatal, error, warn, info, debug, trace

用法示例

kafka安裝參考：https://www.cnblogs.com/sanduzxcvbnm/p/13932933.html

<source>
  @type tail
  @id input_tail
  <parse>
    @type nginx
  </parse>
  path /usr/local/openresty/nginx/logs/host.access.log
  tag td.nginx.access
</source>

<match td.nginx.access>
  @type kafka2
  brokers 192.168.0.253:9092
  use_event_time true
  
  <buffer app>
    @type memory
  </buffer>
  
  <format>
    @type json
  </format>
  
  topic_key app
  default_topic messagesb # 注意，kafka中消費使用的是這個topic
  required_acks -1
  compression_codec gzip
</match>

kafka消費的數據顯示：

{"remote":"192.168.0.145","host":"-","user":"-","method":"GET","path":"/00000","code":"404","size":"131","referer":"-","agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:82.0) Gecko/20100101 Firefox/82.0","http_x_forwarded_for":"-"}
{"remote":"192.168.0.145","host":"-","user":"-","method":"GET","path":"/99999","code":"404","size":"131","referer":"-","agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:82.0) Gecko/20100101 Firefox/82.0","http_x_forwarded_for":"-"}