logstash收集nginx日志寫入kafka

實驗簡介:

    由logstash收集nginx日志寫入kafka中，在由另一台主機logstash讀取kafka日志寫入elasticsearch

一 logstash收集日志寫入kafka

1.1.1 編寫logstash配置文件

[root@localhost ~]# cat /etc/logstash/conf.d/nginx-kafka.conf
 input {                                             
       file {
           path => "/opt/vhosts/fatai/logs/access_json.log"
           start_position => "beginning"
           type => "nginx-accesslog"
           codec => "json"
           stat_interval => "2"
           }
}
output {

    kafka {
         bootstrap_servers => "192.168.10.10:9092"
         topic_id => 'nginx-access-kafkaceshi'
         codec => "json"
        }

}

1.1.2 驗證並重啟logstash

[root@localhost ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-kafka.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@localhost ~]# systemctl restart logstash.service 

1.1.3 kafka端驗證主題

[root@DNS-Server tools]# /tools/kafka/bin/kafka-topics.sh --list  --zookeeper 192.168.10.10:2181,192.168.10.167:2181,192.168.10.171:2181
nginx-access-kafkaceshi

二 logstash收集kafka日志並寫入elk

1.1.1 編寫logstash配置文件

[root@Docker ~]# cat /etc/logstash/conf.d/nginx_kafka.conf
input {
    kafka {
      bootstrap_servers => "192.168.10.10:9092"   #kafka地址
      topics => "nginx-access-kafkaceshi"         #定義主題
      group_id => "nginx-access-kafkaceshi"       #自定義
      codec => "json"                             #指定編碼
      consumer_threads => 1                       #消費者線程
      decorate_events => true                     #要不要加kafka標記
    }
}
output {
  if [type] == "nginx-accesslog"{                 #type 是收集時候logstash定義的
    elasticsearch {
      hosts => ["192.168.10.10:9200"]
      index=> "nginx-accesslog-kafka-test-%{+YYYY.MM.dd}"
    }
  }
}

1.1.2 檢測並重啟

[root@Docker ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx_kafka.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@Docker ~]# systemctl restart logstash.service

1.1.3 elasticsearch驗證