filebeat收集nginx的json格式日志

一、在nginx主機上安裝filebeat組件

[root@zabbix_server nginx]# cd /usr/local/src/
[root@zabbix_server src]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-x86_64.rpm
[root@zabbix_server src]# yum localinstall filebeat-6.2.4-x86_64.rpm 

二、配置filebeat配置文件

[root@zabbix_server src]# vim /etc/filebeat/filebeat.yml 
filebeat.prospectors:
  - type: log
    enabled: true
    json.keys_under_root: true    #json格式收集
    json.overwrite_keys: true       #json格式收集
    paths:
      - /var/log/nginx/access.log  #需要收集的日志文件路徑
    fields:
      log_topics: nginx-172.28.18.75 #設置日志標題 
  
  output.logstash:
    hosts: ["172.28.18.69:10001"]  #輸出到logstash服務地址和端口

三、配置nginx.conf文件，設置json格式日志

[root@zabbix_server src]# vim /etc/nginx/nginx.conf 
    log_format access_json_log  '{"@timestamp":"$time_local",'
                                  '"http_host":"$http_host",'
                                  '"clinetip":"$remote_addr",'
                                  '"request":"$request",'
                                  '"status":"$status",'
                                  '"size":"$body_bytes_sent",'
                                  '"upstream_addr":"$upstream_addr",'
                                  '"upstream_status":"$upstream_status",'
                                  '"upstream_response_time":"$upstream_response_time",'
                                  '"request_time":"$request_time",'
                                  '"http_referer":"$http_referer",'
                                  '"http_user_agent":"$http_user_agent",'
                                  '"http_x_forwarded_for":"$http_x_forwarded_for"}';
                                      
    access_log  /var/log/nginx/access.log  access_json_log;

四、檢查配置文件語法，並重載

[root@zabbix_server src]# nginx -tc /etc/nginx/nginx.conf 
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@zabbix_server src]# nginx -s reload

五、查看access.log日志輸出

[root@zabbix_server src]# tail -f /var/log/nginx/access.log
{"@timestamp":"15/Jul/2019:10:33:23 +0800","http_host":"zabbix.9500.cn","clinetip":"219.239.8.14","request":"POST /zabbix.php?sid=311c76f6740bc2f7&action=widget.problems.view HTTP/1.1","status":"200","size":"33684","upstream_addr":"127.0.0.1:9000","upstream_status":"200","upstream_response_time":"1.103","request_time":"1.103","http_referer":"http://zabbix.9500.cn/zabbix.php?action=dashboard.view&ddreset=1","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0","http_x_forwarded_for":"-"}
{"@timestamp":"15/Jul/2019:10:33:23 +0800","http_host":"zabbix.9500.cn","clinetip":"219.239.8.14","request":"POST /zabbix.php?sid=9203532d28b920b3&action=widget.problems.view HTTP/1.1","status":"200","size":"33856","upstream_addr":"127.0.0.1:9000","upstream_status":"200","upstream_response_time":"0.759","request_time":"0.759","http_referer":"http://zabbix.9500.cn/zabbix.php?action=dashboard.view&ddreset=1","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36","http_x_forwarded_for":"-"}

已經變成json格式的日志輸出了。

六、在logstash服務器上的/etc/logstash/conf.d/目錄下，新建一個nginx.conf文件

用於監聽10001端口，負責收集filebeat傳遞過來的日志數據，並向elasticseatch服務發送日志數據

[root@server-1 conf.d]# vim /etc/logstash/conf.d/nginx.conf
input {
  beats {
   port=>10001   #監聽filebeat發送日志的端口
  }
}

output {
  if[fields][log_topics]=="nginx-172.28.18.75"{   #判斷是否是filebeat配置設置的fields字段，是則發送到elastcisearch
    elasticsearch {
     hosts=>["172.28.18.69:9200"]
     index=>"nginx-172.28.18.75-%{+YYYY.MM.dd}"   #設置索引名字
    } 
  }
}

七、重啟logstash服務，並查看端口

[root@server-1 conf.d]# systemctl restart logstash
[root@server-1 conf.d]# netstat -tunlp|grep 10001
tcp6       0      0 :::10001                :::*                    LISTEN      26599/java   

啟動成功

八、重啟nginx主機上的filebeat服務

[root@zabbix_server src]# systemctl restart filebeat

九、查看elasticsearch服務器上是否有新建的索引

[root@zabbix_server etc]# curl 172.28.18.69:9200/_cat/indices?v
health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana                       nQhtAX5YSSKzT1p0G-1XtA   1   0          4            0     25.2kb         25.2kb
yellow open   nginx-172.28.5.214-2019.07.15 tH8oVfZhQy-TdnIPHyFgvQ   5   1    2106272            0    668.8mb        668.8mb
yellow open   system-syslog-2019.07         REp7fM_gSaquo9PX2_sREQ   5   1   10772928            0      2.6gb          2.6gb

可以看到日志已經成功收集到elasticsearch服務器上了

十、配置kibana，展示日志數據

系統管理---索引模式--創建索引模式

發現菜單里已經可以看到數據了