一 Filebeat工作原理
Filebeat由兩個主要組件組成: prospector和 harvester
1.1 harvester
- 負責讀取單個文件的內容
- 如果文件在讀取時被制除或重命名， Filebeat將繼續讀取文件。
1.2 prospector
- prospector負責管理 harvester並找到所有要讀取的文件來源
- 如果輸入類型為日志，則查找器將查找路徑匹配的所有文件，並為每個文件啟動一個 harvester
- Filebeat目前支持兩種 prospector類型:log和 stdin
1.3 Filebeat如何保持文件的狀態
- Filebeat保存每個文件的狀
- 並經常將狀態新到磁盤上的注冊文件中
- 該狀態用於記住 harvester正在讀取的最后偏移量，並確保發送所有日志行。
- 如果輸出（例如 Elasticsearch或 Logstash）無法訪問， Filebeat會跟蹤最后發送的行，並在輸出再次可用時繼續讀取文件
- 在 Filebeat運行時，每個 prospect內存中也會保存的文件狀態信息，當重新啟動 Filebeat時，將使用注冊文件的數據來重建文件狀態， Filebeat將毎個 harvester在從保存的最后偏移量繼續讀取
- 文件狀態記錄在data/ registry文件中
1.4 啟動命令
./filebeat -e -c darren-log.yml ./filebeat -e -c darren-log.yml -d "publish" #參數說明 -e：輸出到標准輸出，默認輸出到syslog和logs下 -c：指定配置文件 -d：輸出debug信息 #測試 [root@node4 filebeat]# ./filebeat -e -c darren-log.yml -d "publish" { "_index": "filebeat-7.4.2-2019.11.24-000001", "_type": "_doc", "_id": "O4vOnG4BNbSd3xvSaBQk", "_version": 1, "_score": 1, "_source": { "@timestamp": "2019-11-24T09:46:40.787Z", "log": { "offset": 24, "file": { "path": "/opt/logs/a.log" } }, "message": "123", "tags": [ "web" , "test" ], "input": { "type": "log" }, "from": "test-web", "ecs": { "version": "1.1.0" }, "host": { "name": "node4" }, "agent": { "version": "7.4.2", "type": "filebeat", "ephemeral_id": "be331c63-1342-432b-b969-1c51955f184d", "hostname": "node4", "id": "2832793b-3bb6-4081-b05f-1955815440d0" } } }
二 把filebeat整合到nginx
[root@node4 ~]# cd /usr/local/filebeat/
[root@node4 filebeat]# vi nginx-log.yml
filebeat.inputs: - type: log enabled: true paths: - /usr/local/nginx/logs/*.log tags: ["nginx"] setup.template.settings: index.number_of_shards: 3 output.elasticsearch: hosts: ["192.168.132.131","192.168.132.132","192.168.132.133"] #output.console: # pretty: true # enable: true
2.1 刪除filebeat*的索引
2.2 啟用這個配置
[root@node4 filebeat]# ./filebeat -e -c nginx-log.yml
已經有數據寫入
2.3 看原始數據
{ "_index": "filebeat-7.4.2-2019.11.24-000001", "_type": "_doc", "_id": "QYvPnW4BNbSd3xvSqhRe", "_version": 1, "_score": 1, "_source": { "@timestamp": "2019-11-24T14:27:41.402Z", "host": { "name": "node4" }, "agent": { "type": "filebeat", "ephemeral_id": "622a2491-72d6-4c5a-936c-2d7b0d796d3b", "hostname": "node4", "id": "2832793b-3bb6-4081-b05f-1955815440d0", "version": "7.4.2" }, "log": { "offset": 421, "file": { "path": "/usr/local/nginx/logs/access.log" } }, "message": "192.168.132.1 - - [24/Nov/2019:03:15:12 -0500] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"", #請求的日志 "tags": [ "nginx" ], "input": { "type": "log" }, "ecs": { "version": "1.1.0" } } }
可以看到，在message中已經獲取到了你滾下日志，但是內容並沒有經過處理，遂於后期的日志處理不利，可以使用Module解決
2.4 Module配置
前面想要實現日志數據的讀取及處理都是使用手動配置的，在filebeat中，有大量的module，可以簡化我們的配置直接使用就可以
[root@node4 filebeat]# ./filebeat modules list
Enabled:
Disabled:
apache
auditd
aws
cef
cisco
coredns
elasticsearch
envoyproxy
googlecloud
haproxy
ibmmq
icinga
iis
iptables
kafka
kibana
logstash
mongodb
mssql
mysql
nats
netflow
nginx
osquery
panw
postgresql
rabbitmq
redis
santa
suricata
system
traefik
zeek
沒有啟用module，如果需要啟用需要進行enable操作
2.5 啟用nginx的module
[root@node4 filebeat]# ./filebeat modules enable nginx #啟用
[root@node4 filebeat]# ./filebeat modules disbale nginx #禁用
[root@node4 filebeat]# ./filebeat modules list
Enabled:
nginx
Disabled:
apache
auditd
aws
cef
cisco
coredns
elasticsearch
envoyproxy
googlecloud
haproxy
ibmmq
icinga
iis
iptables
kafka
kibana
logstash
mongodb
mssql
mysql
nats
netflow
osquery
panw
postgresql
rabbitmq
redis
santa
suricata
system
traefik
zeek
配置nginx module
[root@node4 filebeat]# cd modules.d/
[root@node4 modules.d]# vi nginx.yml
# Module: nginx # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-nginx.html - module: nginx # Access logs access: enabled: true var.paths: ["/usr/local/nginx/logs/access.log*"] # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. #var.paths: # Error logs error: enabled: true var.paths: ["/usr/local/nginx/logs/error.log*"] # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. #var.paths:
2.6 配置yml文件
[root@node4 modules.d]# cd ../
[root@node4 filebeat]# vi nginx-log.yml
filebeat.inputs: #- type: log # enabled: true # paths: # - /usr/local/nginx/logs/*.log # tags: ["nginx"] setup.template.settings: index.number_of_shards: 3 output.elasticsearch: hosts: ["192.168.132.131","192.168.132.132","192.168.132.133"] filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enable: false #output.console: # pretty: true # enable: true
[root@node4 filebeat]# ./filebeat -e -c nginx-log.yml
2019-11-24T09:54:43.129-0500 INFO crawler/crawler.go:72 Loading Inputs: 0 2019-11-24T09:54:43.131-0500 INFO log/input.go:152 Configured paths: [/usr/local/nginx/logs/access.log*] 2019-11-24T09:54:43.132-0500 INFO log/input.go:152 Configured paths: [/usr/local/nginx/logs/error.log*] 2019-11-24T09:54:43.132-0500 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0 2019-11-24T09:54:43.132-0500 INFO cfgfile/reload.go:171 Config reloader started 2019-11-24T09:54:43.134-0500 INFO log/input.go:152 Configured paths: [/usr/local/nginx/logs/access.log*] 2019-11-24T09:54:43.136-0500 INFO log/input.go:152 Configured paths: [/usr/local/nginx/logs/error.log*]
刷新nginx網頁
2.7 查看原始數據
{ "_index": "filebeat-7.4.2-2019.11.24-000001", "_type": "_doc", "_id": "TIvvnW4BNbSd3xvSwhR5", "_version": 1, "_score": 1, "_source": { "agent": { "hostname": "node4", "id": "2832793b-3bb6-4081-b05f-1955815440d0", "type": "filebeat", "ephemeral_id": "1e036f81-346b-42a6-b5c6-f197e9ba149c", "version": "7.4.2" }, "nginx": { "access": { "remote_ip_list": [ "192.168.132.1" ] } }, "log": { "file": { "path": "/usr/local/nginx/logs/access.log" }, "offset": 2533 }, "source": { "address": "192.168.132.1", "ip": "192.168.132.1" }, "fileset": { "name": "access" }, "url": { "original": "/" }, "input": { "type": "log" }, "@timestamp": "2019-11-24T15:02:38.000Z", "ecs": { "version": "1.1.0" }, "service": { "type": "nginx" }, "host": { "name": "node4" }, "http": { "request": { "referrer": "-", "method": "GET" }, "response": { "status_code": 304, "body": { "bytes": 0 } }, "version": "1.1" }, "event": { "timezone": "-05:00", "created": "2019-11-24T15:02:44.599Z", "module": "nginx", "dataset": "nginx.access" }, "user": { "name": "-" }, "user_agent": { "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36", "os": { "name": "Windows 10" }, "name": "Chrome", "device": { "name": "Other" }, "version": "78.0.3904" } } }
配置模塊后，message的信息前后如下
模塊配置后，信息更可讀化
Nginx的filebeat的module配置結束