Filebeat配置module采集nginx日志

環境

CentOS 7.3
Filebeat 7.6.0 （Filebeat安裝和基本使用參考這里）
Elasticsearch 7.6.0
Nginx

Module

Filebeat集成了大量的module，可以簡化我們的配置，命令如下

• 查看module列表

cd /usr/local/filebeat-7.6.0-linux-x86_64
./filebeat modules list
#輸出如下：
Enabled:

Disabled:
activemq
apache
auditd
aws
azure
cef
cisco
coredns
elasticsearch
envoyproxy
googlecloud
haproxy
ibmmq
icinga
iis
iptables
kafka
kibana
logstash
misp
mongodb
mssql
mysql
nats
netflow
nginx
osquery
panw
postgresql
rabbitmq
redis
santa
suricata
system
traefik
zeek

可見，module默認都是關閉的，需要我們手動啟用

• 啟用nginx module

./filebeat modules enable nginx #啟用
./filebeat modules disable nginx #關閉

• 修改nginx module配置

cd modules.d/
vim nginx.yml

主要是加上var.paths：

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/usr/local/nginx/logs/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/usr/local/nginx/logs/error.log*"]

配置filebeat

切換到Filebeat主目錄，創建配置文件test-nginx.yml，內容如下：

setup.template.settings: 
  index.number_of_shards: 3
output.elasticsearch: 
  hosts: ["192.168.25.132:9200","192.168.25.133:9200","192.168.25.134:9200"]
filebeat.config.modules: 
  path: ${path.config}/modules.d/*.yml

啟動nginx

略

啟動filebeat

./filebeat -e -c test-nginx.yml

• 訪問nginx所在的主機，多刷新幾次

• 打開Elasticsearch，查看日志

{
    "_index":"filebeat-7.6.0-2020.08.27-000001",
    "_type":"_doc",
    "_id":"jT7hLnQBO0CPQYdfVdSz",
    "_version":1,
    "_score":1,
    "_source":{
        "agent":{
            "hostname":"cluster01",
            "id":"7f16be81-9321-40e6-8867-124f0bee0da0",
            "ephemeral_id":"6838a5f9-f7ee-4a83-94a3-74f98c8ff3b9",
            "type":"filebeat",
            "version":"7.6.0"
        },
        "nginx":{
            "access":{
                "remote_ip_list":[
                    "192.168.25.1"
                ]
            }
        },
        "log":{
            "file":{
                "path":"/usr/local/nginx/logs/access.log"
            },
            "offset":4348
        },
        "source":{
            "address":"192.168.25.1",
            "ip":"192.168.25.1"
        },
        "fileset":{
            "name":"access"
        },
        "url":{
            "original":"/"
        },
        "input":{
            "type":"log"
        },
        "@timestamp":"2020-08-27T07:45:49.000Z",
        "ecs":{
            "version":"1.4.0"
        },
        "service":{
            "type":"nginx"
        },
        "host":{
            "name":"cluster01"
        },
        "http":{
            "request":{
                "referrer":"-",
                "method":"GET"
            },
            "response":{
                "status_code":304,
                "body":{
                    "bytes":0
                }
            },
            "version":"1.1"
        },
        "event":{
            "timezone":"+08:00",
            "created":"2020-08-27T07:45:52.439Z",
            "module":"nginx",
            "dataset":"nginx.access"
        },
        "user":{
            "name":"-"
        },
        "user_agent":{
            "original":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36",
            "os":{
                "name":"Windows",
                "version":"10",
                "full":"Windows 10"
            },
            "name":"Chrome",
            "device":{
                "name":"Other"
            },
            "version":"78.0.3904.108"
        }
    }
}