Filebeat配置module采集nginx日志

环境

CentOS 7.3
Filebeat 7.6.0 （Filebeat安装和基本使用参考这里）
Elasticsearch 7.6.0
Nginx

Module

Filebeat集成了大量的module，可以简化我们的配置，命令如下

• 查看module列表

cd /usr/local/filebeat-7.6.0-linux-x86_64
./filebeat modules list
#输出如下：
Enabled:

Disabled:
activemq
apache
auditd
aws
azure
cef
cisco
coredns
elasticsearch
envoyproxy
googlecloud
haproxy
ibmmq
icinga
iis
iptables
kafka
kibana
logstash
misp
mongodb
mssql
mysql
nats
netflow
nginx
osquery
panw
postgresql
rabbitmq
redis
santa
suricata
system
traefik
zeek

可见，module默认都是关闭的，需要我们手动启用

• 启用nginx module

./filebeat modules enable nginx #启用
./filebeat modules disable nginx #关闭

• 修改nginx module配置

cd modules.d/
vim nginx.yml

主要是加上var.paths：

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/usr/local/nginx/logs/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/usr/local/nginx/logs/error.log*"]

配置filebeat

切换到Filebeat主目录，创建配置文件test-nginx.yml，内容如下：

setup.template.settings: 
  index.number_of_shards: 3
output.elasticsearch: 
  hosts: ["192.168.25.132:9200","192.168.25.133:9200","192.168.25.134:9200"]
filebeat.config.modules: 
  path: ${path.config}/modules.d/*.yml

启动nginx

略

启动filebeat

./filebeat -e -c test-nginx.yml

• 访问nginx所在的主机，多刷新几次

• 打开Elasticsearch，查看日志

{
    "_index":"filebeat-7.6.0-2020.08.27-000001",
    "_type":"_doc",
    "_id":"jT7hLnQBO0CPQYdfVdSz",
    "_version":1,
    "_score":1,
    "_source":{
        "agent":{
            "hostname":"cluster01",
            "id":"7f16be81-9321-40e6-8867-124f0bee0da0",
            "ephemeral_id":"6838a5f9-f7ee-4a83-94a3-74f98c8ff3b9",
            "type":"filebeat",
            "version":"7.6.0"
        },
        "nginx":{
            "access":{
                "remote_ip_list":[
                    "192.168.25.1"
                ]
            }
        },
        "log":{
            "file":{
                "path":"/usr/local/nginx/logs/access.log"
            },
            "offset":4348
        },
        "source":{
            "address":"192.168.25.1",
            "ip":"192.168.25.1"
        },
        "fileset":{
            "name":"access"
        },
        "url":{
            "original":"/"
        },
        "input":{
            "type":"log"
        },
        "@timestamp":"2020-08-27T07:45:49.000Z",
        "ecs":{
            "version":"1.4.0"
        },
        "service":{
            "type":"nginx"
        },
        "host":{
            "name":"cluster01"
        },
        "http":{
            "request":{
                "referrer":"-",
                "method":"GET"
            },
            "response":{
                "status_code":304,
                "body":{
                    "bytes":0
                }
            },
            "version":"1.1"
        },
        "event":{
            "timezone":"+08:00",
            "created":"2020-08-27T07:45:52.439Z",
            "module":"nginx",
            "dataset":"nginx.access"
        },
        "user":{
            "name":"-"
        },
        "user_agent":{
            "original":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36",
            "os":{
                "name":"Windows",
                "version":"10",
                "full":"Windows 10"
            },
            "name":"Chrome",
            "device":{
                "name":"Other"
            },
            "version":"78.0.3904.108"
        }
    }
}