ELK之使用filebeat收集系統數據及其他程序並生成可視化圖表

　　當您要面對成百上千、甚至成千上萬的服務器、虛擬機和容器生成的日志時，請告別 SSH 吧。Filebeat 將為您提供一種輕量型方法，用於轉發和匯總日志與文件，讓簡單的事情不再繁雜。

　　1,安裝filebeat

rpm -ivh /nas/nas/softs/elk/6.5.4/filebeat-6.5.4-x86_64.rpm

　　查看模塊

[root@prd-elk-kafka-01 ~]# filebeat modules list
Enabled:
kafka
system

Disabled:
apache2
auditd
elasticsearch
haproxy
icinga
iis
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
suricata
traefik

　　如需要啟用某個模塊使用命令

filebeat modules enable system

　　修改配置文件把filebeat輸出至elasticsearch和kibana

/etc/filebeat/filebeat.yml

　　

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
output.elasticsearch:
  hosts: ["172.16.90.24:9200"]
#  username: "admin"
#  password: "admin"
setup.kibana:
  host: "172.16.90.24:5601"
processors:
  - add_host_metadata:
  - add_cloud_metadata:
#  - drop_fields:
#      fields: ["beat", "input", "source", "offset", "prospector","host"]

　　模塊路徑為/etc/filebeat/modules.d/

　　2，設置系統模塊system

　　啟用系統模塊

/etc/filebeat/filebeat.yml

　　編輯系統模塊配置文件，這里使用默認

/etc/filebeat/modules.d/system.yml

　　

- module: system
  # Syslog
  syslog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    #var.convert_timezone: false

  # Authorization logs
  auth:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    #var.convert_timezone: false

　　系統模塊每一台主機都需要安裝，啟用后查看kibana

 

　　3，設置nginx模塊

　　啟動nginx模塊

filebeat modules enable nginx

　　修改配置文檔

/etc/filebeat/modules.d/nginx.yml

　　輸入access日志及error日志路徑

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - "/opt/log/wwwaccess.log"
      - "/opt/log/workaccess.log"
      - "/opt/log/datavaccess.log"

  # Error logs
  error:
    enabled: true


    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - "/opt/log/wwwerror.log"
      - "/opt/log/workerror.log"
      - "/opt/log/dataverror.log"

　　kibana查看

　　要顯示客戶端城市及瀏覽器信息需要安裝elasticsearch插件

bin/elasticsearch-plugin install ingest-geoip
bin/elasticsearch-plugin install ingest-user-agent

　　

　　4，設置redis模塊

　　啟用redis模塊

filebeat modules enable redis

　　修改redis配置文件

 /etc/filebeat/modules.d/redis.yml

　　

- module: redis
  # Main logs
  log:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/nas/nas/logs/redis/redis-server.log"]

  # Slow logs, retrieved via the Redis API (SLOWLOG)
  slowlog:
    enabled: true

    # The Redis hosts to connect to.
    var.hosts: ["localhost:6379"]

    # Optional, the password to use when connecting to Redis.
    var.password: "password"

　　kibana頁面查看

 

　　

　　5，設置kafka模塊

　　啟用kafka模塊

filebeat modules enable kafka

　　修改配置文件

/etc/filebeat/modules.d/kafka.yml

　　

- module: kafka
  # All logs
  log:
    enabled: true

    # Set custom paths for Kafka. If left empty,
    # Filebeat will look under /opt.
    #var.kafka_home:

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - "/usr/local/kafka/logs/controller.log*"
      - "/usr/local/kafka/logs/server.log*"
      - "/usr/local/kafka/logs/state-change.log*"
      - "/usr/local/kafka/logs/kafka-*.log*"
    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    #var.convert_timezone: false

　　kibana查看

 

　　

　　6，設置MySQL模塊

　　開啟mysql模塊

filebeat modules enable mysql

　　修改配置文件

/etc/filebeat/modules.d/mysql.yml

　　輸入日志及慢查詢日志文件路徑

- module: mysql
  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/mysql3306.log"]

  # Slow logs
  slowlog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/nas/nas/logs/mysql/slow3306.log"]

　　kibana查看

 

　　更多模塊查看官方文檔 https://www.elastic.co/guide/en/beats/filebeat/6.5/filebeat-modules.html