Filebeat介紹與使用

介紹：

Filebeat附帶預構建的模塊，這些模塊包含收集、解析、充實和可視化各種日志文件格式數據所需的配置，每個Filebeat模塊由一個或多個文件集組成，這些文件集包含攝取節點管道、Elasticsearch模板、Filebeat勘探者配置和Kibana儀表盤。

filebeat和logstash是一樣的作用
ELK都是Java程序寫的 
filebeat是golang寫的  #速度非常快

Filebeat模塊很好的入門，它是輕量級單用途的日志收集工具，用於在沒有安裝java的服務器上專門收集日志，可以將日志轉發到logstash、elasticsearch或redis等場景中進行下一步處理。

filebeat 支持的服務

1.安裝filebeat

#上傳代碼包
[root@logstash ~]# rz filebeat-6.6.0-x86_64.rpm

#安裝
[root@logstash ~]# rpm -ivh filebeat-6.6.0-x86_64.rpm

2.配置文件

[root@logstash ~]# rpm -qc filebeat
/etc/filebeat/filebeat.yml

3.日志

[root@logstash ~]# less /var/log/filebeat/filebeat

二、Filebeat收集單類型日志到本地文件

配置Filebeat

1.配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml  
filebeat.inputs:
- type: log
  enabled: true                   #配置文件的格式必須遵守yml的格式寫法，否則報錯
  paths:
    - /var/log/messages

output.file:
  path: "/tmp"
  filename: "filebeat_message.log"    

2.啟動

[root@logstash ~]# systemctl start filebeat.service
[root@logstash ~]# ps -ef | grep filebeat
[root@logstash ~]# ps -ef | grep filebeat
root      12418      1  2 16:55 ?        00:00:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root      12438   7019  0 16:56 pts/0    00:00:00 grep --color=auto filebeat

3.測試

[root@logstash ~]# echo 1111 >> /var/log/messages

#得到內容
[root@logstash ~]# tail -f /tmp/filebeat_message.log
{"@timestamp":"2020-07-21T08:58:00.373Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.6.0"},"source":"/var/log/messages","offset":230243,"prospector":{"type":"log"},"input":{"type":"log"},"beat":{"version":"6.6.0","name":"logstash","hostname":"logstash"},"host":{"name":"logstash"},"log":{"file":{"path":"/var/log/messages"}},"message":"1111"}

三、filebeat收集單個日志到ES

1.配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]    #收集日志到elasticsearch 不需要指定索引名字
  
[root@logstash ~]# systemctl restart filebeat.service

2.訪問nginx測試

3.指定ES索引名配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx_json-%{+yyyy.MM.dd}"  #filebeat 不支持自定義索引，需要添加下面的setup幾項
  #index: "nginx_json-%{[beat.version]}-%{+yyyy.MM.dd}"  如果有多個filebeat版本號，可以加上filebeat版本     
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true     #注意 這幾行配置必須頂個寫
setup.template.enabled: false
setup.ilm.enabled: false

[root@logstash ~]# systemctl restart filebeat

4.修改kibana中日志展示格式

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx_json-%{+yyyy.MM.dd}"   
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

四、收集單個日志到reids

1.配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.redis:
  hosts: ["172.16.1.53:6381"]
  db: "0"
  key: "nginx_json_redis"

2.啟動

[root@logstash ~]# systemctl restart filebeat

3.訪問nginx，查看redis

[root@db03 ~]# redis-cli -p 6381 --raw
127.0.0.1:6381> keys *
nginx_json_redis
127.0.0.1:6381> LLEN nginx_json_redis
8

4.配置將redis數據取出到ES

[root@logstash ~]# vim /etc/logstash/conf.d/filebeat_redis_es.conf
input {
  redis {
    host => "172.16.1.53"
    port => 6381
    data_type => "list"
    db => "0"
    key => "nginx_json_redis"
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "filebeat_redis_es_%{+YYYY-MM-dd}"
  }
}

[root@logstash ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/filebeat_redis_es.conf

五、filebeat收集日志到logstash

1.配置filebeat

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.logstash:
  hosts: ["10.0.0.54:6666"]

2.配置logstash接收數據傳給ES

[root@logstash ~]# vim /etc/logstash/conf.d/filebeat_logstash_es.conf
input {
  beats {
    port => 6666
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "ngx_file_log_es_%{+YYYY-MM-dd}"
  }
}

一、filebeat收集多個日志到ES

1.配置方式一：

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx_json_%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        source: "/var/log/nginx/nginx_json.log"
    - index: "nginx_access_%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        source: "/var/log/nginx/access.log"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

2.配置方式二：

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["json"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  tags: ["access"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx_json_%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "json"
    - index: "nginx_access_%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "access"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false

二、filebeat收集java報錯日志

1.配置

[root@logstash ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/nginx_json.log
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx_json_%{[beat.version]}-%{+yyyy.MM.dd}"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.overwrite: false
setup.template.json.enabled: true
setup.template.enabled: false
setup.ilm.enabled: false