ELK之filebeat收集多類型日志

1.IP規划

10.0.0.33:filebeat+tomcat,filebeat收集系統日志、tomcat日志發送到logstash

10.0.0.32:logstash,將日志寫入reids(input、output)

10.0.0.31:redis,大量緩存數據

10.0.0.30:logstash,從redis取出數據寫入es(input、output)

10.0.0.29:es+kibana,es接收傳來的數據寫入磁盤,等待kibana來取

a.10.0.0.33:filebeat輸出到logstash

vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
    - /var/log/messages
  exclude_lines: ['^DBG',"^$"]
  document_type: filebeat-systemlog-0033
- input_type: log
  paths:
    - /usr/local/tomcat/logs/tomcat_access_log.*.log
  exclude_lines: ['^DBG',"^$"]
  document_type: tomcat-accesslog-0033
output.logstash:
  hosts: ["10.0.0.32:5044"]
  enabled: true
  worker: 2
  compression_level: 3

systemctl restart filebeat

b.10.0.0.32:logstash將日志寫入reids(向redis寫數據不需要給key加日期)

vim beats.conf 

input {
  beats {
    port => "5044"
  }
}
output {
  if [type] == "filebeat-systemlog-0033" {
    redis {
      data_type => "list"
      host => "10.0.0.31"
      db => "3"
      port => "6379"
      password => "123456"
      key => "filebeat-systemlog-0033"
    }
  }
  if [type] == "tomcat-accesslog-0033" {
    redis {
      data_type => "list"
      host => "10.0.0.31"
      db => "4"
      port => "6379"
      password => "123456"
      key => "tomcat-accesslog-0033"
    }
  }
}

systemctl restart logstash

c.10.0.0.31:redis不用做什么操作

d.10.0.0.30:logstash從redis取出數據寫入es

vim redis-es.conf
input {
  redis {
    data_type => "list"
    host => "10.0.0.31"
    db => "3"
    port => "6379"
    key => "filebeat-systemlog-0033"
    password => "123456"
  }
  redis {
    data_type => "list"
    host => "10.0.0.31"
    db => "4"
    port => "6379"
    key => "tomcat-accesslog-0033"
    password => "123456"
  }
}

output {
  if [type] == "filebeat-systemlog-0033" {
    elasticsearch {
      hosts => ["10.0.0.29:9200"]
      index => "redis31-systemlog-%{+YYYY.MM.dd}"
    }
  }
  if [type] == "tomcat-accesslog-0033" {
    elasticsearch {
      hosts => ["10.0.0.29:9200"]
      index => "tomcat-accesslog-0033-%{+YYYY.MM.dd}"
    }
  }
}
systemctl restart logstash

e.10.0.0.29:es+kibana

es插件頁面出現這個日志索引時tomcat-accesslog-0033-xxxx.xx.xx,代表整個流程是通的.

 

ELK架構實用演示:http://blog.51cto.com/jinlong/2056717