由Logstash收集日志寫入Redis,再有Logstash讀取Redis在寫入elasticsearch
作為日志緩存介質
官方文檔:https://www.elastic.co/guide/en/logstash/current/plugins-outputs-redis.html
一 配置Logstash寫入Redis
1.1.1 配置logstash配置文件
[root@localhost ~]# cat /etc/logstash/conf.d/nginx.conf input { file { path => "/opt/vhosts/fatai/logs/access_json.log" start_position => "beginning" type => "nginx-accesslog" codec => json stat_interval => "2" } } output { if [type] == "nginx-accesslog" { redis { data_type => "list" key => "nginx-accesslog-test" host => "192.168.10.240" port => "6379" db => "0" password => "123456" } } }
1.1.2 驗證配置文件並重啟
[root@localhost ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console Configuration OK [root@localhost ~]# systemctl restart logstash.service
1.1.3 檢查redis是否有key
二 另一台機器配置logstash讀取redis文件並寫入elasticsearch
[root@DNS-Server tools]# cat /etc/logstash/conf.d/nginx.conf input { redis { data_type => "list" key => "nginx-accesslog-test" host => "192.168.10.240" port => "6379" db => "0" password => "123456" codec => "json" } } output { elasticsearch { hosts => ["192.168.10.10:9200"] index => "logstash-redis-logg-%{+YYYY.MM.dd}" } }
elasticsearch-head驗證