ELK之logstash收集日志寫入redis及讀取redis

logstash->redis->logstash->elasticsearch

1.安裝部署redis

cd /usr/local/src
wget http://download.redis.io/releases/redis-3.2.8.tar.gz
tar xf redis-3.2.8.tar.gz
cd redis-3.2.8/
make
ln -s /usr/local/src/redis-3.2.8 /usr/local/redis
cd /usr/local/redis/

vim redis.conf 
bind 10.0.0.22
daemonize yes
save ""
#save 900 1
#save 300 10
#save 60 10000
requirepass root123

cp src/redis-server /usr/bin/
cp src/redis-cli /usr/bin/
redis-server /usr/local/redis/redis.conf

登錄redis需要認證

配置logstash的systemlog_to_redis.conf

vim systemlog_to_redis.conf
input {
  file {
    path => "/var/log/messages"
    type => "systemlog"
    start_position => "beginning"
    stat_interval => "2"
  }
}

output {
  if [type] == "systemlog" {
    redis {
      data_type => "list"
      host => "10.0.0.22"
      db => "1"
      port => "6379"
      password => "root123"
      key => "systemlog"
    }
  }
}
systemctl restart logstash
# 手動寫入messages日志
cat /etc/hosts >> /var/log/messages
echo "helloword" >> /var/log/messages

登陸redis查看

2.配置logstash從reids中取出數據到elasticsearch

# 使用linux-elk2(10.0.0.33)上的logstash從redis取數據
vim redis-es.conf 
input {
  redis {
    data_type => "list"
    host => "10.0.0.22"
    db => "1"
    port => "6379"
    key => "systemlog"
    password => "root123"
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.33:9200"]
    index => "redis-systemlog-%{+YYYY.MM.dd}"
  }
}
systemctl restart logstash

logstash統計日志,有兩個以上的key時,就必須加判斷

 

收集日志寫入redis及讀取redis:http://blog.51cto.com/jinlong/2056563