elk +redis 環境搭建

本文轉載自查看原文 2018-02-24 16:01 2046 elk

這個是最新的elk+redis搭建日志分析平台，今年時間是2015年9月11日。

Elk分別為 elasticsearch，logstash， kibana

官網為：https://www.elastic.co/products

Elasticsearch:

https://www.elastic.co/downloads/elasticsearch
https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.tar.gz

logstash:

https://www.elastic.co/downloads/logstash

https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz

kibana:

https://www.elastic.co/products/kibana

https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz

先說一下規划

192.168.15.62為日志查看服務器，該機器需要安裝redis, elasticserch, logstatsh, kibana這四個應用程序。
192.168.15.2 為應用程序nginx應用程序，我們這次只收集他的日志進行分析。

安裝

Yum install vim unix2dos wget curl curl-devel expect expect-devel

Yum install gcc glibc glib2 libgcc glibc-devel glib2-devel

Yum install m4 automake autoconf cmake cpp

Yum install gcc-c++ libstdc++ libstdc++-devel libstdc++-docs

在192.168.15.62機器上：

以root身份登陸，在root的家目錄“/root”下：

Wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.tar.gz

Wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz

Wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz

Wget http://download.redis.io/releases/redis-3.0.4.tar.gz

#tar –xvzf redis-3.0.4.tar.gz

#cd redis-3.0.4

#make

#make install

#cd utils

#./install_server.sh

這樣redis就安裝好了
#which redis-server在/usr/local/bin/redis-server

配置文件放在/etc/redis/6379.conf文件

#redis-cli

>set aa aavalue

>get aa 即可測試。
#cd

#tar –xvzf elasticsearch-1.7.1.tar.gz

#cp –a elasticsearch-1.7.1 /usr/local

#cd /usr/local

#ln –s elasticsearch-1.7.1 elasticsearch

#cd

#tar –xvzf logstash-1.5.4.tar.gz

#cp –a logstash-1.5.4 /usr/local

#cd /usr/local

#ln –s logstash-1.5.4 logstash

#cd

#tar –xvzf kibana-4.1.2-linux-x64.tar.gz

#cp –a kibana-4.1.2-linux-x64 /usr/local

#cd /usr/local

#ln –s kibana-4.1.2-linux-x64 kibana

程序安裝全部完成。

由於elasticsearch和logstash是安裝在一台機器上所以elasticsearch默認配置即可。
/usr/local/elasticsearch/bin/elasticsearch –d （以deamon方式啟動elasticsearch）
打開192.168.15.62:9200即可看到

配置logstash程序
# cd /usr/local/logstash下，mkdir etc logs兩個文件夾，etc用於存放配置文件，logs用於存放日志文件
在該etc目錄下建立一個central.conf配置文件：

該文件說明是以redis為輸入，輸出到elasticsearch程序，格式為json協議為http.

如果：從文件輸入，輸出到redis

look@DZG-SERVICE-36:/etc/logstash/conf.d$ cat wealth_product_service.conf 
input {


   file {
        path => "/data/dubbo/wealth-xxx-service/logs/xxx-service.log"
        path => "/data/dubbo/wealth-xxx-service/logs/error.log"
        type=> "wealth_xxx_service"
#        start_position => "beginning"
}

}


output {


if [type] == "wealth_xxx_service" {
# elasticsearch {
#    hosts => "172.16.1.34"
#    index => "wealth_xxx_service-%{+YYYY.MM.dd}" #匹配的索引模式
#        }

        redis {
                host => "127.0.1.23"
                port => 6379
                data_type => "list"        #數據類型也要保持一致
                key => "wealth_xxx_service"          #key 要與服務器端的相同
        }

}


}

然后從redis寫數據到elasticsearch

input {
redis {
host => "127.0.0.1" #本地的reds地址
port => 6379 #redis端口
type => "redis-input" #輸入類型
data_type => "list" #使用redis的list存儲數據
key => "wealth_xxx_service"
}
}

output {
# stdout {}

elasticsearch {
hosts => "172.16.1.34" #elasticsearch地址
index => "wealth_xxx_service-%{+YYYY.MM.dd}"
codec => "json"
}
}

啟動logstash程序如圖：
#/usr/local/logstash agent –verbose --config /usr/local/logstash/etc/central.conf –log /usr/local/logstash/logs/stdou.log