使用Docker鏡像部署ELK日志系統

使用Docker部署elasticsearch、logstash、kibana

指定版本：6.7.1 （建議使用同一的版本、屏蔽三個軟件間的不兼容性）

下載鏡像：

docker pull elasticsearch:6.7.1

docker pull logstash:6.7.1

docker pull kibana:6.7.1

 

修改vm.max_map_count

vim /etc/sysctl.conf

添加配置：vm.max_map_count=262144

執行命令，確保生效配置生效： sysctl -p

依據服務器配置而定

 

es集群

這里es集群用了3個節點，配置文件放在 /root/es/config/ 目錄下，

注意：es是不允許使用root用戶啟動的，/root/es/config/  目錄最好改成 777權限，

es1.yml配置文件內容：

cluster.name: elasticsearch-cluster
node.name: es-node1
network.bind_host: 0.0.0.0
network.publish_host: 10.90.101.48
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true 
node.data: true 
discovery.zen.ping.unicast.hosts: ["10.90.101.48:9300","10.90.101.48:9301","10.90.101.48:9302"]
discovery.zen.minimum_master_nodes: 2
xpack.ml.enabled: false
xpack.monitoring.enabled: false
xpack.security.enabled: false
xpack.watcher.enabled: false

啟動當前配置文件的es命令：

　　docker run -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -d -p 9200:9200 -p 9300:9300 -v /root/es/config/es1.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /root/es/data1:/usr/share/elasticsearch/data --name ES01 elasticsearch:6.7.1

 

es2.yml配置文件內容：

cluster.name: elasticsearch-cluster
node.name: es-node2
network.bind_host: 0.0.0.0
network.publish_host: 10.90.101.48
http.port: 9201
transport.tcp.port: 9301
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true 
node.data: true 
discovery.zen.ping.unicast.hosts: ["10.90.101.48:9300","10.90.101.48:9301","10.90.101.48:9302"]
discovery.zen.minimum_master_nodes: 2
xpack.ml.enabled: false
xpack.monitoring.enabled: false
xpack.security.enabled: false
xpack.watcher.enabled: false

啟動當前配置文件的es命令：

　　docker run -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -d -p 9201:9201 -p 9301:9301 -v /root/es/config/es2.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /root/es/data2:/usr/share/elasticsearch/data --name ES02 elasticsearch:6.7.1

 

 

es3.yml配置文件內容：

cluster.name: elasticsearch-cluster
node.name: es-node3
network.bind_host: 0.0.0.0
network.publish_host: 10.90.101.48
http.port: 9202
transport.tcp.port: 9302
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true 
node.data: true 
discovery.zen.ping.unicast.hosts: ["10.90.101.48:9300","10.90.101.48:9301","10.90.101.48:9302"]
discovery.zen.minimum_master_nodes: 2
xpack.ml.enabled: false
xpack.monitoring.enabled: false
xpack.security.enabled: false
xpack.watcher.enabled: false

啟動當前配置文件的es命令：

　　docker run -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -d -p 9202:9202 -p 9302:9302 -v /root/es/config/es3.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /root/es/data3:/usr/share/elasticsearch/data --name ES03 elasticsearch:6.7.1

 

 

1、-e JAVA_OPTS="-Xms512m -Xmx512m" 是配置當前es使用jvm的最大內存，內存配置過低會導致CPU非常高，如果服務器內存很大可配置2~4g

2、-p 9200:9200 -p 9300:9300  9200是es提供給外部的通訊端口，9300是es節點之間的通訊端口

3、 -v /root/es/config/es1.yml:/usr/share/elasticsearch/config/elasticsearch.yml  指定啟動的配置文件

4、-v /root/es/data1:/usr/share/elasticsearch/data 指定es的數據掛載到當前服務器的目錄

5、--name ES01  當前鏡像的名字

 

kibana

kibana.yml配置文件內容：

server.name: kibana
server.host: "0"
elasticsearch.url: http://10.90.101.48:9300
xpack.monitoring.ui.container.elasticsearch.enabled: false
xpack.security.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: false

啟動kibana

　　docker run --name kibana -v /root/kibana/config:/usr/share/kibana/config -p 5601:5601 -d kibana:6.7.1

 

logstash

把images中的配置文件拷貝出來：

1. 先運行一個logstash實例

2. docker cp 容器id:/usr/share/logstash/config /root/logstash/config 

    docker cp 容器id:/usr/share/logstash/pipeline /root/logstash/pipeline

pipeline/logstash.conf

input{
    http{
        host => "0.0.0.0"
        port => 5050
        additional_codecs => {"application/json"=>"json"}
        codec => "plain"
        threads => 4
        ssl => false
    }
}
output {
    elasticsearch {
        hosts => ["http://10.90.101.48:9200","http://10.90.101.51:9201","http://10.90.101.51:9202"]
        index => "log_%{logtype}_%{+YYYY.MM.dd}"
    }
}

設置日志輸入輸出方式

 

config/logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.enabled: false

啟動logstash

　　docker run -d --name logstash -p 5050:5050 -v /root/logstash/config:/usr/share/logstash/config -v /root/logstash/pipeline:/usr/share/logstash/pipeline logstash:6.7.1

 

 

cerebro 一個管理es的工具

docker pull lmenezes/cerebro

啟動cerebro

　　docker run --name es-head -p 9000:9000 -d lmenezes/cerebro

 

瀏覽器中打開  ip:9000 鏈接 http://ip:9200 即可看到es 集群的狀態