ELK(ElasticSearch, Logstash, Kibana)搭建實時日志分析平台
日志主要包括系統日志、應用程序日志和安全日志。系統運維和開發人員可以通過日志了解服務器軟硬件信息、檢查配置過程中的錯誤及錯誤發生的原因。經常分析日志可以了解服務器的負荷，性能安全性，從而及時采取措施糾正錯誤。
通常，日志被分散的儲存不同的設備上。如果你管理數十上百台服務器，你還在使用依次登錄每台機器的傳統方法查閱日志。這樣是不是感覺很繁瑣和效率低下。當務之急我們使用集中化的日志管理，例如：開源的syslog，將所有服務器上的日志收集匯總。
集中化管理日志后，日志的統計和檢索又成為一件比較麻煩的事情，一般我們使用grep、awk和wc等Linux命令能實現檢索和統計，但是對於要求更高的查詢、排序和統計等要求和龐大的機器數量依然使用這樣的方法難免有點力不從心。
官方網站
https://www.elastic.co/
ELK中文指南 http://kibana.logstash.es/content/index.html
1、部署環境 [root@elk-node1 ~]# cat /etc/redhat-release CentOS release 6.8 (Final) 關閉防火牆&Sellinux http://blog.csdn.net/xiegh2014/article/details/53031781 配置yum源 http://blog.csdn.net/xiegh2014/article/details/53031894 兩台服務器
節點1安裝部署
主機hosts文件配置
[root@elk-node1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.8.95 elk-node1
172.16.8.96 elk-node2
JAVA安裝（安裝JDK需要重啟操作系統）
[root@elk-node1 ~]# rpm -ivh jdk-8u111-linux-x64.rpm
elasticsearch安裝
[root@elk-node1 ~]# rpm -ivh elasticsearch-5.1.1.rpm
[root@elk-node1 ~]# chkconfig --add elasticsearch
elasticsearch配置
[root@elk-node1 ~]# chkconfig --add elasticsearch
[root@elk-node1 ~]# mkdir -pv /data/elasticsearch/{data,logs}
mkdir: 已創建目錄 "/data"
mkdir: 已創建目錄 "/data/elasticsearch"
mkdir: 已創建目錄 "/data/elasticsearch/data"
mkdir: 已創建目錄 "/data/elasticsearch/logs"
[root@elk-node1 ~]# chown -R elasticsearch.elasticsearch /data/elasticsearch
[root@elk-node1 ~]# grep -n '^[a-z]' /etc/elasticsearch/elasticsearch.yml
[root@elk-node1 ~]# vi /etc/elasticsearch/elasticsearch.yml
[root@elk-node1 ~]# grep -n '^[a-z]' /etc/elasticsearch/elasticsearch.yml
17:cluster.name: app-elk
23:node.name: elk-node1
33:path.data: /data/elasticsearch/data
37:path.logs: /data/elasticsearch/logs
43:bootstrap.memory_lock: true
55:network.host: 0.0.0.0
59:http.port: 9200
修改elasticsearch的參數 # 換個集群的名字，免得跟別人的集群混在一起 cluster.name: es-5.0-test # 換個節點名字 node.name: node-101 # 修改一下ES的監聽地址，這樣別的機器也可以訪問 network.host: 0.0.0.0 # 默認的就好 http.port: 9200 # 增加新的參數，這樣head插件可以訪問es http.cors.enabled: true http.cors.allow-origin: "*" [root@elk-node1 ~]# vi /etc/security/limits.conf # allow user 'elasticsearch' mlockall elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited [root@elk-node2 ~]# vi /etc/security/limits.conf * soft nofile 65536 * hard nofile 131072 * soft nproc 2048 * hard nproc 4096 [root@elk-node1 ~]# vi /etc/security/limits.d/90-nproc.conf 修改如下內容： * soft nproc 1024 #修改為 * soft nproc 2048 [root@elk-node1 ~]# vi /etc/sysctl.conf 添加下面配置： vm.max_map_count=655360 [root@elk-node1 ~]# sysctl -p [root@elk-node1 ~]# /etc/init.d/elasticsearch restart http://172.16.8.95:9200/
節點2安裝部署
主機hosts文件配置
[root@elk-node1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.8.95 elk-node1
172.16.8.96 elk-node2
JAVA安裝（安裝JDK需要重啟操作系統）
[root@elk-node2 ~]# rpm -ivh jdk-8u111-linux-x64.rpm
elasticsearch安裝
[root@elk-node2 ~]# rpm -ivh elasticsearch-5.1.1.rpm
[root@elk-node2 ~]# chkconfig --add elasticsearch
elasticsearch配置
[root@elk-node2 ~]# mkdir -pv /data/elasticsearch/{data,logs}
[root@elk-node2 ~]# chown -R elasticsearch.elasticsearch /data/elasticsearch
[root@elk-node2 ~]# grep -n '^[a-z]' /etc/elasticsearch/elasticsearch.yml
17:cluster.name: app-elk
23:node.name: elk-node2
33:path.data: /data/elasticsearch/data
37:path.logs: /data/elasticsearch/logs
43:bootstrap.memory_lock: true
55:network.host: 0.0.0.0
59:http.port: 9200
[root@elk-node2 ~]# /etc/init.d/elasticsearch restart
報錯信息1
[root@elk-node2 ~]# tail -f /data/elasticsearch/logs/app-elk.log
[2016-09-19T18:08:11,804][INFO ][o.e.t.TransportService ] [elk-node2] publish_address {172.16.8.96:9300}, bound_addresses {[::]:9300}
[2016-09-19T18:08:11,825][INFO ][o.e.b.BootstrapCheck ] [elk-node2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-09-19T18:08:11,830][ERROR][o.e.b.Bootstrap ] [elk-node2] node validation exception
bootstrap checks failed
memory locking requested for elasticsearch process but memory is not locked
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2016-09-19T18:08:11,842][INFO ][o.e.n.Node ] [elk-node2] stopping ...
[2016-09-19T18:08:11,896][INFO ][o.e.n.Node ] [elk-node2] stopped
[2016-09-19T18:08:11,896][INFO ][o.e.n.Node ] [elk-node2] closing ...
[2016-09-19T18:08:11,933][INFO ][o.e.n.Node ] [elk-node2] closed
[root@elk-node2 ~]# vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
[root@elk-node2 ~]# vi /etc/security/limits.d/90-nproc.conf
修改如下內容：
* soft nproc 1024
#修改為
* soft nproc 2048
[root@elk-node2 ~]# vi /etc/sysctl.conf
添加下面配置：
vm.max_map_count=655360
[root@elk-node2 ~]# sysctl -p
[root@elk-node2 ~]# /etc/init.d/elasticsearch restart
報錯信息2
[2016-09-19T18:18:19,270][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=無法分配內存
[2016-09-19T18:18:19,270][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
[2016-09-19T18:18:19,270][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2016-09-19T18:18:19,271][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[2016-09-19T18:18:19,271][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2016-09-19T18:18:20,000][INFO ][o.e.n.Node ] [elk-node2] initializing ...
[2016-09-19T18:18:20,384][INFO ][o.e.e.NodeEnvironment ] [elk-node2] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [39gb], net total_space [43.9gb], spins? [possibly], types [ext4]
[2016-09-19T18:18:20,385][INFO ][o.e.e.NodeEnvironment ] [elk-node2] heap size [3.9gb], compressed ordinary object pointers [true]
[2016-09-19T18:18:20,391][INFO ][o.e.n.Node ] [elk-node2] node name [elk-node2], node ID [KBLSr8zERri083vvtJBQhA]
[2016-09-19T18:18:20,405][INFO ][o.e.n.Node ] [elk-node2] version[5.1.1], pid[25073], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/2.6.32-642.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_111/25.111-b14]
[2016-09-19T18:18:29,227][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [aggs-matrix-stats]
[2016-09-19T18:18:29,228][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [ingest-common]
[2016-09-19T18:18:29,228][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [lang-expression]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [lang-groovy]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [lang-mustache]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [lang-painless]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [percolator]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [reindex]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [transport-netty3]
[2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [transport-netty4]
[2016-09-19T18:18:29,231][INFO ][o.e.p.PluginsService ] [elk-node2] no plugins loaded
[root@elk-node2 ~]# vi /etc/security/limits.conf
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[root@elk-node2 ~]# /etc/init.d/elasticsearch restart
http://172.16.8.96:9200/
