目錄
一、介紹
二、安裝JDK
三、安裝Elasticsearch
四、安裝Logstash
五、安裝Kibana
六、Kibana簡單使用
系統環境：CentOS Linux release 7.4.1708 (Core)
當前問題狀況
- 開發人員不能登錄線上服務器查看詳細日志。
- 各個系統都有日志，日志數據分散難以查找。
- 日志數據量大，查詢速度慢，或者數據不夠實時。
一、介紹
1、組成
ELK由Elasticsearch、Logstash和Kibana三部分組件組成；
Elasticsearch是個開源分布式搜索引擎，它的特點有：分布式，零配置，自動發現，索引自動分片，索引副本機制，restful風格接口，多數據源，自動搜索負載等。
Logstash是一個完全開源的工具，它可以對你的日志進行收集、分析，並將其存儲供以后使用
kibana 是一個開源和免費的工具，它可以為 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以幫助您匯總、分析和搜索重要數據日志。
2、四大組件
Logstash: logstash server端用來搜集日志；
Elasticsearch: 存儲各類日志；
Kibana: web化接口用作查尋和可視化日志；
Logstash Forwarder: logstash client端用來通過lumberjack 網絡協議發送日志到logstash server；
3、工作流程
在需要收集日志的所有服務上部署logstash，作為logstash agent（logstash shipper）用於監控並過濾收集日志，將過濾后的內容發送到Redis，然后logstash indexer將日志收集在一起交給全文搜索服務ElasticSearch，可以用ElasticSearch進行自定義搜索通過Kibana 來結合自定義搜索進行頁面展示。
下面是在兩台節點上都安裝一下環境。
二、安裝JDK
配置阿里源：wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo yum clean all yum makecache Logstash的運行依賴於Java運行環境，Elasticsearch 要求至少 Java 7。 [root@controller ~]# yum install java-1.8.0-openjdk -y [root@controller ~]# java -version openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode) 1、關閉防火牆 systemctl stop firewalld.service #停止firewall systemctl disable firewalld.service #禁止firewall開機啟動 2、關閉selinux sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config setenforce 0
三、安裝Elasticsearch
基礎環境安裝（elk-node1和elk-node2同時操作）
1）下載並安裝GPG Key
[root@elk-node1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2）添加yum倉庫
[root@elk-node1 ~]# vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
3）安裝elasticsearch
[root@elk-node1 ~]# yum install -y elasticsearch
4）添加自啟動
chkconfig --add elasticsearch
5）啟動命令
systemctl daemon-reload
systemctl enable elasticsearch.service
6）修改配置
[root@elk-node1 ~]# cd /etc/elasticsearch/
[root@elk-node1 elasticsearch]# ls
elasticsearch.yml logging.yml scripts
[root@elk-node1 elasticsearch]# cp elasticsearch.yml{,.bak}
[root@elk-node1 elasticsearch]# mkdir -p /data/es-data
[root@elk-node1 elasticsearch]# vim elasticsearch.yml
[root@elk-node1 elasticsearch]# grep '^[a-z]' elasticsearch.yml
cluster.name: hejianlai //集群名稱
node.name: elk-node1 //節點名稱
path.data: /data/es-data //數據存放目錄
path.logs: /var/log/elasticsearch/ //日志存放目錄
bootstrap.memory_lock: true //打開內存
network.host: 0.0.0.0 //監聽網絡
http.port: 9200 //端口
discovery.zen.ping.multicast.enabled: false //改為單播
discovery.zen.ping.unicast.hosts: ["192.168.247.135", "192.168.247.133"]
[root@elk-node1 elasticsearch]# systemctl start elasticsearch
You have new mail in /var/spool/mail/root
[root@elk-node1 elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2018-07-12 22:00:47 CST; 9s ago
Docs: http://www.elastic.co
Process: 22333 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 22331 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 22333 (code=exited, status=1/FAILURE)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at java.nio.file.Files.createDirectory(Files.java:674)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at java.nio.file.Files.createDirectories(Files.java:767)
Jul 12 22:00:47 elk-node1 elasticsearch[22333]: at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:337)
Jul 12 22:00:47 elk-node1 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 12 22:00:47 elk-node1 systemd[1]: Unit elasticsearch.service entered failed state.
Jul 12 22:00:47 elk-node1 systemd[1]: elasticsearch.service failed.
[root@elk-node1 elasticsearch]# cd /var/log/elasticsearch/
[root@elk-node1 elasticsearch]# ll
total 4
-rw-r--r-- 1 elasticsearch elasticsearch 0 Jul 12 22:00 hejianlai_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 Jul 12 22:00 hejianlai_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 Jul 12 22:00 hejianlai_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 2232 Jul 12 22:00 hejianlai.log
[root@elk-node1 elasticsearch]# tail hejianlai.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:337)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:314)
... 7 more
[root@elk-node1 elasticsearch]# less hejianlai.log
You have new mail in /var/spool/mail/root
[root@elk-node1 elasticsearch]# grep elas /etc/passwd
elasticsearch:x:991:988:elasticsearch user:/home/elasticsearch:/sbin/nologin
#報錯/data/es-data沒權限，賦權限即可
[root@elk-node1 elasticsearch]# chown -R elasticsearch:elasticsearch /data/es-data/
[root@elk-node1 elasticsearch]# systemctl start elasticsearch
[root@elk-node1 elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2018-07-12 22:03:28 CST; 4s ago
Docs: http://www.elastic.co
Process: 22398 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 22400 (java)
CGroup: /system.slice/elasticsearch.service
└─22400 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMe...
Jul 12 22:03:29 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:29,739][WARN ][bootstrap ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
Jul 12 22:03:29 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:29,899][INFO ][node ] [elk-node1] version[2.4.6], pid[22400], build[5376dca/2017-07-18T12:17:44Z]
Jul 12 22:03:29 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:29,899][INFO ][node ] [elk-node1] initializing ...
Jul 12 22:03:30 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:30,644][INFO ][plugins ] [elk-node1] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
Jul 12 22:03:30 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:30,845][INFO ][env ] [elk-node1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [1.7gb], n...types [rootfs]
Jul 12 22:03:30 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:30,845][INFO ][env ] [elk-node1] heap size [1007.3mb], compressed ordinary object pointers [true]
Jul 12 22:03:33 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:33,149][INFO ][node ] [elk-node1] initialized
Jul 12 22:03:33 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:33,149][INFO ][node ] [elk-node1] starting ...
Jul 12 22:03:33 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:33,333][INFO ][transport ] [elk-node1] publish_address {192.168.247.135:9300}, bound_addresses {[::]:9300}
Jul 12 22:03:33 elk-node1 elasticsearch[22400]: [2018-07-12 22:03:33,345][INFO ][discovery ] [elk-node1] hejianlai/iUUTEKhyTxyL78aGtrrBOw
Hint: Some lines were ellipsized, use -l to show in full.
訪問地址：http://192.168.247.135:9200/
安裝ES插件
#統計索引數
[root@elk-node1 ~]# curl -i -XGET 'http://192.168.247.135:9200/_count?pretty' -d '
> "query": {
> "match_all":{}
> }'
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 95
{
"count" : 0,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
}
}
#es插件，收費的不建議使用(這個不安裝)
[root@elk-node1 bin]# /usr/share/elasticsearch/bin/plugin install marvel-agent
#安裝開源的elasticsearch-head插件
[root@elk-node1 bin]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading ...........................................................................................................................................DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
訪問地址：http://192.168.247.135:9200/_plugin/head/
使用POST方法創建查詢
使用GET方法查詢數據
基本查詢
elk-node2配置
[root@elk-node2 elasticsearch]# grep '^[a-z]' /etc/elasticsearch/elasticsearch.yml cluster.name: hejianlai node.name: elk-node2 path.data: /data/es-data path.logs: /var/log/elasticsearch/ bootstrap.memory_lock: true network.host: 0.0.0.0 http.port: 9200 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["192.168.247.135", "192.168.247.133"]
在構建Elasticsearch(ES)多節點集群的時候，通常情況下只需要將elasticsearch.yml中的cluster.name設置成相同即可，ES會自動匹配並構成集群。但是很多時候可能由於不同的節點在不同的網段下，導致無法自動獲取集群。此時可以將啟用單播，顯式指定節點的發現。具體做法是在elasticsearch.yml文件中設置如下兩個參數：
discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["192.168.247.135", "192.168.247.133"]
重啟elk-node1，並開啟elk-node2，訪問：192.168.247.135:9200/_plugin/head/
安裝監控kopf
[root@elk-node1 ~]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf -> Installing lmenezes/elasticsearch-kopf... Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ... Downloading .........................................................................................................................DONE Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ... NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify) Installed kopf into /usr/share/elasticsearch/plugins/kopf
訪問地址：http://192.168.247.135:9200/_plugin/kopf/#!/cluster
四、安裝Logstash(客戶端即需要收集日志的機子，ES節點上都要安裝)
官方文檔地址：https://www.elastic.co/guide/en/logstash/current/index.html
1）下載並安裝GPG Key
[root@elk-node1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2）添加yum倉庫
[root@elk-node1 ~]# vim /etc/yum.repos.d/logstash.repo
[logstash-2.1]
name=Logstash repository for 2.1.x packages
baseurl=http://packages.elastic.co/logstash/2.1/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
3）安裝logstash
[root@elk-node1 ~]# yum install -y logstash
4）測試數據
#簡單的輸入輸出
[root@elk-node1 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
Settings: Default filter workers: 1
Logstash startup completed
hello world
2018-07-13T00:54:34.497Z elk-node1 hello world
hi hejianlai
2018-07-13T00:54:44.453Z elk-node1 hi hejianlai
來賓張家輝
2018-07-13T00:55:35.278Z elk-node1 來賓張家輝
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed
#可以使用rubydebug詳細輸出
[root@elk-node1 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }'
Settings: Default filter workers: 1
Logstash startup completed
mimi
{
"message" => "mimi",
"@version" => "1",
"@timestamp" => "2018-07-13T00:58:59.980Z",
"host" => "elk-node1"
}
#內容寫進elasticsearch中
[root@elk-node1 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch{hosts=>["192.168.247.135"]} }'
Settings: Default filter workers: 1
Logstash startup completed
I love you
1232
hejianlai
渣渣輝
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed
[root@elk-node1 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["192.168.247.135:9200"]} stdout{ codec => rubydebug}}'
Settings: Default filter workers: 1
Logstash startup completed
廣州
{
"message" => "廣州",
"@version" => "1",
"@timestamp" => "2018-07-13T02:17:40.800Z",
"host" => "elk-node1"
}
hehehehehehehe
{
"message" => "hehehehehehehe",
"@version" => "1",
"@timestamp" => "2018-07-13T02:17:49.400Z",
"host" => "elk-node1"
}
logstash日志收集配置文件編寫
#交換式輸入信息
[root@elk-node1 ~]# cat /etc/logstash/conf.d/logstash-01.conf
input { stdin { } }
output {
elasticsearch { hosts => ["192.168.247.135:9200"]}
stdout { codec => rubydebug }
}
執行命令
[root@elk-node1 ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-01.conf
Settings: Default filter workers: 1
Logstash startup completed
l o v e
{
"message" => "l o v e",
"@version" => "1",
"@timestamp" => "2018-07-13T02:37:42.670Z",
"host" => "elk-node1"
}
地久梁朝偉
{
"message" => "地久梁朝偉",
"@version" => "1",
"@timestamp" => "2018-07-13T02:38:20.049Z",
"host" => "elk-node1"
}
#收集系統日志
[root@elk-node1 conf.d]# cat /etc/logstash/conf.d/systemlog.conf
input{
file {
path => "/var/log/messages"
type => "sysstem"
start_position => "beginning"
}
}
output{
elasticsearch{
hosts => ["192.168.247.135:9200"]
index => "systemlog-%{+YYYY.MM.dd}"
}
}
#放在后台執行
[root@elk-node1 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/systemlog.conf &
收集elk錯誤日志配置文件編寫
[root@elk-node1 ~]# cat /etc/logstash/conf.d/elk_log.conf
input {
file {
path => "/var/log/messages"
type => "system"
start_position => "beginning"
}
}
input {
file {
path => "/var/log/elasticsearch/hejianlai.log"
type => "es-error"
start_position => "beginning"
codec => multiline {
pattern => "^\[" //正則匹配[開頭的為一個事件
negate => true
what => "previous"
}
}
}
output {
if [type] == "system"{
elasticsearch {
hosts => ["192.168.247.135:9200"]
index => "systemlog-%{+YYYY.MM.dd}"
}
}
if [type] == "es-error"{
elasticsearch {
hosts => ["192.168.247.135:9200"]
index => "es-error-%{+YYYY.MM.dd}"
}
}
}
#放入后台運行
[root@elk-node1 ~]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/elk_log.conf &
[1] 28574
You have new mail in /var/spool/mail/root
[root@elk-node1 ~]# Settings: Default filter workers: 1
Logstash startup completed
五、安裝Kibana
官方下載地址：https://www.elastic.co/downloads/kibana
官方最新的版本出來了6.3.1太新了，下載后出現很多坑后來就下了4.3.1的·先用着吧~~~
1）kibana的安裝：
[root@elk-node1 local]# cd /usr/local/
[root@elk-node1 local]# wget https://artifacts.elastic.co/downloads/kibana/kibana-4.3.1-linux-x64.tar.gz
[root@elk-node1 local]# tar -xf kibana-4.3.1-linux-x64.tar.gz
[root@elk-node1 local]# ln -s /usr/local/kibana-4.3.1-linux-x64 /usr/local/kibana
[root@elk-node1 local]# cd kibana
[root@elk-node1 kibana]# ls
bin config installedPlugins LICENSE.txt node node_modules optimize package.json README.txt src webpackShims
2）修改配置文件：
[root@elk-node1 kibana]# cd config/
[root@elk-node1 config]# pwd
/usr/local/kibana/config
[root@elk-node1 config]# vim kibana.yml
You have new mail in /var/spool/mail/root
[root@elk-node1 config]# grep -Ev "^#|^$" kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.247.135:9200"
kibana.index: ".kibana"
3)screen是一個全屏窗口管理器，它在幾個進程（通常是交互式shell）之間復用物理終端。每個虛擬終端提供DEC VT100的功能。
[root@elk-node1 local]# yum install -y screen
4)啟動screen命令后運行kibana最后按ctrl+a+d組合鍵讓其在單獨的窗口里運行。
[root@elk-node1 config]# screen
[root@elk-node1 config]# /usr/local/kibana/bin/kibana
log [02:23:34.148] [info][status][plugin:kibana@6.3.1] Status changed from uninitialized to green - Ready
log [02:23:34.213] [info][status][plugin:elasticsearch@6.3.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [02:23:34.216] [info][status][plugin:xpack_main@6.3.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [02:23:34.221] [info][status][plugin:searchprofiler@6.3.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [02:23:34.224] [info][status][plugin:ml@6.3.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
[root@elk-node1 config]# screen -ls
There are screens on:
29696.pts-0.elk-node1 (Detached)
[root@elk-node1 kibana]# /usr/local/kibana/bin/kibana
log [11:25:37.557] [info][status][plugin:kibana] Status changed from uninitialized to green - Ready
log [11:25:37.585] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [11:25:37.600] [info][status][plugin:kbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [11:25:37.602] [info][status][plugin:markdown_vis] Status changed from uninitialized to green - Ready
log [11:25:37.604] [info][status][plugin:metric_vis] Status changed from uninitialized to green - Ready
log [11:25:37.606] [info][status][plugin:spyModes] Status changed from uninitialized to green - Ready
log [11:25:37.608] [info][status][plugin:statusPage] Status changed from uninitialized to green - Ready
log [11:25:37.612] [info][status][plugin:table_vis] Status changed from uninitialized to green - Ready
log [11:25:37.647] [info][listening] Server running at http://0.0.0.0:5601
六、kibana簡單使用
訪問kibana地址：http://192.168.247.135:5601
第一次登錄我們創建一個elk的es-error索引
添加message和path字段
運用搜索欄功能，我們搜soft關鍵字
我們在添加之前寫的systemlog索引
*為正則匹配
好啦~~到此為止ELK日志平台搭建基本搞掂，，，，累得一筆，，后續可以根據需求編寫好需要監控的應用文件添加到kibana上即可。