ELK 日志收集 存儲 分析 展示
20180813 Chenxin
ELK簡單搭建簡單 https://www.cnblogs.com/huangxincheng/p/7918722.html
java日志收集簡單 https://blog.csdn.net/bluetjs/article/details/78770447
ELK搭建-晉級 https://www.cnblogs.com/yuhuLin/p/7018858.html
Elasticsearch-Head https://www.sojson.com/blog/85.html 可以增刪改查elasticsearch的數據,是一個插件(生產環境不建議安裝)
Logstash最佳實踐 https://doc.yonyoucloud.com/doc/logstash-best-practice-cn/get_start/hello_world.html
安裝與配置
logstash: 本地收集日志,並發送給elasticsearch
elasticsearch: 收集logstash發來的日志,並檢索,存儲
kibana: 展示elasticsearch的數據
添加組與用戶
因為elasticsearch進程不允許在root賬號下啟動,故創建單獨賬號.
groupadd elasticsearch
useradd elasticsearch -g elasticsearch -p elasticsearch
echo "xxx" | passwd --stdin elasticsearch
下載軟件(官網下載 https://www.elastic.co/cn/ ),安裝軟件
cd /opt/
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz
tar xzvf elasticsearch-6.3.2.tar.gz
tar xzvf logstash-6.3.2.tar.gz
tar xzvf kibana-6.3.2-linux-x86_64.tar.gz
mv elasticsearch-6.3.2 /usr/local/elasticsearch
mv logstash-6.3.2 /usr/local/logstash
mv kibana-6.3.2-linux-x86_64 /usr/local/kibana
logstash 默認開啟9600
配置
cd logstash/config
vim logstash.conf (新建文件) 自動配置請見最下方腳本
[elasticsearch@MiWiFi-R3P-srv config]$ cat logstash.conf
見最下方logstash.conf內容
logstash在目錄中收集所有日志(單層目錄/多層目錄)
固定文件夾
input{
file {
path => "/usr/local/log_test/.log"
start_position => "beginning"
}
}
output {
stdout { } #標准輸出,會輸出到屏幕
}
多級目錄
創建配置文件，然后設定格式為//.log的方式，其中//表示的是一個目錄，多級目錄，需要使用多個//
input{
file {
path => "/usr/local/log_test//*.log"
start_position => "beginning"
}
}
其他配置文件說明
logstash.yml 文件里可以配置logstash的綁定IP地址與端口
jvm.options 配置啟動logstash的jvm參數(默認1GB)
startup.options 只適用於yum安裝的方式,這里無需關注
log4j2.properties 為log for java 第二版,是apache退出的一個標准化日志插件,這里是配置文件.此處無需關注.
啟動logstash
./bin/logstash -f config/logstash.conf
elasticsearch 默認開啟9200 9300
配置與啟動elasticsearch(不可以在root身份下啟動)
配置采用默認,不做修改
cd elasticsearch
cd bin/
nohup ./elasticsearch > nohup.out 2>&1 &
其他配置文件說明
elasticsearch.yml 配置綁定IP和端口,數據存放路徑.生成環境需要修改此配置.
訪問 http://192.168.31.129:9200/ 會顯示一個json格式的節點信息
報錯處理
cat /etc/sysctl.conf
vm.max_map_count=262144
sysctl -p
cat /etc/security/limits.conf
- soft nproc 65536
- hard nproc 65536
- soft nofile 65536
- hard nofile 65536
退出當前終端,重新進入.
bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks #當綁定非localhost時,提示.(不處理也無妨).
修改配置文件，在配置文件添加一項參數.
vim /etc/elasticsearch/elasticsearch.yml
bootstrap.system_call_filter: false
kibana 默認開啟5601
配置與啟動kibana
vim kibana/config/kibana.yml
cat kibana.yml |grep -v "#"
server.host: 0.0.0.0
elasticsearch.url: "http://localhost:9200"
cd bin/
nohup ./kibana >nohup.out 2>&1 &
kibana平台的使用
logstash會將配置好的日志發給ES,kibana從ES里取數據.
打開kibana頁面,http://192.168.31.129:5601 ,進入management,先建立對應的index.然后進入discover,就可以展示對應index的內容了.
支持一些正則匹配查詢等(比如message : * ),以及時間選擇(比如最后15分鍾的)
關於登陸認證方式的說明
收費認證方式
ELK默認用戶身份認證屬於收費功能(5.0以后版本)
若購買了lisence后,執行
修改配置文件 kibana.yml
elasticsearch.username: "elastic"
elasticsearch.password: "xxx"
重啟kibana后,執行以下curl,對elasticsearch的密碼進行更新(默認用戶名為elastic,默認密碼為changeme)
curl -H "Content-Type: application/json" -XPUT -u elastic '192.168.31.129:9200/_xpack/security/user/kibana/_password' -d '{
"password" : "xxx"
}'
若是免費使用者,報錯如下:
{"error":{"root_cause":[{"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"}],"type":"security_exception","reason":"current license is non-compliant for [security]","license.expired.feature":"security"},"status":403}
免費認證方式
可以使用nginx反向代理的方式.nginx做前端,后端掛elasticsearch和kibana,利用nginx的認證來做訪問控制(如果外網知道后端機器IP,應該也是可以訪問的吧).
https://www.cnblogs.com/configure/p/7607302.html
https://birdben.github.io/2017/02/08/Kibana/
安裝Nginx：
yum -y install nginx
安裝Apache密碼生產工具：
yum install -y httpd-tools
生成密碼文件：
mkdir -p /etc/nginx/passwd; htpasswd -c -b /etc/nginx/passwd/kibana.passwd yunwei xxx
配置Nginx：
vim /etc/nginx/nginx.conf 增加如下內容:
server {
listen 10.0.0.30:18080; #對外開放端口
auth_basic "Kibana Auth";
auth_basic_user_file /etc/nginx/passwd/kibana.passwd;
location / {
proxy_pass http://10.0.0.30:25601;
proxy_redirect off;
}
}
修改Kibana配置文件：
vim /usr/local/elk/kibana/config/kibana.yml
server.port: 25601
server.host: "10.0.0.30"
elasticsearch.url: "http://10.0.0.30:9200"
ES元數據說明
查看ES節點信息 http://192.168.31.129:9200/
查看日志文件 http://192.168.31.129:9200/_cat/indices
ES定期清理數據
先查看日志文件名,
curl http://192.168.31.129:9200/_cat/indices
curl http://10.0.0.201:9200/_cat/indices
執行(注意date的格式)
curl -X DELETE http://192.168.31.129:9200/systemdate +%Y.%m.%d -d "-0 days" #刪除當天system打頭的
返回值 {"acknowledged":true}
curl -X DELETE http://192.168.31.129:9200/systemdate +%Y.%m.%d -d "-1 days" #刪除昨天
{"acknowledged":true}
eg：刪除19號logstash的所有數據
curl -X DELETE 'http://127.0.0.1:9200/logstash-2017.06.19'
eg：刪除2個月之前的數據
_last_data=date -d '-2 months' +%Y.%m
curl -X DELETE 'http://127.0.0.1:9200/-'${_last_data}'-'
刪除指定月份:
curl -X DELETE 'http://10.0.0.201:9200/-2018.09*'
刪除第30天前的那天的日志(crontab)
10 18 * * * /usr/bin/curl -X DELETE http://10.0.0.201:9200/-date +%Y.%m.%d -d "-30 days" #在cron里好像生效有問題
10 17 * * * /usr/bin/curl -X DELETE "http://10.0.0.201:9200/-date +%Y.%m.%d -d \"-15 days\"*" #后來加的,等待一段時間再驗證是否在cron里生效
能否以IP或其他方式展示,便於區分是哪個機器
是不是需要在logstash/config/logstash.conf 文件中用 type => "app1" 這種方式來確認呢?
以下是關於自動化的優化,可以不看
配置文件或腳本文件備份(在測試服上的實現)2018-0814
logstash的配置文件:
es配置文件:
另外,將jvm配置文件里的jvm參數調小
kibana配置文件:
ES上nginx的配置文件:
3個服務啟動或關停腳本文件:
ES和kibana機器上配置(2個服務在同一台機器)
在root下啟動es
sudo -u elasticsearch -E /usr/local/elasticsearch/bin/elasticsearch.sh --start #這里 -E 指的是不是有sudoer里的環境變量(最安全的一種環境變量),而使用root默認的環境變量.
加入到/etc/rc.local 文件
則需要給出javahome.因為rc.local運行在操作系統完全引導成功但是尚未啟動login shell之前，所以我們配置在/etc/profiles或bashrc里的環境變量並未得到執行，因此在rc.local執行階段看不到任何環境變量.將以下內容加入rc.local:
export JAVA_HOME=/usr/local/jvm;sudo -u elasticsearch -E /usr/local/elasticsearch/bin/elasticsearch.sh --start
/usr/local/kibana/bin/kibana.sh --start
service nginx start
為了下載文件方便,可以搭建一個nginx下載服
server {
listen 18081; #端口
server_name 10.0.0.201; #服務名
charset utf-8; # 避免中文亂碼
root /opt/download; #顯示的根索引目錄，注意這里要改成你自己的，目錄要存在
location / {
autoindex on; #開啟索引功能
autoindex_exact_size off; # 關閉計算文件確切大小（單位bytes），只顯示大概大小（單位kb、mb、gb）
autoindex_localtime on; # 顯示本機時間而非 GMT 時間
}
}
向所有游戲服添加logstash服務
下載/解壓/放置到/usr/local/;替換jvm/替換yml/替換logstash.conf/添加logstash.sh/添加/etc/rc.local
cd /opt/
wget http://13.251.64.203:18081/logstash-6.3.2.tar.gz
tar xzvf logstash-6.3.2.tar.gz
mv logstash-6.3.2 /usr/local/logstash
cd /usr/local/logstash/config/
mv jvm.options logstash.yml /home/admin/
wget http://13.251.64.203:18081/jvm.options
wget http://13.251.64.203:18081/logstash.yml
wget http://13.251.64.203:18081/logstash.conf
cd /usr/local/logstash/bin/;
wget http://13.251.64.203:18081/logstash.sh
chmod 755 logstash.sh
echo "/usr/local/logstash/bin/logstash.sh --start" >> /etc/rc.local
cd /usr/local/logstash/config/
修改logstash.conf文件內容
到kibana里去添加對應index
錯誤處理
因當初ES的核心機器配置比較低,CPU剩余額度經常為0.
最后造成logstash的信息發送不暢(ES接收那里有問題,部分機器被阻塞,個別機器的日志還可以發到ES里).
通過升級ES機器的配置到t3.mediam,以及磁盤升級到50GB.
然后,到各個logstash的主機 上重啟java進程,予以解決.
[root@ip-10-30-0-100 ~]# /usr/local/logstash/bin/logstash.sh --stop
[root@ip-10-30-0-100 ~]# vim /usr/local/logstash/logs/logstash-plain.log
重啟前報錯日志:
[2018-09-25T11:53:24,302][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow
delete (api)];"})
[2018-09-25T11:53:24,302][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2018-09-25T11:53:24,302][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow
delete (api)];"})
[2018-09-25T11:53:24,302][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
重啟后的正常信息:
[2018-09-25T11:54:01,275][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-09-25T11:54:01,749][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2018-09-25T11:54:04,007][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-09-25T11:54:04,424][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.0.0.201:9200/]}}
[2018-09-25T11:54:04,433][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.0.0.201:9200/, :path=>"/"}
[2018-09-25T11:54:04,749][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.0.0.201:9200/"}
[2018-09-25T11:54:04,872][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-09-25T11:54:04,875][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-09-25T11:54:04,901][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.0.0.201:9200"]}
[2018-09-25T11:54:04,924][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-09-25T11:54:04,926][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.0.0.201:9200/]}}