一、ELK概述
1、需求背景
- 業務發展越來越龐大、服務器越來越多
- 各種訪問日志、應用日志、錯誤日志量越來越多
- 開發人員排查問題，需要服務器上查日志，不方便
- 運營人員需要一些數據，需要運維到服務器上分析日志
2、ELK介紹
ELK Stack包含：ElasticSearch、Logstash、Kibana。（ELK Stack 5.0版本以后-->Elastic Stack == ELK Stack+Beats）
ElasticSearch是一個搜索引擎，用來搜索、分析、存儲日志。它是分布式的，也就是說可以橫向擴容，可以自動發現，索引自動分片，總之很強大。
Logstash用來采集日志，把日志解析為Json格式交給ElasticSearch。
Kibana是一個數據可視化組件，把處理后的結果通過WEB界面展示。
Beats是一個輕量級日志采集器，其實Beats家族有5個成員。（早起的Logstash對性能資源消耗比較高，Beats性能和消耗可以忽略不計）
X-pach對Elastic Stack提供了安全、警報、監控、報表、圖標於一身的擴展包，收費。
官網：https://www.elastic.co/cn/
中文文檔：https://www.elastic.co/guide/cn/elasticsearch/guide/current/index.html
3、ELK架構
4、流程說明：
- 所有業務服務器安裝Filebeat進行日志采集
- Filebeat將日志采集至Logstash進行過濾和索引
- ElasticSearch索引分析
- Kibana圖形展示
二、ELK安裝
1、環境
2、安裝配置
（1）、安裝ElasticSearch
#安裝 wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.rpm yum localinstall elasticsearch-6.2.3.rpm #配置 vim /etc/elasticsearch/elasticsearch.yml network.host: 0.0.0.0 http.port: 9200 vim /etc/sysconfig/elasticsearch JAVA_HOME=/usr/local/jdk1.8.0_131 #啟動 systemctl daemon-reload systemctl enable elasticsearch.service systemctl start elasticsearch.service
（2）、安裝Kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-x86_64.rpm yum localinstall kibana-6.2.3-x86_64.rpm -y vim /etc/kibana/kibana.yml server.port: 5601 server.host: "0.0.0.0" elasticsearch.url: "http://localhost:9200" logging.dest: /var/log/kibana.log touch /var/log/kibana.log ；chmod 777 /var/log/kibana.log systemctl enable kibana systemctl start kibana
（3）、安裝Logstash
yum localinstall logstash-6.2.3.rpm #Logstash不支持JAVA9 chown -R logstash:root /var/log/logstash /var/lib/logstash vim /etc/logstash/logstash.yml http.host: "0.0.0.0"
（4）、安裝filebeat
yum localinstall filebeat-6.2.3-x86_64.rpm -y
#logstash和filebeat，下章會配置啟動
3、Kibana漢化
git clone https://github.com/anbai-inc/Kibana_Hanization.git cd Kibana_Hanization/ python main.py /usr/share/kibana/ systemctl restart kibana
4、坑
（1）：JAVA環境丟失
elasticsearch: could not find java
解決：
vim /etc/sysconfig/elasticsearch JAVA_HOME=/usr/local/jdk1.8.0_131
（2）：缺少jar包，可能安裝包有問題
error: unpacking of archive failed on file /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.4/vendor/jar-dependencies/runtime-jars/log4j-api-2.8.2.jar;5ab9a80b: cpio: read
解決：
yum install logstash https://www.elastic.co/guide/en/logstash/6.2/installing-logstash.html #官網yum安裝文檔
（3）：JAVA環境丟失
/usr/share/logstash/vendor/jruby/bin/jruby: line 401: /usr/bin/java: No such file or directory
解決：
ln -s /usr/local/jdk1.8.0_131/bin/java /usr/bin/java
（4）：Logstash無法啟動，或沒有日志輸出
[2018-03-27T13:27:33,839][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in `block in validate_all'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:264:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:67:in `<main>'"]}
[2018-03-27T13:27:33,843][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit
解決：
chown -R logstash /var/log/logstash /var/lib/logstash
（5）：Logstash沒有centos6啟動腳本
解決：
/usr/share/logstash/bin/system-install /etc/logstash/startup.options sysv