本文僅供自己學習，不合適轉載。
這是兩篇文章的第一部分。
1. 安裝 ELK
1.1 安裝 ElasticSearch
在海航雲上創建一個 Ubutu 16.4 虛機，2核4GB內存。
（1）執行以下命令，更新系統
sudo apt-get update -y
sudo apt-get upgrade -y
（2）安裝 Java
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update sudo apt-get install oracle-java8-installer -y
（3）安裝ES
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.2.deb sudo dpkg -i elasticsearch-5.2.2.deb
修改/etc/elasticsearch/elasticsearch.yml:
- 將 network.host 修改為本機 ip 即 192.168.10.102
- 將 http.port 修改為 9200
- 將 cluster.name 修改為 elk-test
重啟 ES並檢查服務狀態：
root@elk:/home/ubuntu# service elasticsearch restart root@elk:/home/ubuntu# service elasticsearch status ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enab Active: active (running) since Sat 2017-09-30 11:23:17 CST; 3s ago Docs: http://www.elastic.co Process: 3861 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code= Main PID: 3864 (java) Tasks: 15 Memory: 2.1G CPU: 4.511s
1.2 安裝 Nginx 和 Logstash
創建另一台虛機，安裝 Nginx 和 Logstash。
1.2.1 安裝 Nginx
apt-get install nginx
Ubuntu安裝之后的文件結構大致為：
- 所有的配置文件都在/etc/nginx下，並且每個虛擬主機已經安排在了/etc/nginx/sites-available下
- 程序文件在/usr/sbin/nginx
- 日志放在了/var/log/nginx中
- 並已經在/etc/init.d/下創建了啟動腳本nginx
- 默認的虛擬主機的目錄設置在了/var/www/nginx-default (有的版本 默認的虛擬主機的目錄設置在了/var/www, 請參考/etc/nginx/sites-available里的配置)
啟動並查看服務狀態：
root@elk:/home/ubuntu# /etc/init.d/nginx start [ ok ] Starting nginx (via systemctl): nginx.service. root@elk:/home/ubuntu# /etc/init.d/nginx status ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2017-09-30 11:40:59 CST; 1min 8s ago Main PID: 4320 (nginx) CGroup: /system.slice/nginx.service ├─4320 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─4321 nginx: worker process └─4322 nginx: worker process Sep 30 11:40:59 elk systemd[1]: Starting A high performance web server and a reverse pro...r... Sep 30 11:40:59 elk systemd[1]: Started A high performance web server and a reverse prox...ver. Sep 30 11:42:06 elk systemd[1]: Started A high performance web server and a reverse prox...ver. Hint: Some lines were ellipsized, use -l to show in full.
為了測試起見，將端口修改為 88.修改文件/etc/nginx/sites-available/default，並重啟 Nginx 服務：
server { listen 88 default_server; listen [::]:88 default_server;
鑒於該服務器沒有設置公網IP，在其路由器上設置端口轉發規則，使得可以通過路由器的EIP的88端口訪問到它上面的Nginx服務：
在瀏覽器上測試，Nginx 可用：
1.2.2 安裝和配置 Logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.2.2.tar.gz tar zxvf logstash-5.2.2.tar.gz ln -s logstash-5.2.2 logstash
創建文件 nginxlog2es.conf，內容如下。它會將 Nginx 的日志文件 /var/log/nginx/access.log_json 中的日志發到 ES 服務器 192.168.10.102:9200：
input { file { path => "/var/log/nginx/access.log_json" codec => "json" } } filter { mutate { split => [ "upstreamtime", "," ] } mutate { convert => [ "upstreamtime", "float" ] } } output { stdout { codec => rubydebug } elasticsearch { hosts => ["192.168.10.102:9200"] index => "logstash-%{type}-%{+YYYY.MM.dd}" document_type => "%{type}" flush_size => 20000 idle_flush_time => 10 sniffing => true template_overwrite => true } }
修改 /etc/nginx/nginx.conf，添加：
## # Logging Settings ## log_format json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"url":"$uri",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"agent":"$http_user_agent",' '"status":"$status"}'; access_log /var/log/nginx/access.log_json json;
重啟 Nginx 服務，在瀏覽器上刷新頁面，查看 Nginx 日志，
{"@timestamp":"2017-09-30T12:44:19+08:00","host":"192.168.10.104","clientip":"140.206.84.10","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"120.132.124.103","url":"/index.nginx-debian.html","xff":"-","referer":"-","agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36","status":"304"}
啟動logstash，
nohup logstash/bin/logstash -f nginxlog2es.conf > /tmp/logstash.log 2>&1 &
刷新 Nginx 頁面，能看到 logstash 收集到的 Nginx 日志：
{ "referer" => "-", "agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36", "http_host" => "120.132.124.103", "url" => "/index.nginx-debian.html", "path" => "/var/log/nginx/access.log_json", "upstreamhost" => "-", "@timestamp" => 2017-09-30T04:48:23.000Z, "size" => 0, "clientip" => "140.206.84.10", "host" => "192.168.10.104", "@version" => "1", "responsetime" => 0.0, "xff" => "-", "upstreamtime" => [ [0] 0.0 ], "status" => "304" }
1.3 安裝 Kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.2.2-linux-x86_64.tar.gz ln -s kibana-5.2.2-linux-x86_64 kibana
修改配置文件 kibana/config/kibana.yml，
server.host: "192.168.10.102" elasticsearch.url: "http://192.168.10.102:9200"
啟動 kibana，
nohup kibana/bin/kibana > /tmp/kibana.log 2>&1 &
在瀏覽器里面輸入 http://120.132.124.103:5601/ 就可以打開 kibana 頁面了。可以看到Nginx 的日志：
1.4 小結
從上面的步驟可以看出，ELK 的結構相對簡單：
- Logstack 負責收集日志，並推送到 ES 中
- ES 負責存儲
- Kibana 負責界面展示
ELK 的總體架構如下：
但是，這種架構有不少問題，其中問題之一是處理能力問題。bol.com 公司有如下的ELK架構演進路線：
（1）初始架構（2013年）
問題是單實例的 logstash 有性能瓶頸。
（2）使用 redis 緩存以及多個 logstash 實例（2014年）
使用 redis 作為消息緩存，使用多實例 Logstash 增加處理性能。
參考鏈接：
https://devops.profitbricks.com/tutorials/install-and-configure-apache-kafka-on-ubuntu-1604-1/
http://www.cnblogs.com/xiaoqi/p/elk-part1.html
https://www.slideshare.net/TinLe1/elk-atlinked-in
https://www.slideshare.net/renzotoma39/scaling-an-elk-stack-at-bolcom-39412550
https://www.elastic.co/blog/logstash-kafka-intro
https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part2
https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1