部署使用elk

1.配置8台虛擬機 es[1:5],kibana,logstash,web   ip 192.168.1.61-68

2.開始配置ip和主機名

3.用ansible部署elasticsearch,並能訪問網頁，以下是ansible部署的yml代碼

---
- hosts: es
  remote_user: root
  tasks:
    - copy:
        src: local.repo
        dest: /etc/yum.repos.d/local.repo
        owner: root
        group: root
        mode: 0644
    - name: install elasticsearch
      yum:
        name: java-1.8.0-openjdk,elasticsearch
        state: installed
    - template:
        src: elasticsearch.yml
        dest: /etc/elasticsearch/elasticsearch.yml
        owner: root
        group: root
        mode: 0644
      notify: reload elasticsearch
      tags: esconf
    - service:
        name: elasticsearch
        enabled: yes
  handlers:
    - name: reload elasticsearch
      service:
        name: elasticsearch
        state: restarted

4.以下是當前要改的 hosts

192.168.1.61 es1
192.168.1.62 es2
192.168.1.63 es3
192.168.1.64 es4
192.168.1.65 es5
192.168.1.66 kibana
192.168.1.67 logstash

5.yum源有相應的軟件包和依賴包即可

6.以下是當前要改的 elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: nsd1810
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: {{ansible_hostname}}
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
# path.data: /path/to/data
#
# Path to log files:
#
# path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
# bootstrap.mlockall: true
#
# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
# available on the system and that the owner of the process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
# http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
# discovery.zen.minimum_master_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
# gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
# node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
# action.destructive_requires_name: true

7.elasticsearch搭建完成，可以用http://192.168.1.61：9200 訪問檢驗

8.部署插件

插件裝在哪一台機器上，只能在哪台機器上使用（這里安裝在es5機器上面）

1）使用遠程 uri 路徑可以直接安裝

es5 ~]# cd /usr/share/elasticsearch/bin

es5 bin]# ./plugin install \

ftp://192.168.1.254/elk/elasticsearch-head-master.zip        //安裝head插件

es5 bin]# ./plugin install \

ftp://192.168.1.254/elk/elasticsearch-kopf-master.zip        //安裝kopf插件

es5 bin]# [root@es5 bin]# ./plugin install \

ftp://192.168.1.254/elk/bigdesk-master.zip

//安裝bigdesk插件    

es5 bin]# ./plugin list        //查看安裝的插件

Installed plugins in /usr/share/elasticsearch/plugins:

1. - head
2. - kopf
3. - bigdesk

2）訪問head插件

1. [root@room9pc01 ~]# firefox http://192.168.1.65:9200/_plugin/head

2. ）訪問kopf插件 

   1. [root@room9pc01 ~]# http://192.168.1.65:9200/_plugin/kopf

4）訪問bigdesk插件

1. [root@room9pc01 ~]# http://192.168.1.65:9200/_plugin/bigdesk

 

9：安裝kibana

1）在另一台主機，配置ip為192.168.1.66，配置yum源，更改主機名

2）安裝kibana

1. [root@kibana ~]# yum -y install kibana
2. [root@kibana ~]# rpm -qc kibana
3. /opt/kibana/config/kibana.yml
4. [root@kibana ~]# vim /opt/kibana/config/kibana.yml
5. 2 server.port: 5601        
6. //若把端口改為80，可以成功啟動kibana，但ss時沒有端口，沒有監聽80端口，服務里面寫死了，不能用80端口，只能是5601這個端口
7. 5 server.host: "0.0.0.0"        //服務器監聽地址
8. 15 elasticsearch.url: http://192.168.1.61:9200    
9. //聲明地址，從哪里查，集群里面隨便選一個
10. 23 kibana.index: ".kibana"    //kibana自己創建的索引
11. 26 kibana.defaultAppId: "discover"    //打開kibana頁面時，默認打開的頁面discover
12. 53 elasticsearch.pingTimeout: 1500    //ping檢測超時時間
13. 57 elasticsearch.requestTimeout: 30000    //請求超時
14. 64 elasticsearch.startupTimeout: 5000    //啟動超時
15. [root@kibana ~]# systemctl restart kibana
16. [root@kibana ~]# systemctl enable kibana
17. Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /usr/lib/systemd/system/kibana.service.
18. [root@kibana ~]# ss -antup | grep 5601 //查看監聽端口

3）瀏覽器訪問kibana，

1. [root@kibana ~]# firefox 192.168.1.66:5601

4）點擊Status，查看是否安裝成功，全部是綠色的對鈎,說明安裝成功

5）用head插件訪問會有.kibana的索引信息，如圖所示：

[root@es5 ~]# firefox http://192.168.1.65:9200/_plugin/head/   #插件安裝再es5上

 

 

 10.安裝jdk 和 logstash

1. [root@logstash ~]# yum -y install java-1.8.0-openjdk
2. [root@logstash ~]# yum -y install logstash
3. [root@logstash ~]# java -version

11.自己配置logstash.conf 文件，參考代碼如下，input,output,filter 可以參考elastic官網https://www.elastic.co/guide/en/logstash/current/index.html

      yes的為必須選項，下面有id和example（與id在一起的地方）

      注意這個代碼包含很多測試輸入輸出還有調用宏（

1. root@logstash ~]# cd /opt/logstash/vendor/bundle/ \
2. jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/
3. [root@logstash ~]# vim grok-patterns //查找COMBINEDAPACHELOG
4. COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

）查找宏路徑，寫入數據到/tmp/a.log，以及echo > /dev/tcp/192.168.1.67 /8888 ,也可以用syslog部分，在/etc/rsyslog.conf在加上

1. local0.info @192.168.1.67:514
2. //寫一個@或兩個@@都可以，一個@代表udp，兩個@@代表tcp

logger -p local0.info -t nds "001 elk" 然后上述操作都能被logstash，收集並處理，看到分析結果。調用宏用的是grok（因為寫的正則有點復雜和頭疼就用宏定義了）

input{
    stdin{ codec => "json"}
    beats{
      port => 5044
 }
file {
    path => ["/tmp/a.log","/tmp/b.log"]
    sincedb_path => "/var/lib/logstash/sincedb"
    start_position => "beginning"
    type => "testlog"
   }
 tcp {
  host => "0.0.0.0"
  port => "8888"
  type => "tcplog"
  }
 udp {
     host => "0.0.0.0"
     port => "8888"
     type => "udplog"
}

syslog {
           type => "syslog"
}

}
filter{
 grok {
  match => ["message", "%{COMBINEDAPACHELOG}"]
  }}

output{
    stdout{
    codec => "rubydebug"
   }
  elasticsearch {
    hosts => ["es1", "es2", "es3"]
    index => "weblog"
    flush_size => 2000
    idle_flush_time => 10
}
}

12.

步驟二：安裝Apache服務，用filebeat收集Apache服務器的日志，存入elasticsearch

1）在之前安裝了Apache的主機上面安裝filebeat

1. [root@web~]# yum -y install filebeat
2. [root@web~]# vim/etc/filebeat/filebeat.yml
3. paths:
4.     - /var/log/httpd/access_log //日志的路徑，短橫線加空格代表yml格式
5. document_type: apachelog //文檔類型
6. elasticsearch:        //加上注釋
7. hosts: ["localhost:9200"]                //加上注釋
8. logstash:                    //去掉注釋
9. hosts: ["192.168.1.67:5044"]     //去掉注釋,logstash那台主機的ip
10. [root@web ~]# systemctl start filebeat

13.然后在/etc/logstash.conf中寫入beats port相關配置（filebeat是給logstash的小插件，使得各服務器不用裝logstash就可以自動把數據發送給logstash服務器）

14都配置好后，就可以在logstash執行/opt/logstash/bin/logstash -f /etc/logstash/logstash.conf 

  來進行解析數據了

並且可以用netstat -antup | grep 5044 查到 兩個5044的端口 （用netstat -lntup | grep 5044只有一個）

15.訪問web服務器后，logstash能收集到數據，而elastic的網頁可以看到weblog，在.kibana的網頁輸入weblog可以看到訪問的條形圖

 weblog圖類似下面，名字不一樣

 

.kibana圖如下

 

不能畫出餅圖，數據太少，訪問網頁要多刷新幾次，數據太少，可能看不到圖片。 

 

上述的logstash.conf文件是用來解析httpd的log日志的如果是nginx或者是其他服務就要調用他的宏了，可以if來判斷，如以下代碼，提供一些思路

input{
    stdin{ codec => "json"}
    beats{
      port => 5044
 }
file {
    path => ["/tmp/a.log","/tmp/b.log"]
    sincedb_path => "/var/lib/logstash/sincedb"
    start_position => "beginning"
    type => "testlog"
   }
 tcp {
  host => "0.0.0.0"
  port => "8888"
  type => "tcplog"
  }
 udp {
     host => "0.0.0.0"
     port => "8888"
     type => "udplog"
}

syslog {
           type => "syslog"
}

}
filter{
 if [type] == "httplog"{
 grok {
  match => ["message", "%{COMBINEDAPACHELOG}"]
  }}
}

output{
    stdout{
    codec => "rubydebug"
   }
if [type] == "httplog"{
  elasticsearch {
    hosts => ["es1", "es2", "es3"]
    index => "weblog"
    flush_size => 2000
    idle_flush_time => 10
}
}}

 elk 整體思路是 客戶通過訪問web，然后web服務器的filebeat把數據發送給logstash，logstash解析后發送各elasticsearch檢索存儲，kibana上可以從elastic提取數據並有用web和圖形展示出來