Elastic Stack配置和使用

Elastic Stack是一個開源的解決方案，可以收集各種類型，各種格式的源數據，同時提供數據搜索，分析和可視化的展示
＃ 通用搜索引擎
索引組件：獲取數據-->建立文檔-->文檔分析-->文檔索引（倒排索引），如Lucene
搜索組件：用戶搜索接口-->建立查詢（將用戶鍵入的信息轉換為可處理的查詢對象）-->搜索查詢-->展現結果，如Solr，ElasticSearch

各組件介紹

Lucene Core

Apache LuceneTM is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.

Solr

SolrTM is a high performance search server built using Lucene Core, with XML/HTTP and JSON/Python/Ruby APIs, hit highlighting, faceted search, caching, replication, and a web admin interface.

ElasticSearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

Elastic Stack組件

架構圖

以index為界線，下半部分的主要功能完成索引，上半部分完成搜索。

ElasticSearch

功能如上。

Logstash

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” (Ours is Elasticsearch, naturally.)

Beats

Filebeat：Log Files
Metricbeat：Metrics
Packetbeat：Network Data
Winlogbeat：Windows Event Logs
Heartbeat：Uptime Monitoring        

Kibana

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack, so you can do anything from learning why you're getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers.
小結：對應於架構圖，Logstash作為agent，和Beats組件同時可以完成獲取內容，logstash比較重量級在消耗系統資源方面，所以實際中用Beats較多；Logstash作為服務器端，可以完成創建文檔；ElasticSearch負責索引，同時提供搜索功能；Kibana提供可視的圖形管理和展示界面。

ElasticSearch

配置文件：

/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/jvm.options
/etc/elasticsearch/log4j2.properties
Unit File：elasticsearch.service

程序文件：

/usr/share/elasticsearch/bin/elasticsearch
/usr/share/elasticsearch/bin/elasticsearch-keystore：
/usr/share/elasticsearch/bin/elasticsearch-plugin：管理插件程序

搜索服務：9200/tcp

集群服務：9300/tcp

集群配置和使用

工作邏輯：所有節點選舉一個主節點，負責管理整個集群的狀態(green/yellow/red)，以及各shards的分布方式；
### ELS構成
    集群：一個或多個節點的集合；
    節點：運行的單個els實例；
    索引：切成多個獨立的shard；（以Lucene的視角，每個shard即為一個獨立而完整的索引）
###　集群配置：
    1. 到官網下載elasticsearch-5.6.10.rpm
    2. 准備三台服務器
    3. yum install elasticsearch-5.6.10.rpm
    4. elasticsearch.yml配置文件：
		cluster.name: myels (三個節點上的cluster.name一致)
		node.name: node1 （各個不同的節點更改為自已的名字，分別為node2，node3）
                    #node.attr.rack: r1 （機櫃感知配置，需要定義好服務器處於的機架）
		path.data: /data/els/data （ chown elasticsearch.elasticsearch）
		path.logs: /data/els/logs
		network.host: 0.0.0.0
		http.port: 9200
		discovery.zen.ping.unicast.hosts: ["node1", "node2", "node3"]
		discovery.zen.minimum_master_nodes: 2	
    5. #  curl -XGET 'http://node01:9200/'
        {
              "name" : "node01",
              "cluster_name" : "myels-evan",
              "cluster_uuid" : "w_N3c2aXQnWBEe1UFrIQ8A",
              "version" : {
                "number" : "5.6.10",
                "build_hash" : "b727a60",
                "build_date" : "2018-06-06T15:48:34.860Z",
                "build_snapshot" : false,
                "lucene_version" : "6.6.1"
          },
          "tagline" : "You Know, for Search"
            }
        # 檢查集群狀態 curl -XGET 'http://node01:9200/_cluster/health?pretty=true'
            {
              "cluster_name" : "myels-evan",
              "status" : "green",
              "timed_out" : false,
              "number_of_nodes" : 3,
              "number_of_data_nodes" : 3,
              "active_primary_shards" : 0,
              "active_shards" : 0,
              "relocating_shards" : 0,
              "initializing_shards" : 0,
              "unassigned_shards" : 0,
              "delayed_unassigned_shards" : 0,
              "number_of_pending_tasks" : 0,
              "number_of_in_flight_fetch" : 0,
              "task_max_waiting_in_queue_millis" : 0,
              "active_shards_percent_as_number" : 100.0
            }
### 集群使用：（RTESful接口風格）
    RESTful API: CRUD(Create, Read, Update, Delete)
	curl  -X<VERB> '<PROTOCOL>://<HOST>:<PORT>/<PATH>?<QUERY_STRING>' -d '<BODY>'
		<BODY>：json格式的請求主體；
                    <VERB>：GET，POST，PUT，DELETE
                    特殊PATH：/_cat, /_search, /_cluster （檢查狀態）
                    <PATH> /index_name/type/Document_ID/
                    curl  -XPUT 創建文檔
                    文檔語法：{"key1": "value1", "key2": value, ...}
    實例：
       # curl -XGET 'http://node02:9200/_cluster/stats?pretty=true' （檢查集群的詳細信息）
       # curl -XPUT http://node02:9200/myindex （創建索引）
       # curl -XGET http://node02:9200/_cat/indices（檢查索引）
       # curl -XDELETE http://node02:9200/myindex （刪除索引）
       # curl -XGET http://node02:9200/_cat/shards
       # curl -XPUT http://node02:9200/myindex/students/1?pretty -d ‘{"name":"Liang Jindong","age":30,"major":"good english"}’ 創建文檔
       #  curl -XGET http://node02:9200/_search? （可以在不同的路徑上搜索來定義搜索范圍）                

Logstash日志收集工具

Logstash可以同時作為agent和server來從指定的位置（如file，mysql, redis）抽取數據，並進行文檔化，然后發送給ElasticSearch，也可以只作為服務端，配合輕量化的filebeat抽取數據，在此應用中，logstash只作日志文檔化，並發送給ElasticSearch。以下是幾種應用場景的架構圖：

配置文件有三部分組成，分別定義輸入，過濾，輸出，由不同類型的插件支持。（注意任何定義在conf.d下的文件都會作為配置文件加載，不同於httpd下必須以.conf結尾才能作為配置文件)

	input {
		...
	}
	
	filter{
		...
	}
	
	output {
		...
	}

Logstash安裝使用

1. 下載logstash rpm
2. 安裝JAVA SDK
3. rpm -ivh logstash-5.6.10.rpm
4. vim /etc/profile.d/logstash.sh
    export PATH=/usr/share/logstash/bin/logstash:$PATH

Logstash配置示例1（標准輸入和輸出）

#vim /etc/logstash/conf.d/test.conf
            input {
                    stdin{}
            }

            output {
                    stdout {
                        codec => rubydebug
            }
            }
# logstash -f test.conf -t (測試語法)
# logstash -f test.conf  （以test.conf配置文件啟動）
# 生成新的文件test.conf，加上filter
    input {
    stdin{}
        }

    filter {
        grok {
                match => { "message" => "%{NUMBER:duration} %{IP:client}" }
        }
    }
    output {
            stdout {
                    codec => rubydebug
            }
    }
    #logstash -f test.conf (啟動)
    # 輸入32 1.2.3.4 
    # 輸出為如下，模式中的NUMBER和IP用key duration和client把數字和IP進行拆分，輸出新的值。
        The stdin plugin is now waiting for input:
        {
          "@version" => "1",
             "host" => "kvm",
              "duration" => "32",
                "client" => "1.2.3.4",
                "@timestamp" => 2018-08-01T06:15:23.613Z,
               "message" => "32 1.2.3.4"

Logstash配置示例2（web服務生成日志，logstash加載並輸出，同時agent和server）

# yum install httpd
# for i in {1..50}; do echo "Test page $i." > test$i.html; done (生成40個頁面)
# systemctl start httpd
# LogFormat "%{X-Forwarded-For}i  (更改httpd的logformat)
# while true; do client=$[$RANDOM%254+1]; curl -s --header "X-Forwarded-For: 172.20.0.$client" http://172.20.42.239/test$client.html;sleep 1;done （模擬客戶端訪問，生成httpd的access_log）
以文件為輸入插件，同時加載Grok過濾器插件對日志分段，每一段加有特定意義的key，才能顯示數據的價值。Grok內置支持120多種模式，去匹配如htpd, redis等，可以查看/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns，每個模式匹配都有唯一的名字對應，在logstash配置文件中直接引用內置模式的名稱。
# vim test2.conf
   input {
    file{
            start_position => end
            path => ["/var/log/httpd/access_log"]
        }
    }

    filter {
            grok {
                match => { "message" => "%{HTTPD_COMBINEDLOG}" }
        }
    }
    output {
            stdout {
                    codec => rubydebug
                }
    }
# logstash -f test3.conf
# 顯示結果，模式匹配切割正常，達到輸出內容結構化。
       "request" => "/test54.html",
      "agent" => "\"curl/7.29.0\"",
       "auth" => "-",
      "ident" => "-",
       "verb" => "GET",
    "message" => "172.20.0.54 - - [01/Aug/2018:02:38:48 -0400] \"GET /test54.html HTTP/1.1\" 200 14 \"-\" \"curl/7.29.0\"",
       "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2018-08-01T06:38:48.897Z,
       "response" => "200",
      "bytes" => "14",
       "clientip" => "172.20.0.54",
       "@version" => "1",
       "host" => "kvm",
    "httpversion" => "1.1",
      "timestamp" => "01/Aug/2018:02:38:48 -0400"
    }
插件date，能把上述的@timestamp字段以timestamp字段顯示，同時把timestamp字段刪除（插件date的默認功能）
# vim test3.conf
input {
    file{
            start_position => end
            path => ["/var/log/httpd/access_log"]
        }
    }

    filter {
            grok {
                    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                    remove_field => "message"
        }
            date {
                    match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
                    remove_field => "timestamp"
        }
    }
    output {
            stdout {
                    codec => rubydebug
                }
    }

# 輸出 
{
    "request" => "/test111.html",
      "agent" => "\"curl/7.29.0\"",
       "auth" => "-",
      "ident" => "-",
       "verb" => "GET",
       "path" => "/var/log/httpd/access_log",
       "referrer" => "\"-\"",
     "@timestamp" => 2018-08-01T06:47:35.000Z,
       "response" => "404",
      "bytes" => "210",
       "clientip" => "172.20.0.111",
       "@version" => "1",
       "host" => "kvm",
    "httpversion" => "1.1"
    }

Geoip插件，能把訪問的IP用圖形展示所處的詳細。
# 到https://dev.maxmind.com/geoip/geoip2/geolite2/下載簡單版本
# vim test4.conf
input {
    file{
            start_position => end
            path => ["/var/log/httpd/access_log"]
        }
    }

    filter {
            grok {
                    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                    remove_field => "message"
            }
    date {
            match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
            remove_field => "timestamp"
    }
    geoip {
            source => "clientip"
            target => "geoip"
            database => "/etc/logstash/GeoLite2-City_20180703/GeoLite2-City.mmdb"
                }
    }
   output {
   stdout {
            codec => rubydebug
            }
    }
# 輸出IP所在的國家，城市，經緯度
{
    "request" => "/test208.html",
      "agent" => "\"curl/7.29.0\"",
      "geoip" => {
                "ip" => "208.20.0.100",
          "latitude" => 37.751,
      "country_name" => "United States",
     "country_code2" => "US",
    "continent_code" => "NA",
     "country_code3" => "US",
          "location" => {
        "lon" => -97.822,
        "lat" => 37.751
    },

輸出插件ElasticSearch
# vim test5.conf
input {
    file{
            start_position => end
            path => ["/var/log/httpd/access_log"]
    }
    }

filter {
        grok {
            match => { "message" => "%{HTTPD_COMBINEDLOG}" }
            remove_field => "message"
    }
    date {
            match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
            remove_field => "timestamp"
    }
    geoip {
            source => "clientip"
            target => "geoip"
            database => "/etc/logstash/GeoLite2-City_20180703/GeoLite2-City.mmdb"
                }
    }
    output {
    elasticsearch {
           hosts => ["http://node1:9200/","http://node2:9200/","http://node3:9200/"]
           index => "logstash-%{+YYYY.MM.dd}"
           document_type => "apache_logs"
           }
    }
# logstash -f test5.conf 啟動
# 查看ElasticSearch，可以看到以logstash為索引的
    curl -XGET node02:9200/_cat/indices
    green open logstash-2018.08.01 LceRz3DlQkiBx8BlEDhs_g 5 1 611 0 1.1mb 746kb
# 查詢此index的具體信息
    curl -XGET node02:9200/logstash-2018.08.01/_search?q=clientip:109.20.0.100
    curl -XGET node02:9200/logstash-2018.08.01/_search?q=response:404

安裝Kibana圖表展示工具：
只要Kinba能訪問ElasticSearch中的任何一個結點， 就可以圖形展示
# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.8-x86_64.rpm
# rpm -ivh kibana-5.6.8-x86_64.rpm
# cd /etc/kibana/
# server.port: 5601; server.basePath: "/kibina"; server.host: "0.0.0.0"; elasticsearch.url: "http://node02:9200"
# systemctl start kibana
# http://172.20.42.240:5601/app/kibana 訪問網頁
# 選擇加載的index可以用通配符匹配
# 設置搜索item如geoip.timezone:Asia
response:[200 TO 302] 范圍查詢
agent:curl 客戶端訪問類型
agent:curl OR chrome or firefox OR safari
agent:curr~ 模糊搜索
# 還可以用Visualize

Logstash配置示例3（web服務生成日志，beats組件輕量級的收集日志，logstash只做為server，輸出到ElasticSearch）

filebeat需要從指定的位置讀入日志，然后再發給logstash，filebeat無法做日志的文檔化
# vim filebeat.yml
     paths:
    - /var/log/httpd/access_log*
    output.elasticsearch:
      output.elasticsearch:
      hosts: ["node03:9200","node02:9200"]
# systemctl start filebeat
# curl node03:9200/_cat/indices
        yellow open filebeat-2018.08.01 M2TgTLhFTe2MtprkyR9lag 5 1    0 0   324b   324b
# 更改filebeat輸出到logstash
    output.logstash:
    hosts: ["172.20.42.239:5044"]

# 更改logstash的配置文件（input更改為beats，其余保持不變）
    input {
    beats {
            port => 5044
        }
    }
# curl node03:9200/_cat/indices
    yellow open logstash-2018.08.01 PbCtEn_UTZOfknX_5LnGaA 5 1    148 0 838.5kb 838.5kb

Logstash配置示例4（web服務生成日志，beats組件輕量級的收集日志，發送給redis消息隊列，logstash只做為server去redis取數據，輸出到ElasticSearch）

# yum install redis
# vim /etc/redis.conf
    bind 0.0.0.0
    requirepass redhat
# systemctl start redis
# filebeat.full.yml中有redis的配置示例 ,復制到filebeat.yml，更改相關的配置
    output.redis:
    enabled: true
     hosts: ["172.20.42.222:6379"]
     port: 6379
     key: filebeat
     password: redhat
     db: 0
     datatype: list
# logstash從redis中加載日志
    input {
        redis {
                data_type => "list"
                db => 0
                host => "172.20.42.239"
                key => "filebeat"
                port => 6379
                password => "redhat"
        }
    }
# 在ElasticSearch上 curl node03:9200/_cat/indices
    yellow open logstash-2018.08.02 dKULdxU_SzuxmJQToCMbug 5 1 2246 0  1.3mb  1.3mb
# 在redis服務器，redis-cli中查看，filebeat中消息隊列為0，即消息隊列每增加一個，都會被logstash取走。
    127.0.0.1:6379> LLEN filebeat
    (integer) 0