5分鍾部署filebeat + ELK 5.1.1

標題有點噱頭，不過網絡環境好的情況下也差不多了^_^

 

1. 首先保證安裝了jdk。

 

elasticsearch, logstash, kibana，filebeat都可以通過yum安裝，這里前三者通過直接下載壓縮包安裝啟動，filebeat通過yum安裝。

 

2. 下載elasticsearch-5.1.1解壓， 配置

elasticsearch-5.1.1/conf/elasticsearch.yml

# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0 # # Set a custom port for HTTP: # http.port: 9200

 啟動（可以放到supervisor下監管）：

elasticsearch-5.1.1/bin/elasticsearch

注意： es啟動要求提高一些系統參數配置，否則會報錯

a. 增大vm.max_map_count到至少262144

sudo vim  /etc/sysctl.conf
添加  vm.max_map_count=262144
sudo sysctl -p

b. 增大文件句柄數至少 65536  ulimit -a查看

sudo vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536

 

 

2. 下載logstash5.1.1解壓， 配置

logstash-5.1.1/conf.d/pro-log.conf

input {
   beats {
      port => 5044
   }
}

filter {
   if [fields][logIndex] == "nginx" {
      grok {
         patterns_dir => "/home/elk/apps/logstash-5.1.1/patterns"
         match => {
            "message" => "%{NGINXACCESS}"
         }
      }
      urldecode {
         charset => "UTF-8"
         field => "url"
      }
      if [upstreamtime] == "" or [upstreamtime] == "null" {
         mutate {
            update => { "upstreamtime" => "0" }
         }
      }
      date {
         match => ["logtime", "dd/MMM/yyyy:HH:mm:ss Z"]
         target => "@timestamp"
      }
      mutate {
         convert => {

            "responsetime" => "float"
            "upstreamtime" => "float"
            "size" => "integer"
         }
         remove_field => ["port","logtime","message"]
      }

   }
}

output {
   elasticsearch {
      hosts => "{your-es-ip}:9200"
      manage_template => false
      index => "%{[fields][logIndex]}-%{+YYYY.MM.dd}"
      document_type => "%{[fields][docType]}"
   }

}

這里使用grok解析nginx日志

nginx日志格式：

log_format app_log_format '[$time_local] $server_addr $remote_addr $body_bytes_sent $request_time $upstream_response_time '
                        '$upstream_addr $upstream_status "$request_uri" "$http_x_forwarded_for" "$http_referer" "$http_user_agent" $status';

配置grok的自定義pattern（可以使用grok debugger工具進行驗證 http://grokdebug.herokuapp.com/）：

vim logstash-5.1.1/patterns/nginx
NGINXACCESS \[%{HTTPDATE:logtime}\] %{IPORHOST:host} %{IPORHOST:remoteaddr} (?:%{NUMBER:size}|-) %{NUMBER:responsetime} (?:%{NUMBER:upstreamtime}|-) %{URIHOST:upstreamhost} %{BASE10NUM:upstreamstatus} %{QS:url} %{QS:clientip} %{QS:referrer} %{QS:agent} %{INT:status}

啟動（可以放到supervisor下監管）：

logstash-5.1.1/bin/logstash -f logstash-5.1.1/conf.d/pro-log.conf

 

3. 安裝filebeat，filebeat可以直接使用yum安裝。

配置yum源：

vim  /etc/yum.repos.d/elastic5.repo
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

安裝

sodu yum install filebeat

配置（默認開始output到es，需要注釋掉）

sudo vim /etc/filebeat/filebeat.yml

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /opt/nginx/logs/app.access.log
  fields:
    logIndex: nginx
    docType: nginx-access
    project: app-nginx
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["{your-logstash-ip}:5044"]

啟動

sudo systemctl start filebeat

 

4. 下載kibana-5.1.1解壓，配置

kibana-5.1.1/config/kibana.yml

server.port: 5601
server.host: 0.0.0.0
elasticsearch.url: "http://{your-es-ip}:9200"

 啟動（可以放到supervisor下監管）

kibana-5.1.1-linux-x86_64/bin/kibana

 

5. （選擇性安裝）安裝x-pack，x-pack包含了security（需要用戶名密碼訪問kibana）、watcher（監控報警）等插件

elasticsearch.5.1.1/bin/elasticsearch-plugin install x-pack  （很慢，最好可以通過VPN安裝，或者通過下載x-pack zip包，通過離線安裝）

離線安裝：

elasticsearch.5.1.1/bin/elasticsearch-plugin install file:///home/elk/apps/x-pack-5.1.1.zip
kibana-5.1.1/bin/kibana-plugin install file:///home/elk/apps/x-pack-5.1.1.zip

 裝完后，重啟es，kibana生效。

不讓某個插件生效，如不讓security生效，則在es與kibana的配置里加入

xpack.security.enabled: false

 

最后貼兩張kibana統計查詢nginx日志得出的api調用次數，及平均響應時間圖表。