elasticsearch + fluentd + kibana 日志收集

軟件包使用說明

說明：

1.這里采用離線安裝所有的軟件(elasticsearch、fluentd、kibana、jdk)，且使用rpm包方式進行安裝。
本文所使用軟件包 下載鏈接 提取碼：uq8o

軟件版本：

Software	Version	MD5
jdk	1.8.0_211	561abbcd9cc9214714de8429c679d56e
elasticsearch	6.8.1	6a95250e603710fc515c91831734665b
kibana	6.8.1	79a9bb38de1508e5fe5695ebc1514bbd
fluentd(td-agent)	3.6.0	ff093b5ee4350f81bce45597bca435b6

Elasticsearch部署

因為elasticsearch服務運行需要java環境，所以首先安裝jdk

安裝jdk

# yum localinstall jdk-8u211-linux-x64.rpm -y
# java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

安裝elasticsearch

1.安裝
# yum localinstall elasticsearch-6.8.1.rpm -y

2.編輯配置文件
# cp /etc/elasticsearch/elasticsearch.yml{,.bck}
# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: ELK-Cluster    #ELK的集群名稱，名稱相同即屬於是同一個集群
node.name: elk-node1         #本機在集群內的節點名稱
path.data: /var/lib/elasticsearch    #數據存放目錄
path.logs: /var/log/elasticsearch    #日志保存目錄
bootstrap.memory_lock: true  #服務啟動的時候鎖定足夠的內存，防止數據寫入swap
network.host: 192.168.3.60   #監聽的IP地址
http.port: 9200              #服務監聽的端口
discovery.zen.ping.unicast.hosts: ["192.168.3.60"]    #單播配置一台即可

3.修改啟動內存限制，內存鎖定需要配置2g以上的內存，否則會導致啟動失敗
# vim /usr/lib/systemd/system/elasticsearch.service
# 在[Service]下加入下面這行內容
LimitMEMLOCK=infinity

4.修改內存限制，這里是測試服務器所以只給2g
# vim /etc/elasticsearch/jvm.options
-Xms2g
-Xmx2g

5.啟動elasticsearch並加入開機啟動
# systemctl start elasticsearch.service 
# systemctl enable elasticsearch.service 
# netstat -nltp |grep java
tcp6       0      0 192.168.3.60:9200       :::*                    LISTEN      35849/java          
tcp6       0      0 192.168.3.60:9300       :::*                    LISTEN      35849/java

6.可以通過shell命令獲取集群狀態,可以對status進行分析，如果等於green(綠色)就是運行在正常，等於yellow(黃色)表示副本分片丟失，red(紅色)表示主分片丟失。
# curl http://192.168.3.60:9200/_cluster/health?pretty=true
{
  "cluster_name" : "ELK-Cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

web界面訪問查看狀態 http://IP:PORT

安裝elasticsearch插件head

head插件主要用來做集群管理的插件

這里同樣采用離線安裝方式，說明，需要安裝npm
1.安裝npm 
# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
# yum install npm -y

2.上傳軟件並解壓
# ls elasticsearch-head.tar.gz 
elasticsearch-head.tar.gz
# tar xf elasticsearch-head.tar.gz -C /usr/local/

3.啟動
# cd /usr/local/elasticsearch-head/
# npm run start &

4.由於上面啟動不方便，故編寫一個啟動腳本
# cat /usr/bin/elasticsearch-head
#!/bin/bash
#desc: elasticsearch-head service manager

data="cd /usr/local/elasticsearch-head/; nohup npm run start > /dev/null 2>&1 & "

function START (){
    eval $data && echo -e "elasticsearch-head start\033[32m     ok\033[0m"
}

function STOP (){
    ps -ef |grep grunt |grep -v "grep" |awk '{print $2}' |xargs kill -s 9 > /dev/null && echo -e "elasticsearch-head stop\033[32m      ok\033[0m"
}

case "$1" in
    start)
        START
        ;;
    stop)
        STOP
        ;;
    restart)
        STOP
        sleep 3
        START
        ;;
    *)
        echo "Usage: elasticsearch-head (start|stop|restart)"
        ;;
esac

添加執行權限
# chmod +x /usr/bin/elasticsearch-head
啟動
# elasticsearch-head start 
# netstat -nltp |grep 9100
tcp        0      0 0.0.0.0:9100            0.0.0.0:*               LISTEN      36484/grunt

5.修改elasticsearch配置文件，開啟跨域訪問支持，並重啟elasticsearch
# vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: "*"
# systemctl restart elasticsearch.service

訪問elasticsearch-head

fluentd部署

安裝fluentd

無網絡情況：

1.安裝td-agent
# yum localinstall td-agent-3.6.0-0.el7.x86_64.rpm -y

2.安裝插件(通過這些插件fluend才能將消息推送到elasticsearch)
插件下載地址：https://rubygems.org/
# /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch-4.0.7.gem
# /usr/sbin/td-agent-gem install fluent-plugin-typecast-0.2.0.gem

安裝fluent-plugin-secure-forward插件依賴於proxifier和resolve-hostname插件，
# /usr/sbin/td-agent-gem install fluent-plugin-secure-forward-0.4.5.gem 
ERROR:  Could not find a valid gem 'proxifier' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - no such name (https://rubygems.org/specs.4.8.gz)
# /usr/sbin/td-agent-gem install fluent-plugin-secure-forward-0.4.5.gem 
ERROR:  Could not find a valid gem 'resolve-hostname' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - no such name (https://rubygems.org/specs.4.8.gz)
故而先安裝這兩個插件，再安裝fluent-plugin-secure-forward
# /usr/sbin/td-agent-gem install proxifier-1.0.3.gem
# /usr/sbin/td-agent-gem install resolve-hostname-0.1.0.gem
# /usr/sbin/td-agent-gem install fluent-plugin-secure-forward-0.4.5.gem

有網絡情況：

# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
# /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch
# /usr/sbin/td-agent-gem install fluent-plugin-typecast
# /usr/sbin/td-agent-gem install fluent-plugin-secure-forward

配置fluentd

1.編輯配置文件
這里配置收集系統日志，/var/log/messages ，需要給予可讀權限
# cp /etc/td-agent/td-agent.conf{,.bck}
# cat /etc/td-agent/td-agent.conf
<source>
@type forward
port 24224
</source>

<source>
@type tail
path /var/log/messages
pos_file /var/log/td-agent/messages.log.pos
tag message
<parse>
@type json
</parse>
</source>

<match debug.**>
@type stdout
</match>

<match *.**>
@type copy
<store>
@type elasticsearch
host 192.168.3.60
port 9200
logstash_format true
logstash_prefix message-${tag}
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s
</store>
<store>
@type stdout
</store>
</match>

2.給予被收集的日志可讀權限
# chmod o+r /var/log/messages  

3.啟動td-agent，並測試
# systemctl start td-agent
# systemctl enable td-agent

# tail -2 /var/log/td-agent/td-agent.log 
2020-03-29 14:50:01 +0800 [warn]: #0 pattern not matched: "Mar 29 14:50:01 localhost systemd: Starting Session 38 of user root."
2020-03-29 14:50:01.433035076 +0800 fluent.warn: {"message":"pattern not matched: \"Mar 29 14:50:01 localhost systemd: Starting Session 38 of user root.\""}

elasticsearch-head界面查詢

kibana部署

1.安裝kibana
# yum localinstall kibana-6.8.1-x86_64.rpm -y

2.配置
# cp /etc/kibana/kibana.yml{,.bck}
# cat /etc/kibana/kibana.yml
server.port: 5601
server.host: "192.168.3.60"
elasticsearch.hosts: ["http://192.168.3.60:9200"]
i18n.locale: "zh-CN"

3.啟動kibana
# systemctl start kibana
# systemctl enable kibana
# netstat -nltp |grep 5601
tcp        0      0 192.168.3.60:5601       0.0.0.0:*               LISTEN      39743/node

web界面添加索引 http://IP:PORT