部署elasticsearch（三節點）集群+filebeat+kibana

1. 用途
   ▷ 通過各個beat實時收集日志、傳輸至elasticsearch集群
   ▷ 通過kibana展示日志

2. 實驗架構
   名稱：IP地址：CPU：內存
   kibana&cerebro：192.168.75.20：1核：2G
   es-1：192.168.75.21：2核：4G
   es-2：192.168.75.22：2核：4G
   es-3：192.168.75.23：2核：4G

各beat均安裝在ES集群上，也就是三個es主機節點

3. 軟件安裝

• 版本說明：
  均為7.3.0版本
• 注意事項
  各組件版本必須一致，elasticsearch必須3台及其以上且總數量為單數
• 軟件保存路徑：/usr/local/src
• 采用rpm文件方式安裝
• 官方下載地址：https://www.elastic.co/cn/downloads/past-releases

3.1 elasticsearch

3.1.1 安裝

3台es均執行相同的安裝步驟

echo "vm.max_map_count = 655350" >> /etc/sysctl.conf
sysctl -p

cd /usr/local/src
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.3.0-x86_64.rpm
rpm -vi elasticsearch-7.3.0-x86_64.rpm

默認配置文件路徑：/etc/elasticsearch/
默認程序安裝路徑：/usr/share/elasticsearch/
默認存儲文件路徑：/var/lib/elasticsearch/
默認日志文件路徑：/var/log/elasticsearch/

3.1.2 配置

# es-1主機

# cat /etc/elasticsearch/elasticsearch.yml | grep -v '^#'

# 集群名字
cluster.name: my-application

# 節點名稱
node.name: 192.168.75.21

# 數據路徑
path.data: /var/lib/elasticsearch

# 日志路徑
path.logs: /var/log/elasticsearch

# 本界面訪問IP
network.host: 192.168.75.21

# 本屆點訪問端口
http.port: 9200

# 節點運輸端口
transport.port: 9300

# 集群中其他主機列表
discovery.seed_hosts: ["192.168.75.21", "192.168.75.22","192.168.75.23"]

# 首次啟動全新的Elasticsearch集群時，在第一次選舉中便對其票數進行計數的master節點的集合
cluster.initial_master_nodes: ["192.168.75.21", "192.168.75.22","192.168.75.23"]

# 啟用跨域資源共享
http.cors.enabled: true
http.cors.allow-origin: "*"

# es-2主機
# cat /etc/elasticsearch/elasticsearch.yml | grep -v '^#'
cluster.name: my-application
node.name: 192.168.75.22
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.75.22
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.75.21", "192.168.75.22","192.168.75.23"]
cluster.initial_master_nodes: ["192.168.75.21", "192.168.75.22","192.168.75.23"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# es-3主機
# cat /etc/elasticsearch/elasticsearch.yml | grep -v '^#'
cluster.name: my-application
node.name: 192.168.75.23
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.75.23
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.75.21", "192.168.75.22","192.168.75.23"]
cluster.initial_master_nodes: ["192.168.75.21", "192.168.75.22","192.168.75.23"]
http.cors.enabled: true
http.cors.allow-origin: "*"

3.1.3 啟動/重啟/停止

systemctl start elasticsearch
systemctl stop elasticsearch
systemctl restart elasticsearch
systemctl status elasticsearch

3.1.4 查看集群健康、節點狀態

# 注：第一個IP為集群中任一節點ip,第二個ip為界面名稱

# elasticsearch啟動后查看是否啟動成功
curl -XGET "http://192.168.75.21:9200/_cluster/health?pretty=true"

# 停止elasticsearch應用
curl -XPOST "http://192.168.75.21:9200/_shutdown"

# 查看集群健康
curl 192.168.75.21:9200/_cluster/health?pretty

# 檢查集群狀態
curl 192.168.75.21:9200/_cluster/stats?pretty

# 節點狀態
curl 192.168.75.21:9200/_nodes/process?pretty

curl 192.168.75.21:9200/_nodes/192.168.75.21/process?pretty

# 當你不知道有那些屬性可以查看時,會返回可以查看的屬性
curl '192.168.75.21:9200/_cat/'

3.2 kibana

3.2.1 安裝

cd /usr/local/src
curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-7.3.0-x86_64.rpm
rpm -vi kibana-7.3.0-x86_64.rpm

默認配置文件路徑：/etc/kibana/
默認程序安裝路徑：/usr/share/kibana/

3.2.3 配置

# cat /etc/kibana/kibana.yml| grep -v '^#'

# 訪問端口號
server.port: 5601
# 訪問地址
server.host: "192.168.75.20"
# 名稱
server.name: "192.168.75.20"
# es集群地址
elasticsearch.hosts: ["http://192.168.75.21:9200", "http://192.168.75.22:9200","http://192.168.75.23:9200"]
# 中文界面
i18n.locale: "zh-CN"

3.2.3 啟動/重啟/停止

systemctl start kibana
systemctl stop kibana
systemctl restart kibana
systemctl status kibana

3.2.4 瀏覽器訪問
地址： http://192.168.75.20:5601

3.3 cerebro

官方地址：https://github.com/lmenezes/cerebro
下載地址：https://github.com/lmenezes/cerebro/releases

前提條件：需要有java環境

wget https://github.com/lmenezes/cerebro/releases/download/v0.8.5/cerebro-0.8.5.tgz
tar xzf cerebro-0.8.5.tgz -C /usr/local


# 配置ES服務器
# 非必須：如果經常使用的話，可以先在conf/application.conf中配置好ElasticSearch服務器地址
# 第一個是不加密的，第二個是加密的
hosts = [
  {
    host = "http://192.168.75.21:9200" # 設置集群中的一個地址
    name = "Localhost cluster"
    headers-whitelist = [ "x-proxy-user", "x-proxy-roles", "X-Forwarded-For" ]
  }
  # Example of host with authentication
  # {
  #  host = "http://172.17.107.187:9203"
  #  name = "my-application"
  #  auth = {
  #    username = "elastic"
  #    password = "escluter123456"
  #  }
  # }
]


# 啟動,默認使用9000端口
cerebro-0.8.1/bin/cerebro
[info] play.api.Play - Application started (Prod)
[info] p.c.s.AkkaHttpServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

# 指定端口啟動
bin/cerebro -Dhttp.port=8080

# 指定地址啟動
bin/cerebro -Dhttp.address=192.168.75.20

# 使用瀏覽器訪問http://192.168.75.20:9000

3.4 Filebeat

在三台es主機節點上進行安裝,收集elasticsearch的日志，默認索引為filebeat-7.3.0-*
Filebeat 模塊 elasticsearch 解析 Elasticsearch 創建的日志

# 下載並安裝 Filebeat
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.0-x86_64.rpm
rpm -vi filebeat-7.3.0-x86_64.rpm

# 修改 /etc/filebeat/filebeat.yml 以設置連接信息：

setup.kibana:
  host: "192.168.75.20:5601"

output.elasticsearch:
  hosts: ["192.168.75.21:9200","192.168.75.22:9200","192.168.75.23:9200"]

# 啟用和配置 elasticsearch 模塊,在 /etc/filebeat/modules.d/elasticsearch.yml 文件中修改設置
filebeat modules enable elasticsearch

# 啟動 Filebeat
filebeat setup
service filebeat start

3.5 Metricbeat

在三台es主機節點上進行安裝,收集es主機的日志，默認索引為metricbeat-7.3.0-*

# 下載並安裝 metricbeat
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.3.0-x86_64.rpm
rpm -vi metricbeat-7.3.0-x86_64.rpm

# 修改 /etc/metricbeat/metricbeat.yml 以設置連接信息：

setup.kibana:
  host: "192.168.75.20:5601"

output.elasticsearch:
  hosts: ["192.168.75.21:9200","192.168.75.22:9200","192.168.75.23:9200"]

# 啟用和配置 system 模塊，在 /etc/metricbeat/modules.d/system.yml 文件中修改設置
metricbeat modules enable system

# 啟動 Metricbeat
metricbeat setup
service metricbeat start

3.6 Heartbeat

在三台es主機節點上進行安裝,通過主動探測來監測服務的可用性，默認索引為heartbeat-7.3.0-*

# 下載並安裝 heartbeat
curl -L -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-7.3.0-x86_64.rpm
rpm -vi heartbeat-7.3.0-x86_64.rpm

# 修改 /etc/heartbeat/heartbeat.yml 以設置連接信息

setup.kibana:
  host: "192.168.75.20:5601"

output.elasticsearch:
  hosts: ["192.168.75.21:9200","192.168.75.22:9200","192.168.75.23:9200"]

# 添加監測,注意每台es節點主機ip不同，其中 <http://192.168.75.21:9200> 是受監測 URL
heartbeat.monitors:
- type: http
  urls: ["http://192.168.75.21:9200"] 
  schedule: "@every 10s"


# 啟動 Heartbeat
heartbeat setup
service heartbeat-elastic start