elasticsearch配置集群+elk報錯總結

配置ELK的時候，我平常遇到了以下幾種報錯情況，整理如下（持續更新中）：

1. elasticsearch啟動失敗

   # systemctl start elasticsearch
   Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
   
   #這個時候，直接查看系統日志，因為elasticsearch沒有專門的日志審計
   tail -f /var/log/messages
   

   出現如下報錯

   Dec 13 10:16:30 oldboy elasticsearch: ERROR: [1] bootstrap checks failed
   Dec 13 10:16:30 oldboy elasticsearch: [1]: initial heap size [536870912] not equal to maximum heap size [775946240]; this can cause resize pauses and prevents mlockall from locking the entire heap
   
   

   其實提示已經很明顯了，jvm給的內存不足，那么我們直接把內存調大就可以了

   #修改jvm內存大小
   # vim /etc/elasticsearch/jvm.options
   -Xms1500m 
   -Xms1500m
   #因為剛才把內存改的很小，改回來就行了
   

   如果不是使用的systemd方法啟動，直接調用bin/elasticsearch 啟動，那么有幾點需要注意

   #1.不能使用root進行登錄
   useradd elk #創建用戶elk
   
   #2.將涉及的用戶權限賦予elk
   

2. kibana顯示中文亂碼

   #首先查看要拉取的日志的格式是什么
   file file.txt  #在linux上查看
   
   以記事本打開log文件，點擊另存為查看，如果顯示為ANSI，那么就是gbk  #在windows上查看
   
   #在filebeat中配置字符集
   
   # vim /etc/filebeat/filebeat.yml
   
   filebeat.inputs:
   
   - type: log
   
    
     enabled: true
   
     paths:
       - c:\work\CA*
     encoding: gbk   #此處加入字符格式，如果是utf8，那么不需要添加
   

   繼續生成測試日志，登錄kibana查看，發現中文字符已經正常顯示，沒有亂碼了。

3. es集群配置xpack啟動后，創建密碼失敗

   [root@db01 elasticsearch]# bin/elasticsearch-setup-passwords interactive
   
   Failed to determine the health of the cluster running at http://10.0.0.200:9200
   Unexpected response code [503] from calling GET http://10.0.0.200:9200/_cluster/health?pretty
   Cause: master_not_discovered_exception
   
   It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
   It is very likely that the password changes will fail when run against an unhealthy cluster.
   
   Do you want to continue with the password setup process [y/N]y
   
   Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
   You will be prompted to enter passwords as the process progresses.
   Please confirm that you would like to continue [y/N]y
   
   
   #錯誤原因，因為臟數據的原因，當開始xpack的時候，集群鏈接失敗
   
   #終極大招（只適用於初始創建集群，或者測試環境）
   
   1.停止服務
   2.刪除數據目錄
   3.三個節點只配置xpack.security.enabled: true,啟動
   4.設置密碼
   
   #配置文件（三台除了ip之外都一樣）
   cluster.name: think
   node.name: node-1
   path.data: /var/lib/elasticsearch
   path.logs: /var/log/elasticsearch
   bootstrap.memory_lock: true
   network.host: 10.0.0.200,127.0.0.1
   http.port: 9200
   discovery.seed_hosts: ["10.0.0.200", "10.0.0.201"]
   cluster.initial_master_nodes: ["10.0.0.200", "10.0.0.201","10.0.0.202"]
   http.cors.enabled: true
   http.cors.allow-origin: "*"
   xpack.security.enabled: true
   
   
   #測試效果
   [root@db01 elasticsearch]# bin/elasticsearch-setup-passwords interactive
   Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
   You will be prompted to enter passwords as the process progresses.
   Please confirm that you would like to continue [y/N]y
   
   
   Enter password for [elastic]: 
   Reenter password for [elastic]: 
   Enter password for [apm_system]: 
   Reenter password for [apm_system]: 
   Enter password for [kibana]: 
   Reenter password for [kibana]: 
   Enter password for [logstash_system]: 
   Reenter password for [logstash_system]: 
   Enter password for [beats_system]: 
   Reenter password for [beats_system]: 
   Enter password for [remote_monitoring_user]: 
   Reenter password for [remote_monitoring_user]: 
   Changed password for user [apm_system]
   Changed password for user [kibana]
   Changed password for user [logstash_system]
   Changed password for user [beats_system]
   Changed password for user [remote_monitoring_user]
   Changed password for user [elastic]
   
   #成功
   

4.隔天上班又出現和標題3同樣的情況，如下解決方案

#直接配上ca證書驗證，開啟ssl

# 設置默認的角色密碼
bin/elasticsearch-setup-passwords interactive  #這一步我是不成功的，不過標題3已經創建過了，所以跳過

再elasticsearch.yml加入如下
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate # 證書驗證級別
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 # 節點證書路徑
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

#創建證書
# 創建keystore文件
# bin/elasticsearch-keystore create # config文件夾下有的話這一步就不用再執行了

# 生成CA證書，一直回車
bin/elasticsearch-certutil ca (CA證書：elastic-stack-ca.p12)

# 生成節點使用的證書，一直回車
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12  (節點證書：elastic-certificates.p12)

# 創建證書保存目錄，並移動到config文件下
mkdir -p /etc/elasticsearch/certs
mv elastic-certificates.p12 /etc/elasticsearch/certs 
chmod 777 /etc/elasticsearch/certs   #不給授權就無法登錄，可以自己測測到底給多少合適

#重啟