背景： 生產環境大量使用 elasticsearch 集群，不同的業務使用不同版本的elasticsearch es經常曝出一些大的漏洞，需要進行版本升級，並且使用x-pack的基本驗證功能，避免用戶數據泄露 x-pack免費版本特征： 基本的TLS 功能，可對通信進行加密 文件和原生 Realm，可用於創建和管理用戶 基於角色的訪問控制，可用於控制用戶對集群 API 和索引的訪問權限； 通過針對 Kibana Spaces 的安全功能，還可允許在 Kibana 中實現多租戶。 升級的兩種策略： 1.滾動性升級，即不中斷業務服務，一台一台進行升級 2.全新部署新版本，然后將數據遷移到新版本的es集群中 這兩種方式都需要將數據恢復到新版本的es集群中，可以先進行快照備份 1.升級前先備份低版本的elasticserch的數據：快照方式 原理：即將老版本的es數據打個快照備份出來寫入到 /opt/esback 目錄中並進行nfs掛載到某台服務器中(作為nfs服務端的機器要有足夠量的磁盤空間，最好磁盤性能比較好)，新、舊兩個es集群的配置文件中都引用配置 path.repo: ["/opt/esback/"]， 這樣新的集群也能對這個目錄進行操作了，等待新集群搭建好后，直接把 /opt/esback 目錄中的文件恢復到新集群的 索引 index 中即可 使用Mount nfs進行掛載共享(所有的es集群節點都可以訪問)： 目標：將本地es備份出來的數據目錄/opt/esback 目錄掛載到nfs的共享目錄 /opt/es_snapshot，這樣恢復的時候就都可以訪問這個共享目錄進行恢復了 // 在10.10.18.92上創建共享目錄 創建共享目錄，即作為nfs的共享目錄 mkdir /opt/es_snapshot 創建本地備份出來的目錄 /opt/esback # 在集群所有節點中創建 /opt/esback 目錄，即將es數據備份出來的目標目錄 # 將其中一台es客戶端作為nfs服務端 #nfs服務端的操作 # vim /etc/exports # 注意此處的anonuid和gid要和運行es程序的用戶保持一致 # 添加指定 uid 和 gid 的用戶 groupadd -g 1000 elastic useradd -u 1000 -g elastic elastic # 修改 gid和 uid為500 命令示例： usermod -u 500 es groupmod -g 500 es /opt/es_snapshot *(insecure,rw,no_root_squash,sync,anonuid=1000,anongid=1000) // 查看共享文件夾 yum install -y exportfs exportfs -rv // nfs服務端修改nfs配置 vim /etc/sysconfig/nfs 修改如下： RPCNFSDARGS="-N 2 -N 3" ----->啟用 # Turn off v4 protocol support RPCNFSDARGS="-N 4" ---->啟用 重啟生效 systemctl restart nfs // 客戶端操作 yum install -y nfs-utils // 重啟啟動新集群機器的NFS服務 systemctl restart nfs //每一台es節點服務器上進行Mount掛載 mount -t nfs 10.10.18.90:/opt/es_snapshot /opt/esback -o proto=tcp -o nolock 列出nfs服務端共享的目錄: [root@sz_kp_wanghong_dev02_18_93:/home/wanxing]# showmount -e 10.10.18.90 Export list for 10.10.18.92: /opt/es_snapshot * // 在舊機器上將共享目錄的權限付給ES的運行用戶 chown elastic:elastic -R /opt/esback 2.創建ES倉庫my_backup 修改配置文件： vim elasticsearch.yml # 添加如下配置(需要在舊集群的每個節點上添加)，重新啟動集群 path.repo: ["/opt/esback"] 創建快照倉庫 my_backup 命令： curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup -d ' { "type": "fs", "settings": { "location": "/opt/esback", "compress": true } } ' # 返回值 {"acknowledged":true} # 報錯的處理 'RemoteTransportException[[ictr_node1][10.10.18.93:9300][internal:admin/repository/verify]] # 權限不夠 chown -R es.es /opt/es_snapshot/ chown -R es.es /opt/esback_20191104/ # 創建所有索引的備份 # curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107 {"accepted":true} 查看備份 [elastic@szyyelk01t slave02]$ curl -XGET http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107?pretty { "snapshots" : [ { "snapshot" : "snapshot20191107", "uuid" : "0_4SOntVS1GH-7irHjKBMQ", "version_id" : 6030299, "version" : "6.3.2", "indices" : [ "support_faq_categorys", "ticket_list", "templates_search", "site_page_search", "support", "templates_page_search", "support_new_articles", "article_version", "blocks_version", "search", "version", "article_search", "templates", "learn", "templates_version", "blocks_search", "templates_page_version" ], "include_global_state" : true, "state" : "SUCCESS", "start_time" : "2019-11-07T01:35:00.811Z", "start_time_in_millis" : 1573090500811, "end_time" : "2019-11-07T01:35:03.702Z", "end_time_in_millis" : 1573090503702, "duration_in_millis" : 2891, "failures" : [ ], "shards" : { "total" : 71, "failed" : 0, "successful" : 71 } } ] } 升級方式1：滾動升級 elasticsearch5.6.16 --> elasticsearch6.8.4 1.備份數據，避免出現問題后回滾 2.先升級到新版本，然后安裝x-pack，此時再要求開發同事修改代碼適配 a.先下載新版本的6.8.4 ①關閉自動分片 curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{ "persistent": { "cluster.routing.allocation.enable": "none" } }' [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{ > "persistent": { > "cluster.routing.allocation.enable": "none" > } > }' * Hostname was NOT found in DNS cache * Trying 10.10.18.92... * Connected to 10.10.18.92 (10.10.18.92) port 9200 (#0) > PUT /_cluster/settings HTTP/1.1 > User-Agent: curl/7.36.0 > Host: 10.10.18.92:9200 > Accept: */* > Content-Length: 73 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 73 out of 73 bytes < HTTP/1.1 200 OK < Warning: 299 Elasticsearch-5.6.15-fe7575a "Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header." "Tue, 05 Nov 2019 08:14:44 GMT" < content-type: application/json; charset=UTF-8 < content-length: 106 < * Connection #0 to host 10.10.18.92 left intact {"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"none"}}}},"transient":{}} ②暫時禁用非必要的索引並執行同步刷新 curl -XPOST http://10.10.18.92:9200/_flush/synced [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# curl -XPOST http://10.10.18.92:9200/_flush/synced {"_shards":{"total":28,"successful":28,"failed":0},"channel_rel":{"total":4,"successful":4,"failed":0},".kibana":{"total":2,"successful":2,"failed":0},"channel":{"total":6,"successful":6,"failed":0},"video":{"total":4,"successful":4,"failed":0},"channel_list":{"total":6,"successful":6,"failed":0},"influecer":{"total":6,"successful":6,"failed":0}} 注意: 如果是從6.3之前的版本升級上來的，需要注意提前要移除X-Pack插件，然后再去升級版本。執行bin/elasticsearch-plugin remove x-pack a. 備份原來的elasticsearch目錄，然后解壓新版的elasticsearch。 b. 如果使用外部的配置路徑，配置ES_PATH_CONF環境變量到那個位置。如果沒有的話，拷貝老的配置目錄過來新的elasticsearch目錄就可以了。 c. 檢查path.data是否指向正確的數據目錄 d. 檢查path.log是否指向正確的日志目錄 新集群的配置文件 [es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ more config/elasticsearch.yml cluster.name: kp-dev-application node.name: ictr_node2 node.master: true node.attr.rack: r1 node.max_local_storage_nodes: 3 network.host: 10.10.18.92 http.port: 9200 transport.tcp.port: 9300 path.repo: ["/opt/esback_20191104"] discovery.zen.minimum_master_nodes: 1 http.cors.enabled: true http.cors.allow-origin: "*" # 新集群的數據還是指向老版本 es5.6.15 的數據存儲目錄 path.data: /opt/es-node/elasticsearch-5.6.15/data path.logs: /opt/es-node/elasticsearch-5.6.15/logs # 啟用安全認證 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 ③關閉節點 ④重新啟動節點，注意要切換到es用戶，不能使用root用戶 chown -R es.es elasticsearch-6.8.4 [es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ bin/elasticsearch -d 在其他節點重復以上過程 啟動升級后的節點，並通過查看日志和使用下面命令來檢查節點是否正確加入到集群 [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cat/nodes 10.10.18.93 16 98 56 1.22 0.50 0.29 di - ictr_node1 10.10.18.92 16 88 8 0.08 0.26 0.31 mdi * ictr_node2 [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cat/indices yellow open channel vRFQoIhmT8WmSbDCfph0ag 3 1 53374 0 44.2mb 44.2mb yellow open channel_rel ZeeBbkogT5KtxzziUYtu_Q 2 1 459528 0 168.8mb 168.8mb yellow open channel_list 1dk8uH8bTeikez0lFR2mJg 3 1 5509390 78630 7gb 7gb yellow open video HNhyt9ioSEayAotGVXRCVg 2 1 798369 228155 1.6gb 1.6gb yellow open .kibana lY82G_-XSniyd_bnMOLuQg 1 1 15 1 146.3kb 146.3kb yellow open influecer RQtQWXKIRE2UYyZlCvv7bA 3 1 148526 48641 272.8mb 272.8mb 節點加入集群后，刪除cluster.routing.allocation.enable設置以啟用分片分配並開始使用節點： curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{ "persistent": { "cluster.routing.allocation.enable": "all" } }' 重新打開分片報錯： [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{ > "persistent": { > "cluster.routing.allocation.enable": "true" > } > }' * Hostname was NOT found in DNS cache * Trying 10.10.18.92... * Connected to 10.10.18.92 (10.10.18.92) port 9200 (#0) > PUT /_cluster/settings HTTP/1.1 > User-Agent: curl/7.36.0 > Host: 10.10.18.92:9200 > Accept: */* > Content-Length: 73 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 73 out of 73 bytes < HTTP/1.1 406 Not Acceptable < content-type: application/json; charset=UTF-8 < content-length: 97 < * Connection #0 to host 10.10.18.92 left intact {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406} [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cluster/health?pretty { "cluster_name" : "kp-dev-application", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 14, "active_shards" : 28, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } 安裝新版本中文分詞插件 https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.8.4/elasticsearch-analysis-ik-6.8.4.zip # 解壓在plugin目錄重新啟動elasticsearch即可 cd /opt/es-node/elasticsearch-6.8.4/plugins unzip -d elasticsearch-analysis-ik elasticsearch-analysis-ik-6.8.4.zip ot@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4/plugins]# curl http://10.10.18.92:9200/_xpack?pretty { "build" : { "hash" : "bca0c8d", "date" : "2019-10-16T06:19:49.319352Z" }, "license" : { "uid" : "4de9d1c1-59f6-4dfd-8d48-baefd0a583d0", "type" : "basic", "mode" : "basic", "status" : "active" }, "features" : { "ccr" : { "description" : "Cross Cluster Replication", "available" : false, "enabled" : true }, "graph" : { "description" : "Graph Data Exploration for the Elastic Stack", "available" : false, "enabled" : true }, "ilm" : { "description" : "Index lifecycle management for the Elastic Stack", "available" : true, "enabled" : true }, "logstash" : { "description" : "Logstash management component for X-Pack", "available" : false, "enabled" : true }, "ml" : { "description" : "Machine Learning for the Elastic Stack", "available" : false, "enabled" : true, "native_code_info" : { "version" : "6.8.4", "build_hash" : "93ad89b02ff490" } }, "monitoring" : { "description" : "Monitoring for the Elastic Stack", "available" : true, "enabled" : true }, "rollup" : { "description" : "Time series pre-aggregation and rollup", "available" : true, "enabled" : true }, "security" : { "description" : "Security for the Elastic Stack", "available" : true, "enabled" : false }, "sql" : { "description" : "SQL access to Elasticsearch", "available" : true, "enabled" : true }, "watcher" : { "description" : "Alerting, Notification and Automation for the Elastic Stack", "available" : false, "enabled" : true } }, "tagline" : "You know, for X" } 3.啟用x-pack的密碼驗證 # 生成證書 [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# bin/elasticsearch-certutil ca This tool assists you in the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack. The 'ca' mode generates a new 'certificate authority' This will create a new X.509 certificate and private key that can be used to sign certificate when running in 'cert' mode. Use the 'ca-dn' option if you wish to configure the 'distinguished name' of the certificate authority By default the 'ca' mode produces a single PKCS#12 output file which holds: * The CA certificate * The CA's private key If you elect to generate PEM format certificates (the -pem option), then the output will be a zip file containing individual files for the CA certificate and private key Please enter the desired output file [elastic-stack-ca.p12]: Enter password for elastic-stack-ca.p12 : [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# ls bin config elastic-stack-ca.p12 lib LICENSE.txt logs modules NOTICE.txt plugins README.textile [root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 This tool assists you in the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack. The 'cert' mode generates X.509 certificate and private keys. * By default, this generates a single certificate and key for use on a single instance. * The '-multiple' option will prompt you to enter details for multiple instances and will generate a certificate and key for each one * The '-in' option allows for the certificate generation to be automated by describing the details of each instance in a YAML file * An instance is any piece of the Elastic Stack that requires an SSL certificate. Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats may all require a certificate and private key. * The minimum required value for each instance is a name. This can simply be the hostname, which will be used as the Common Name of the certificate. A full distinguished name may also be used. * A filename value may be required for each instance. This is necessary when the name would result in an invalid file or directory name. The name provided here is used as the directory name (within the zip) and the prefix for the key and certificate files. The filename is required if you are prompted and the name is not displayed in the prompt. * IP addresses and DNS names are optional. Multiple values can be specified as a comma separated string. If no IP addresses or DNS names are provided, you may disable hostname verification in your SSL configuration. * All certificates generated by this tool will be signed by a certificate authority (CA). * The tool can automatically generate a new CA for you, or you can provide your own with the -ca or -ca-cert command line options. By default the 'cert' mode produces a single PKCS#12 output file which holds: * The instance certificate * The private key for the instance certificate * The CA certificate If you specify any of the following options: * -pem (PEM formatted output) * -keep-ca-key (retain generated CA key) * -multiple (generate multiple certificates) * -in (generate certificates from an input file) then the output will be be a zip file containing individual certificate/key files Enter password for CA (elastic-stack-ca.p12) : Please enter the desired output file [elastic-certificates.p12]: Enter password for elastic-certificates.p12 : Certificates written to /opt/es-node/elasticsearch-6.8.4/elastic-certificates.p12 This file should be properly secured as it contains the private key for your instance. This file is a self contained file and can be copied and used 'as is' For each Elastic product that you wish to configure, you should copy this '.p12' file to the relevant configuration directory and then follow the SSL configuration instructions in the product guide. For client applications, you may only need to copy the CA certificate and configure the client to trust this certificate. # 修改config/elasticsearch.yml配置 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /usr/local/elasticsearch/config/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/local/elasticsearch/config/elastic-certificates.p12 # 配置密碼 [es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ bin/elasticsearch-setup-passwords interactive Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana]: Reenter password for [kibana]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]: Changed password for user [apm_system] Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [beats_system] Changed password for user [remote_monitoring_user] Changed password for user [elastic] 密碼：espass [es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ curl --user elastic:espass -XGET 'http://10.10.18.92:9200/_cat/indices' green open channel_rel ZeeBbkogT5KtxzziUYtu_Q 2 1 459528 0 337.7mb 168.8mb green open .security-6 iQHndFBqRe2Ss2o7KMxyFg 1 1 6 0 38.3kb 19.1kb green open .kibana lY82G_-XSniyd_bnMOLuQg 1 1 15 1 292.6kb 146.3kb green open influecer RQtQWXKIRE2UYyZlCvv7bA 3 1 148526 48641 545.6mb 272.8mb green open channel vRFQoIhmT8WmSbDCfph0ag 3 1 53374 0 88.4mb 44.2mb green open channel_list 1dk8uH8bTeikez0lFR2mJg 3 1 5522172 78630 14gb 7gb green open video HNhyt9ioSEayAotGVXRCVg 2 1 798369 228155 3.3gb 1.6gb 升級方式2：完全重啟集群升級 即配置好全新的elasticsearch7.4.2集群，然后把數據恢復到新集群中 下載地址：wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.2-linux-x86_64.tar.gz 老版本的配置 # cms elasticsearch [root@szyyelk01t opt]# egrep -v '^#|^$' elk-master/config/elasticsearch.yml cluster.name: cms-uat-elastic node.name: master path.data: /opt/elk-master/data/data01,/opt/elk-master/data/data02 path.logs: /opt/elk-master/logs bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.90 http.port: 9200 http.cors.enabled: true http.cors.allow-origin: "*" [root@szyyelk01t elk-slave]# egrep -v '^$|^#' slave01/config/elasticsearch.yml cluster.name: cms-uat-elastic node.name: slave01 path.data: /opt/elk-slave/slave01/data/data01,/opt/elk-slave/slave01/data/data02 path.logs: /opt/elk-slave/slave01/logs bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.90 http.port: 8200 discovery.zen.ping.unicast.hosts: ["10.10.18.90"] http.cors.enabled: true http.cors.allow-origin: "*" [root@szyyelk01t elk-slave]# egrep -v '^$|^#' slave02/config/elasticsearch.yml cluster.name: cms-uat-elastic node.name: slave02 path.data: /opt/elk-slave/slave02/data/data01,/opt/elk-slave/slave02/data/data02 path.logs: /opt/elk-slave/slave02/logs bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.90 http.port: 8201 discovery.zen.ping.unicast.hosts: ["10.10.18.90"] http.cors.enabled: true http.cors.allow-origin: "*" # 已經升級的線上配置參考： [root@eus_filmora_db01:/usr/local/elasticsearch-7.4.1]# egrep -v '^$|^#' config/elasticsearch.yml cluster.name: UOS_CLUSTER_ES node.name: uos_node_1 path.data: /data/elasticsearch_data/data path.logs: /data/elasticsearch_data/logs bootstrap.memory_lock: true network.host: 172.20.103.199 http.port: 9200 transport.tcp.port: 9300 node.master: true node.data: true discovery.seed_hosts: ["172.20.103.199:9300", "172.20.73.200:9300", "172.20.73.212:9300"] cluster.initial_master_nodes: ["172.20.103.199", "172.20.73.200", "172.20.73.212"] gateway.recover_after_nodes: 2 transport.tcp.compress: true path.repo: ["/data/bak_es"] xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /usr/local/elasticsearch/config/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /usr/local/elasticsearch/config/elastic-certificates.p12 ############## cms 系統的 elasticsearch 6升級7.4.2 整體策略： 1.升級cms測試環境的es到7.4.2，然后做適配性的開發，再次升級內網的生產環境 --> cms海外環境(找海外業務不繁忙的時候操作，先和開發協商好) 測試環境其他人依賴進行測試，所以升級需要兩套並存，新版本的es7.4.2使用自帶的openjdk13.0 1.配置新版本的elasticsearch使用指定的jdk環境 # vim bin/elasticsear export JAVA_HOME=/opt/elk7_onenode/elasticsearch-7.4.2/jdk export PATH=$JAVA_HOME/bin:$PATH mkdir /opt/elk7_onenode/elasticsearch-7.4.2/data # 主節點配置 [elastic@szyyelk01t elasticsearch-7.4.2]$ more config/elasticsearch.yml cluster.name: cms-uat-elastic7 node.name: cms_node01 node.master: true node.data: true discovery.seed_hosts: ["10.10.18.90:19300", "10.10.18.117:19300"] cluster.initial_master_nodes: ["10.10.18.90"] path.data: /opt/cms_elk7/elasticsearch-7.4.2/data path.logs: /opt/cms_elk7/elasticsearch-7.4.2/logs discovery.zen.minimum_master_nodes: 1 bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.90 http.cors.enabled: true http.cors.allow-origin: "*" transport.tcp.compress: true path.repo: ["/opt/esback/"] gateway.recover_after_nodes: 1 # 增加新的參數head插件可以訪問es http.port: 19200 transport.tcp.port: 19300 gateway.recover_after_time: 8m # 以下配置可以減少當es節點短時間宕機或重啟時shards重新分布帶來的磁盤io讀寫浪費 discovery.zen.fd.ping_timeout: 300s discovery.zen.fd.ping_retries: 8 discovery.zen.fd.ping_interval: 30s discovery.zen.ping_timeout: 180s # 啟用安全認證 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 # 第二個節點配置 [elastic@cms-test:/opt/cms_elk7/elasticsearch-7.4.2]$ more config/elasticsearch.yml cluster.name: cms-uat-elastic7 node.name: cms_node02 node.master: false node.data: true discovery.seed_hosts: ["10.10.18.90:19300", "10.10.18.117:19300"] cluster.initial_master_nodes: ["10.10.18.90"] path.data: /opt/cms_elk7/elasticsearch-7.4.2/data path.logs: /opt/cms_elk7/elasticsearch-7.4.2/logs bootstrap.memory_lock: false bootstrap.system_call_filter: false network.host: 10.10.18.117 http.cors.enabled: true http.cors.allow-origin: "*" transport.tcp.compress: true path.repo: ["/opt/esback/"] gateway.recover_after_nodes: 1 # 增加新的參數head插件可以訪問es http.port: 19200 transport.tcp.port: 19300 gateway.recover_after_time: 8m # 以下配置可以減少當es節點短時間宕機或重啟時shards重新分布帶來的磁盤io讀寫浪費 discovery.zen.fd.ping_timeout: 300s discovery.zen.fd.ping_retries: 8 discovery.zen.fd.ping_interval: 30s discovery.zen.ping_timeout: 180s # 啟用安全認證 xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 # 設置密碼 # 啟用安全認證，只添加這個選項 xpack.security.enabled: true #xpack.security.transport.ssl.enabled: true #xpack.security.transport.ssl.verification_mode: certificate #xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 #xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 elastic密碼： espass 在集群上配置TLS: 如果你在操作單節點ES則可以跳過本內容。 1.生成CA證書 : bin/elasticsearch-certutil ca 將產生新文件 elastic-stack-ca.p12。該 elasticsearch-certutil 命令還會提示你輸入密碼以保護文件和密鑰，請保留該文件的副本並記住其密碼，此處我們設置為空 2.為集群中的每個節點生成證書和私鑰 bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 將產生新文件 elastic-certificates.p12。系統還會提示你輸入密碼，你可以輸入證書和密鑰的密碼，也可以按Enter鍵將密碼留空。默認情況下 elasticsearch-certutil 生成沒有主機名信息的證書，這意味着你可以將證書用於集群中的每個節點，另外要關閉主機名驗證。 將 elastic-certificates.p12 文件復制到每個節點上Elasticsearch配置目錄中 無需將 elastic-stack-ca.p12 文件復制到此目錄。 mkdir config/certs mv elastic-certificates.p12 config/certs/ 配置集群中的每個節點以使用其簽名證書標識自身並在傳輸層上啟用TLS 啟用TLS並指定訪問節點證書所需的信息，將以下信息添加到每個節點的 elasticsearch.yml 文件中： xpack.security.enabled: true 3.設置密碼 # 報錯 [elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Failed to determine the health of the cluster running at http://10.10.18.90:19200 Unexpected response code [503] from calling GET http://10.10.18.90:19200/_cluster/health?pretty Cause: master_not_discovered_exception It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords. It is very likely that the password changes will fail when run against an unhealthy cluster. Do you want to continue with the password setup process [y/N]y Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana]: Reenter password for [kibana]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]: Unexpected response code [503] from calling PUT http://10.10.18.90:19200/_security/user/apm_system/_password?pretty Cause: Cluster state has not been recovered yet, cannot write to the [null] index Possible next steps: * Try running this tool again. * Try running with the --verbose parameter for additional messages. * Check the elasticsearch logs for additional error details. * Use the change password API manually. ERROR: Failed to set password for user [apm_system]. [elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Connection failure to: http://10.10.18.90:19200/_security/_authenticate?pretty failed: Connection refused ERROR: Failed to connect to elasticsearch at http://10.10.18.90:19200/_security/_authenticate?pretty. Is the URL correct and elasticsearch running? [elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Failed to determine the health of the cluster running at http://10.10.18.90:19200 Unexpected response code [503] from calling GET http://10.10.18.90:19200/_cluster/health?pretty Cause: master_not_discovered_exception It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords. It is very likely that the password changes will fail when run against an unhealthy cluster. Do you want to continue with the password setup process [y/N]^C[elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Failed to determine the health of the cluster running at http://10.10.18.90:19200 Unexpected response code [503] from calling GET http://10.10.18.90:19200/_cluster/health?pretty Cause: master_not_discovered_exception It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords. It is very likely that the password changes will fail when run against an unhealthy cluster. Do you want to continue with the password setup process [y/N] 解決辦法：只配置一個主節點：cluster.initial_master_nodes: ["10.10.18.90"] # 錯誤2處理 [2019-11-07T16:12:31,563][INFO ][o.e.c.c.JoinHelper ] [cms_node02] failed to join {cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, xpack.installed=true, ml.max_open_jobs=20}, targetNode={cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, ml.max_open_jobs=20, xpack.installed=true}}]} org.elasticsearch.transport.RemoteTransportException: [cms_node01][10.10.18.90:19300][internal:cluster/coordination/join] Caused by: java.lang.IllegalArgumentException: can't add node {cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, ml.max_open_jobs=20, xpack.installed=true}, found existing node {cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, xpack.installed=true, ml.max_open_jobs=20} with the same id but is a different node instance at org.elasticsearch.cluster.node.DiscoveryNodes$Builder.add(DiscoveryNodes.java:618) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.coordination.JoinTaskExecutor.execute(JoinTaskExecutor.java:147) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.coordination.JoinHelper$1.execute(JoinHelper.java:119) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.4.2.jar:7.4.2] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.4.2.jar:7.4.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] # 拷貝連着目錄 都拷貝，刪除 data 下面的所有文件重啟即可 # 最終密碼配置成功 [elastic@szyyelk01t elasticsearch-7.4.2]$ bin/elasticsearch-setup-passwords interactive Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana]: Reenter password for [kibana]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]: Changed password for user [apm_system] Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [beats_system] Changed password for user [remote_monitoring_user] Changed password for user [elastic] # 查看集群狀態 [elastic@szyyelk01t elasticsearch-7.4.2]$ curl -H "Content-Type: application/json" -u elastic:espass http://10.10.18.90:19200/_cluster/health?pretty { "cluster_name" : "cms-uat-elastic7", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 1, "active_shards" : 2, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } # 查看剛才創建所有索引的備份 # curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107 {"accepted":true} # 恢復全索引快照 #保證elasticsearch用戶擁有快照目錄的權限 chown -R elastic.elastic /opt/esback #創建倉庫 curl -H "Content-Type: application/json" -XPUT -u elastic:espass http://10.10.18.90:19200/_snapshot/backup -d ' { "type":"fs", "settings":{"location":"/opt/esback"} }' #查詢全索引快照備份 $ curl -XGET -u elastic:espass "http://10.10.18.90:19200/_snapshot/backup/_all" | python -m json.tool % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 700 100 700 0 0 160k 0 --:--:-- --:--:-- --:--:-- 170k { "snapshots": [ { "duration_in_millis": 2891, "end_time": "2019-11-07T01:35:03.702Z", "end_time_in_millis": 1573090503702, "failures": [], "include_global_state": true, "indices": [ "support_faq_categorys", "ticket_list", "templates_search", "site_page_search", "support", "templates_page_search", "support_new_articles", "article_version", "blocks_version", "search", "version", "article_search", "templates", "learn", "templates_version", "blocks_search", "templates_page_version" ], "shards": { "failed": 0, "successful": 71, "total": 71 }, "snapshot": "snapshot20191107", "start_time": "2019-11-07T01:35:00.811Z", "start_time_in_millis": 1573090500811, "state": "SUCCESS", "uuid": "0_4SOntVS1GH-7irHjKBMQ", "version": "6.3.2", "version_id": 6030299 } ] } #恢復全索引快照 [elastic@szyyelk01t elasticsearch-7.4.2]$ curl -XPOST -u elastic:espass 'http://10.10.18.90:19200/_snapshot/backup/snapshot20191107/_restore'?wait_for_completion=true {"snapshot":{"snapshot":"snapshot20191107","indices":["templates_page_search","article_search","blocks_version","learn","templates_page_version","templates","version","site_page_search","support_new_articles","support_faq_categorys","search","templates_search","blocks_search","ticket_list","article_version","support","templates_version"],"shards":{"total":71,"failed":0,"successful":71}}} # 查看已經恢復成功 [elastic@szyyelk01t elasticsearch-7.4.2]$ curl -H "Content-Type: application/json" -u elastic:espass http://10.10.18.90:19200/_cat/indices green open templates_page_search tUKh1vaHRla6QamphIByLQ 5 1 104 10 965.3kb 482.6kb green open article_search _LE5n_-KRSGVH6Z3I1YLNQ 5 1 44 2 1.5mb 797.8kb green open blocks_version VRmv8fyESY6iclBYkhKJ_w 5 1 9 0 145.5kb 72.7kb green open learn W4RyJnkrStaRJwQgS4MAug 3 1 89 1 841.6kb 420.8kb green open templates_page_version _hHckKOfRuCPEojviySxVw 5 1 945 0 1.5mb 777kb green open templates 7iJqDoBwTbOEHcyEzPLHbA 5 1 138 0 2mb 1mb green open version mLbfHoA7SAu4RWHSHM3vtw 3 1 1 0 39.9kb 19.9kb green open support_new_articles HvGe-CklRU-iua-_T1pLNA 3 1 1534 170 12mb 6mb green open site_page_search xxk8IetTSr2HF2tEe2Vc1w 5 1 516 2 1.5mb 817.2kb green open .security-7 xdRnCeykQGGPcqM3-_WFCw 1 1 6 0 39.5kb 19.8kb green open search fOteaZd0QfaU_2fKBaWPdA 3 1 0 0 1.5kb 783b green open support_faq_categorys h61nZp5bSQqV1UGVyHL7WA 3 1 0 0 1.5kb 783b green open templates_search ru8oFeQDTtKovOmkjP6A0w 5 1 111 3 1.5mb 802.8kb green open blocks_search 8vMOY6ebTs-4iJIwM2VG0Q 5 1 0 0 2.5kb 1.2kb green open article_version qcF3Nft6QMezKqtPHyYLlA 5 1 344 0 5mb 2.5mb green open ticket_list xpvXuhlqRFq5Y_zugq0qKw 3 1 403 0 2.1mb 1mb green open support LypmJq0pRDy428-TKOy6Yg 3 1 0 0 1.5kb 783b green open templates_version gI28sYWJT3GVgfBeyJhSLg 5 1 220 0 4.2mb 2.1mb