ES版本升級工作
當前版本：v5.4.1 目標版本：v7.3.2
注意事項：
備份數據
修改配置
優先升級主節點數最少的節點
必須等集群狀態恢復green后再升級下一個節點
官方提供的三種升級方案：
-
方案一：Rolling Upgrade 滾動升級，每次升級一個節點，服務不會宕 Rolling upgrade to 5.6 Rolling upgrade to 6.8 Rolling upgrade to 7.3.2
-
方案二：Full cluster restart upgrade If you are running a version prior to 6.0, upgrade to 6.8 and reindex your old indices or bring up a new 7.3.2 cluster and reindex from remote. 由於版本低於6.0：需要搭建好V7.3.2的集群(建好索引)，再reindex from remote Reindex from remote You must set up the destination index before calling _reindex. Reindex does not copy the settings from the source index. Mappings, shard counts, replicas, and so on must be configured ahead of time. -
方案三：Reindex befor upgrading Create a new 7.3.2 cluster and Reindex from remote由於當前使用的版本相對新版本較低，方案二和方案三的動作是一樣的，都是先搭建好一個V7.3.2的集群，重建好所有索引，再執行Reindex from remote（遠程拉取數據）
方案一測試：
測試集群：10.0.6.121,10.0.6.101,10.0.6.140
測試數據：通過es-analysor項目寫一份學習統計數據到測試集群（數據為持續寫入狀態）
采用方案一進行升級，自定義插件沒有使用，中間版本不進行升級官方文檔
使用測試集群，暫不備份數據（快照備份數據）官方文檔
一個快照表示的是這個索引在快照被創建時間點的索引視圖，所以在索引過程開始之后被添加到索引中的記錄不會出現在快照中。
快照是增量的，可以包含在多個ES版本中創建的索引。如果在一個快照中的任何索引時在不兼容的ES版本中創建的，你將不能恢復該快照。
恢復操作可以在正常運行的集群上執行。已存在的索引只能在關閉狀態下才能恢復，並且要跟快照中索引擁有相同數目的分片。還原操作自動打開關閉狀態的索引，如果被還原索引在集群不存在，將創建新索引
備份數據神坑：
1.需要賦予用戶權限；
2.各個節點es用戶UID和GID需要一致,否則創建倉庫的時候會連接失敗
elasticsearch:x:501:501::/home/elasticsearch:/bin/bash
准備好V5.6、V6.8、V7.3.2安裝包,下載地址
更改配置，與V5.4.1一致，已知需要配置elasticsearch.yml，jvm.options(測試集群出現OOM)
PUT _snapshot/my_backup
{
"type": "fs",
"settings": {
"location": "/mnt/mfs/es-bak"
}
}
POST _snapshot/my_backup/
{
"type": "fs",
"settings": {
"location": "/mnt/mfs/es-bak",
"max_snapshot_bytes_per_sec" : "50mb",
"max_restore_bytes_per_sec" : "50mb"
}
}
1.升級到V5.6.0
分配elasticsearch用戶權限：chown -R elasticsearch /opt/servers/elasticsearch-5.6.0
1.Disable shard allocation 因為節點重啟很快就會返回集群，避免大量I/O
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "none"
}
}
2.Stop non-essential indexing and perform a synced flush (Optional). A synced flush request is a “best effort” operation
POST _flush/synced
檢查是否用了supervisor
安裝了superviso：
修改配置，用http://es-host:7777來啟動節點
未安裝supervisor：
kill -9 pid 殺掉低版本es進程
集群狀態變為yellow
啟動當前節點V5.6.0服務
Reenable shard allocation
PUT _cluster/settings
{
"transient": {
"cluster.routing.allocation.enable": "all"
}
}
新加入的高版本節點開始分配shard，集群狀態恢復為green
repeat 直到所有節點迭代完畢
集群狀態恢復green
至此測試集群完成從V5.4.1到V5.6.0升級
優勢：平滑迭代，服務不會宕機，應用無感，沒有重建索引的過程
弊端：每個節點迭代后需要等集群恢復到green,正式集群數據量大，這個恢復過程會更長，由於版本關系Rollnig upgrade 到V7.3.2整個過程會很漫長。
2.升級到V6.8.0
首先確認以下兩個注意事項：
If the Elasticsearch security features are enabled on your 5.x cluster, before you can do a rolling upgrade you must encrypt the internode-communication with SSL/TLS, which requires a full cluster restart. For more information about this requirement and the associated bootstrap check, see SSL/TLS check.
The format used for the internal indices used by Kibana and X-Pack has changed in 6.x. When upgrading from 5.6 to 6.x, these internal indices have to be upgraded before the rolling upgrade procedure can start. Otherwise the upgraded node will refuse to join the cluster.
- Disable shard allocation.
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": "primaries"
}
}
2.Stop non-essential indexing and perform a synced flush. (Optional)
POST _flush/synced
3.Stop any machine learning jobs that are running.
4.Shut down a single node.
5.Upgrade the node you shut down.
If you are upgrading from a version prior to 6.3 and use X-Pack then you must remove the X-Pack plugin before upgrading with bin/elasticsearch-plugin remove x-pack. As of 6.3, X-Pack is included in the default distribution so make sure to upgrade to that one. If you upgrade without removing the X-Pack plugin first the node will fail to start. If you did not remove the X-Pack plugin and the node fails to start then you must downgrade to your previous version, remove X-Pack, and then upgrade again. In general downgrading is not supported but in this particular case it is.
6.Upgrade any plugins.
7.Start the upgraded node.
出現第一個異常：
Caused by: java.lang.IllegalArgumentException: the [action.auto_create_index] setting value [false] is too restrictive. disable [action.auto_create_index] or set it to [.watches,.triggered_watches,.watcher-history-*]
解決辦法：注釋該配置
出現的第二個異常：
max number of threads [2048] for user [elasticsearch] is too low, increase to at least [4096]
解決辦法：
vi /etc/security/limits.d/90-nproc.conf
* soft nproc 4096
root soft nproc unlimited
出現第三個異常：
客戶端版本過低導致，需要升級客戶端
[2019-11-11T18:38:30,591][WARN ][o.e.t.TcpTransport ] [estest-log-node-101] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.0.6.101:9300, remoteAddress=/10.0.6.147:35746}], closing connection
java.lang.IllegalStateException: Received message from unsupported version: [5.4.1] minimal compatible version is: [5.6.0]
8.Reenable shard allocation.
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.enable": null
}
}
9.Wait for the node to recover.
10.Repeat
11.Restart machine learning jobs.
Elasticsearch 7.85集群 V5.4.1滾動升級到V5.6.0的流程
共八個節點：
10.0.7.82,10.0.7.83,10.0.7.86,10.0.7.87,10.0.7.88,10.0.7.54,10.0.7.85
10.0.7.81上無ES進程，不升級
優先升級primary sahrd較少的節點
- 准備好安裝包，上傳到服務器，更改配置，包含elasticsearch.yml（復制），jvm.options（復制），是否還需要配置其他地方？
- 快照備份數據
- 節點升級到V5.6.0
- 整個集群升級完成后刪除快照
- 升級160集群到V5.6.0（再續）
- 升級客戶端---還需要單獨再做測試
風險評估：
1.單個節點升級失敗
解決方案：刪除節點對應的es-data下的數據，開啟V5.4.1，重新加入集群
2.集群的數據被影響
解決方案：從快照恢復數據
恢復期間需要關停索引，對應索引不可用，數據為快照生成時的數據，意味着會丟失一部分數據