Elasticsearch使用小結之冷熱分離

Elasticsearch使用小結之冷熱分離

索引遷移

索引setting中的index.routing.allocation.exclude和index.routing.allocation.include可以用於指定索引分配與哪些節點。同時，這兩個配置是可以在index存在的時候修改的，我們可以通過修改這兩個配置的方式來遷移索引。
比如：
ES集群存在5個節點，ip分別為：
    192.168.1.101， 
    192.168.1.102，
    192.168.1.103，
    192.168.1.104，
    192.168.1.105
1. 我們先創建一個索引，使其分配在192.168.1.101，192.168.1.102上
PUT test_index
{
    "settings":{
        "index":{
            "routing.allocation.exclude._ip":"192.168.1.101,192.168.1.102"
        }
    }
}
2. 使用cat shards API查看test_index的分片分布
GET _cat/shards/test_index
返回如下：
    test_index                   2  p STARTED            0     162b 192.168.1.105  Z1SgiFF
    test_index                   0  p STARTED            0     162b 192.168.1.103  qEu6eMp
    test_index                   3  p STARTED            0     162b 192.168.1.104   xlTltSO
    test_index                   5  p STARTED            0     162b 192.168.1.104  xlTltSO
    test_index                   4  p STARTED            0     162b 192.168.1.105   Z1SgiFF
    test_index                   1  p STARTED            0     162b 192.168.1.103  qEu6eMp

3. 修改配置
PUT test_index/_settings
{
    "settings":{
        "index":{
            "routing.allocation.exclude._ip":"192.168.1.103,192.168.1.104,192.168.1.105",
            "number_of_shards": 6
        }
    }
}
4. 查看分片
GET _cat/shards/test_index
    test_index                   2  p STARTED            0     162b 192.168.1.102  9OHYhSa
    test_index                   4  p STARTED            0     162b 192.168.1.101   -RdAJHx
    test_index                   0  p STARTED            0     162b 192.168.1.102  9OHYhSa
    test_index                   5  p STARTED            0     162b 192.168.1.104  xlTltSO -> 192.168.1.101   -RdAJHx
    test_index                   3  p STARTED            0     162b 192.168.1.101   -RdAJHx
    test_index                   5  p STARTED            0     162b 192.168.1.102  9OHYhSa

節點tag

在節點啟動時，在192.168.1.101和192.168.1.102上指定了
bin/elasticsearch -Enode.attr.rack=rack1
其余三台指定了
bin/elasticsearch -Enode.attr.rack=rack2
使用 GET _nodes/stats 查看：
{
  "_nodes": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "cluster_name": "elasticsearch",
  "nodes": {
    "-RdAJHxHTf2kHNyfUwHHBw": {
      "timestamp": 1559467108509,
      "name": "-RdAJHx",
      "transport_address": "192.168.1.101:9200",
      "host": "192.168.1.101",
      "ip": "192.168.1.101:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack1"
      }
      ....
    },
    "9OHYhSaxRgKKu_H0q18KyA": {
      "timestamp": 1559467108628,
      "name": "9OHYhSaxRgKKu_H0q18KyA",
      "transport_address": "192.168.1.102:9200",
      "host": "192.168.1.102",
      "ip": "192.168.1.102:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack1"
      }
      ....
    },
    "qEu6eMp9SEK7mlB9HmjCFA": {
      "timestamp": 1559467110507,
      "name": "qEu6eMp",
      "transport_address": "192.168.1.103:9200",
      "host": "192.168.1.103",
      "ip": "192.168.1.103:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack2"
      }
      ....
    },
    "xlTltSOCQmibG9HICbnQyw": {
      "timestamp": 1559467110507,
      "name": "xlTltSO",
      "transport_address": "192.168.1.104:9200",
      "host": "192.168.1.104",
      "ip": "192.168.1.104:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack2"
      }
      ....
    },
    "Z1SgiFFrQMqrPKKGir1EXg": {
      "timestamp": 1559467110507,
      "name": "Z1SgiFF",
      "transport_address": "192.168.1.105:9200",
      "host": "192.168.1.105",
      "ip": "192.168.1.105:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack2"
      }
      ....
    }
我們也可以通過指定attributes來控制索引的遷移，這里需要配置index.routing.allocation.exclude.tag或者index.routing.allocation.include.tag

冷熱分離

結合上述，我們可以很方便的進行冷熱分離，先將節點分類，划分為兩塊，分別用於存儲熱數據和冷數據，在新建索引時指定索引的index.routing.allocation.exclude.tag為冷節點(或者index.routing.allocation.include.tag為熱節點，效果一樣)，然后在索引不再有數據寫入時指定index.routing.allocation.exclude.tag為熱節點(與前面用的對應, 建議統一使用exclude或者include，兩者混用時，由於兩者不能覆蓋，需要注意置空)
示例：
在上述的ES中，我們將rack1的作為熱節點，rack2的作為冷節點。
1. PUT test_index
   {
      "settings":{
        "routing.allocation.exclude.tag":"rack2",
            "number_of_shards": 6
      }
   }
2. 數據寫入
3. PUT test_index/_settings
   {
      "settings":{
        "routing.allocation.exclude.tag":"rack1"
      }
   }