elastchsearch基於kibana Index lifecycle Managent管理模塊做生命周期管理

Index lifecycle Managent 是ES6.6后才提供基於X-Pack Basic License（可以免費使用的一個功能）

大致作用如下入,可以根據時間,條目數,數據塊大小進行數據生命周期管理,具體好處還很多,這里記錄下用法。

 

 

 一.創建生命周期的規則,可以通過代碼進行創建,也可以通過kibana界面進行創建。

　　1.界面模式

　　可以填入控制的條件來制定周期規則

 

 

 

 

 

 　　2.代碼模式

　　eg:其實和代碼和界面模式是一致的，代碼創建完依然可以kibana中進行查看和更改

PUT /_ilm/policy/log_ilm_policy2
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_docs": 5
          }
        }
      },
      "warm": {
        "min_age": "10s",
        "actions": {
          "allocate": {
            "include": {
              "box_type": "warm"
            }
          }
        }
      },
     "delete": {
        "min_age": "60s",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

 二.設置索引模板 對所有lm_index-模板索引都做一個設定

PUT /_template/log_ilm_template
{
  "index_patterns": [
    "ilm_index-*"
    ],
    "settings": {
      "index":{
        "lifecycle":{
          "name":"log_ilm_policy2",
          "rollover_alias":"ilm_alias"#這個別名記得和索引別名用的要一致不然會報illegal_argument_exception: index.lifecycle.rollover_alias [ilm_alias] does not point to index [ilm_index-000001]

        },
        "routing":{
          "allocation":{
            "include":{
              "box_type":"hot"
            }
          }
        },
        "number_of_shards":"1",
        "number_of_replicas":"0"
      }
    },
    "mappings": {
         "_source": {
            "enabled": true
         },
         "properties": {
            
        }
    }
}

 

三.建立索引

#創建索引
#別名為 myindex
#允許索引被寫入數據
PUT ilm_index-000001 #索引名字必須以數字結尾不然會報illegal_argument_exception: index name [ilm_index-hot] does not match pattern '^.*-\d+$'
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"aliases": {
"ilm_alias":{ #這里的名字要和上面的模板一致
"is_write_index": true
}
}

}

#插入測試數據

POST /ilm_index-000002/_doc
{
"log":"something"
}

 

#1.默認是10分鍾
PUT _cluster/settings
{
"persistent": {
"indices.lifecycle.poll_interval": "1s"
}
}

 

到此只要條件達到更換生命周期和kibana一旦觸發,數據就會進行轉移。比如滿足60條或者時間超過60秒

 

 

 

上面是自測通過,以下是網上查找親測的可用模板

Creating an alias with the lifecycle policy is a 3 step process.

Elastic provides a great tutorial

In short:

Create a lifecycle policy that defines the appropriate phases and actions.
Create an index template to apply the policy to each new index.
Bootstrap an index as the initial write index.
I think you are creating the template AFTER you create the first index. You should first create the ilm, after that the template where you specify what ilm policy you want to use for the indexes and finally create the first index (bootstrap).
var indexName = "index_name";
var indexPattern = $"{indexName}-*";
var aliasName = $"{indexName}-alias";
var policyName = $"{indexName}-policy";
var firstIndexName = $"{indexName}-000001";

PUT _ilm/policy/index_name-policy
{
    "policy": {
        "phases": {
            "hot": {
                "actions": {
                    "rollover": {
                        "max_size": "5gb",
                        "max_docs": 10000,
                        "max_age":"2d"
                    }
                }
            },
            "warm": {
                "min_age": "5d",
                "actions": {
                }
            },
            "delete": {
                "min_age": "10d",
                "actions": {
                    "delete": {}
                }
            }
        }
    }
}

PUT _template/index_name-template
{
    "index_patterns": ["{{.IndexPattern}}"],
    "settings": {
        "index.number_of_shards": "1",
        "index.number_of_replicas": "1",
        "index.lifecycle.name": "{{.PolicyName}}", 
        "index.lifecycle.rollover_alias": "{{.AliasName}}" 
    },
    "mappings": {
         "_source": {
            "enabled": true
         },
         "properties": {
            {{Properties}}
        }
    }
}

PUT index_name-000001
{
   "aliases": {
        "{{.AliasName}}":{
            "is_write_index": true 
        }
    }
}

　　

模板解說：

PUT /_ilm/policy/fact_terminal_phone_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "300s"
}
}
},
"warm": {
"min_age": "120s",
"actions": {
"allocate": {
"include": {
"box_type": "warm"
}
}
}
},
"delete": {
"min_age": "180s",
"actions": {
"delete": {}
}
}
}
}
}

第一個300秒會進行新的索引創建,第二個重新計算開始120秒會進行索引遷移到冷節點，數據遷移到warm節點（這個120秒和前面300秒不重疊），最后在經過60秒刪除。這個是和120重疊的，所以要比120大。