Elasticsearch數據備份與恢復(基於HDFS)

1.(所有機子上)安裝es hdfs倉庫插件repository-hdfs

# repository-hdfs一定要和es版本匹配
# 在線
bin/elasticsearch-plugin install repository-hdfs

# 離線
bin/elasticsearch-plugin install file:///xxx/repository-hdfs-x.x.x.zip.zip

2. (所有機子上)設置ES-HDFS倉庫安全策略

plugins/repository-hdfs/plugin-security.policy追加

permission java.lang.RuntimePermission "accessDeclaredMembers";
permission java.lang.RuntimePermission "getClassLoader";
permission java.lang.RuntimePermission "shutdownHooks";
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
permission javax.security.auth.AuthPermission "doAs";
permission javax.security.auth.AuthPermission "getSubject";
permission javax.security.auth.AuthPermission "modifyPrivateCredentials";
permission java.security.AllPermission;
permission java.util.PropertyPermission "*", "read,write";
permission javax.security.auth.PrivateCredentialPermission "org.apache.hadoop.security.Credentials * \"*\"", "read";

config/jvm.options追加

Djava.security.policy=file:////xxx/plugins/repository-hdfs/plugin-security.policy

3.(所有機子上)設置JAVA安全管理器

$JAVA_HOME/jre/lib/security/java.policy追加

permission java.security.AllPermission;

重啟Es集群

4. (一台機子上)創建HDFS倉庫

# 創建遠程倉庫,目錄為/user/xxx/es/repository/es_hdfs_repository
# 在HDFS 50070查看
# uri需要是active的namenode節點
curl -XPUT 'http://host:esPort/_snapshot/es_hdfs_repository' -d '
{
    "type":"hdfs",
    "settings":{
        "uri":"hdfs://host:hdfsPort/",
        "path":"es/repository/es_hdfs_repository"
    }
}'

5. (一台機子上)備份

# 對所有open的index進行備份,創建一個snapshot_1的備份
# wait_for_completion會堵塞到備份完成
curl -XPUT 'http://host:esPort/_snapshot/es_hdfs_repository/snapshot_1?wait_for_completion=true'

# 指定索引備份
curl -XPUT 'http://host:esPort/_snapshot/es_hdfs_repository/snapshot_1?wait_for_completion=true' -d '
{
    "indices":["index1","index2"],
    "ignore_unavailable":true,
    "include_global_state":false,
    "partial":true
}'

# 查看備份進度和結果
curl -XGET 'http://host:esPort/_snapshot/es_hdfs_repository/snapshot_1?pretty'

6.(一台機子上)恢復

# 恢復指定索引
#include_global_state=true 集群狀態也備份
#ignore_unavailable 部分索引不存在時也通過,否則認為不成功
#partial=true 部分失效也可以通過,否則認為不成功
curl -XPOST 'http://host:esPort/_snapshot/es_hdfs_repository/snapshot_1/_restore?wait_for_completion=true' -d '
{
    "indices":["redisindex"],
    "ignore_unavailable":true,
    "include_global_state":false,
    "partial":true
}'

# 恢復snapshot_1全部索引
curl -XPOST 'http://host:esPort/_snapshot/es_hdfs_repository/snapshot_1/_restore?wait_for_completion=true'

#恢復進度和結果
curl -XGET 'http://host:esPort/index1/_recovery?pretty'

Tips:

恢復之前最好先把所有close,防止恢復期間還有新數據產生

如果是不存在的索引,不在考慮范圍

7.其他

# 關閉索引
curl -XPOST 'hhtp://host:esPort/eventindex/_close'

# 打開索引
curl -XPOST 'http://host:esPort/eventindex/_open'

# 刪除備份
# 備份是增量備份的,所以不要輕易手動刪,否則會出現索引損壞而無法恢復
# 正確的姿勢
curl -XDELETE 'http://host:esPort/_snapshot/es_hdfs_repository/snapshot_1'