ELK數據遷移,ES快照備份遷移

　　通過curl命令或者kibana快照備份，恢復的方式進行數據遷移

環境介紹

　　之前創建的ELK 因為VPC環境的問題，需要對ELK從新部署，但是還需要保留現有的數據，於是便有了這篇文檔。

10.0.20.74 old es

10.0.10.229 new es

方法一、命令行操作

一、創建快照存儲庫

1.修改老的es配置文件，用於快照存儲庫。

[root@elk-server /home]# cat /etc/elasticsearch/elasticsearch.yml |grep  path.repo
path.repo: /home/esdata  

 

如果你也是集群模式的話 必須要創建並掛載共享目錄

2、分配權限並重啟es

[root@elk-server /home]# chown -R elasticsearch:elasticsearch  esdata
[root@elk-server /home]# systemctl  restart elaseicesarch
[root@elk-server /home]# ll |grep esdata
drwxr-xr-x  3 elasticsearch elasticsearch        134 Jun  8 08:13 esdata

 

3、創建ES本地存儲庫

curl -XPUT -uxuewenlong:bsh@123  http://10.0.20.74:9200/_snapshot/my_back -H "Content-Type: application/json"  -d '
{
          "type": "fs",
          "settings": {
              "location": "/home/esdata/",   #存儲庫位置，要和配置文件里一致用戶組是es
              "max_restore_bytes_per_sec":"10mb",          #恢復最大速率
              "compress":"true",                           #是否壓縮
              "max_snapshot_bytes_per_sec":"10mb",         #創建最大速率
              "chunk_size":"100mb"                          #壓縮塊大小
          }
      }'

 

4、驗證存儲庫

curl -XGET -uxuewenlong:bsh@123 http://10.0.20.74:9200/_snapshot/
{
    "my_back": {
        "type": "fs",
        "settings": {
            "chunk_size": "100M",
            "location": "/home/esdata/",
            "max_restore_bytes_per_sec": "10mb",
            "compress": "true",
            "max_snapshot_bytes_per_sec": "10mb"
        }
    }
}

 

二、備份索引

1、備份指定的索引

curl -XPUT -uxuewenlong:bsh@123 http://10.0.20.74:9200/_snapshot/my_back/esdata?wait_for_completion=true -H "Content-Type: application/json"  -d '
   {
 "indices": "miniprogram-prod*"   #寫入需要備份的索引,多個所以以,分隔 如"index1,index2" 支持正則    
   }'

 

2、備份當前庫所有索引

curl -XPUT -uxuewenlong:bsh@123 http://10.0.20.74:9200/_snapshot/my_back/esdata?wait_for_completion=true -H "Content-Type: application/json"

 

3、驗證備份快照

curl -XGET -uxuewenlong:bsh@123 http://10.0.20.74:9200/_snapshot/my_back/esdata
{
    "snapshots": [{
        "snapshot": "esdata",
        "uuid": "t3BNDuiCRwK_uAIz71RCbQ",
        "version_id": 7050099,
        "version": "7.5.0",
        "indices": ["miniprogram-prod-aspect-az.log.2020.01", "miniprogram-prod-aspect-az.log.2020.03", "miniprogram-prod-aspect-az.log.2020.06", "miniprogram-prod-access-az.log.2019.12", "miniprogram-prod-aspect-az.log.2020.02", "miniprogram-prod-aspect-bz.log.2020.02", "miniprogram-prod-access-az.log.2020.06", "miniprogram-prod-aspect-az.log.2020.05", "miniprogram-prod-aspect-az.log.2020.04", "miniprogram-prod-access-az.log.2020.04", "miniprogram-prod-access-az.log.2020.03", "miniprogram-prod-aspect-bz.log.2020.05", "miniprogram-prod-access-az.log.2020.01", "miniprogram-prod-access-az.log.2020.05", "miniprogram-prod-access-bz.log.2020.03", "miniprogram-prod-aspect-bz.log.2020.04", "miniprogram-prod-access-bz.log.2020.06", "miniprogram-prod-aspect-bz.log.2020.03", "miniprogram-prod-access-bz.log.2020.02", "miniprogram-prod-access-az.log.2020.02", "miniprogram-prod-aspect-bz.log.2020.01", "miniprogram-prod-access-bz.log.2020.01", "miniprogram-prod-access-bz.log.2020.05", "miniprogram-prod-access-bz.log.2020.04", "miniprogram-prod-aspect-bz.log.2020.06"],
        "include_global_state": true,
        "metadata": {
            "policy": "miniesdata"
        },
        "state": "SUCCESS",
        "start_time": "2020-06-08T07:48:00.006Z",
        "start_time_in_millis": 1591602480006,
        "end_time": "2020-06-08T08:01:57.507Z",
        "end_time_in_millis": 1591603317507,
        "duration_in_millis": 837501,
        "failures": [],
        "shards": {
            "total": 25,
            "failed": 0,
            "successful": 25
        }
    }]
}

 

三、遷移到新的es集群中

[root@elk-server /home]# zip -r esdata.zip  esdata
[root@elk-server /home]# scp /home/esdata.zip xuewenlong@10.0.10.229:/home

 

1、登錄新es集群操作

解壓es數據，分配權限

[root@ip-10-0-10-229 home]# unzip  esdata.zip
[root@ip-10-0-10-229 home]# ll
drwxr-xr-x. 9 root          root          115 Jun  9 00:58 bsh
drwx------. 3 ec2-user      ec2-user      113 Jun  8 08:25 ec2-user
drwxr-xr-x. 3 xuewenlong    xuewenlong    134 Jun  9 01:31 esdata
[root@ip-10-0-10-229 home]#chown  -R elasticsearch:elasticsearch esdata

 

2、修改es配置文件

[root@ip-10-0-10-229 home]# cat /etc/elasticsearch/elasticsearch.yml |grep  path.repo
path.repo: /home/esdata

 

3、創建一樣的存儲庫

**路徑以及權限一定要一樣

curl -XPUT -uxuewenlong:bsh@123  http://10.0.10.229:9200/_snapshot/my_back -H "Content-Type: application/json"  -d '
{
          "type": "fs",
          "settings": {
              "location": "/home/esdata/",   #存儲庫位置，要和配置文件里一致用戶組是es
              "max_restore_bytes_per_sec":"10mb",          #恢復最大速率
              "compress":"true",                           #是否壓縮
              "max_snapshot_bytes_per_sec":"10mb",         #創建最大速率
              "chunk_size":"100mb"                          #壓縮塊大小
          }
      }'

 

4、驗證備份是否存在

curl -XGET -uelastic:bsh@123 http://10.0.10.229:9200/_snapshot/my_back/esdata
    "snapshots": [{
        "snapshot": "esdata",
        "uuid": "t3BNDuiCRwK_uAIz71RCbQ",
        "version_id": 7050099,
        "version": "7.5.0",
        "indices": ["miniprogram-prod-aspect-az.log.2020.01", "miniprogram-prod-aspect-az.log.2020.03", "miniprogram-prod-aspect-az.log.2020.06", "miniprogram-prod-access-az.log.2019.12", "miniprogram-prod-aspect-az.log.2020.02", "miniprogram-prod-aspect-bz.log.2020.02", "miniprogram-prod-access-az.log.2020.06", "miniprogram-prod-aspect-az.log.2020.05", "miniprogram-prod-aspect-az.log.2020.04", "miniprogram-prod-access-az.log.2020.04", "miniprogram-prod-access-az.log.2020.03", "miniprogram-prod-aspect-bz.log.2020.05", "miniprogram-prod-access-az.log.2020.01", "miniprogram-prod-access-az.log.2020.05", "miniprogram-prod-access-bz.log.2020.03", "miniprogram-prod-aspect-bz.log.2020.04", "miniprogram-prod-access-bz.log.2020.06", "miniprogram-prod-aspect-bz.log.2020.03", "miniprogram-prod-access-bz.log.2020.02", "miniprogram-prod-access-az.log.2020.02", "miniprogram-prod-aspect-bz.log.2020.01", "miniprogram-prod-access-bz.log.2020.01", "miniprogram-prod-access-bz.log.2020.05", "miniprogram-prod-access-bz.log.2020.04", "miniprogram-prod-aspect-bz.log.2020.06"],
        "include_global_state": true,
        "metadata": {
            "policy": "miniesdata"
        },
        "state": "SUCCESS",
        "start_time": "2020-06-08T07:48:00.006Z",
        "start_time_in_millis": 1591602480006,
        "end_time": "2020-06-08T08:01:57.507Z",
        "end_time_in_millis": 1591603317507,
        "duration_in_millis": 837501,
        "failures": [],
        "shards": {
            "total": 25,
            "failed": 0,
            "successful": 25
        }
    }]
}

 

四.數據恢復到新集群中

1、恢復單條索引

curl -XPOST -uxuewenlong:bsh@123 http://10.0.10.229:9200/_snapshot/my_back/esdata/_restore -H "Content-Type: application/json" -d '
   {
 "miniprogram-prod*"
   }'

 

2、恢復全部索引

curl -XPOST -uxuewenlong:bsh@123 http://10.0.10.229:9200/_snapshot/my_back/esdata/_restore -H "Content-Type: application/json"

 

3、刪除快照

curl -XDELETE -uxuewenlong:bsh@123  -uxuewenlong:bsh@123 http://10.0.10.229:9200/_snapshot/my_back/esdata/

 

方法二、kibana操作

1.修改老的es配置文件，用於快照存儲庫。

[root@elk-server /home]# cat /etc/elasticsearch/elasticsearch.yml |grep  path.repo
path.repo: /home/esdata 

 

一、創建快照存儲庫

1、在舊的kibana上創建快照存儲庫

2、在舊的kibana驗證存儲庫

二、創建策略備份索引

1、在舊的kibana創建策略

支持cron定時任務 還是比較方便的

 

 

 

2、在舊的kibana進行數據備份

保存完策略之后點擊小按鈕就會立馬進行備份

3、在舊的kibana備份完查看備份

三、遷移到新的es集群中

1、登陸服務器old es 服務器講備份數據打包 移至新的es服務器

[root@elk-server /home]# zip -r esdata.zip  esdata
[root@elk-server /home]# scp /home/esdata.zip xuewenlong@10.0.10.229:/home

 

2、登錄新es集群操作解壓es數據，分配權限

[root@ip-10-0-10-229 home]# unzip  esdata.zip
[root@ip-10-0-10-229 home]# ll
drwxr-xr-x. 9 root          root          115 Jun  9 00:58 bsh
drwx------. 3 ec2-user      ec2-user      113 Jun  8 08:25 ec2-user
drwxr-xr-x. 3 xuewenlong    xuewenlong    134 Jun  9 01:31 esdata
[root@ip-10-0-10-229 home]#chown  -R elasticsearch:elasticsearch esdata

 

3、在新的kibana創建存儲庫

4、在新的kibana刷新瀏覽器 查看恢復快照

 

 

 

5、在新的kibana查看恢復狀態

 

 

四、驗證es數據

在新的kibana 上創建索引，數據是存在的

至此es數據遷移成功