012 Ceph多區域網關

一、基本概念

1.1 多區域概念

Multi-Size功能是從J版本開始的。一個single zone配置通常由一個zone group組成，該zone group包含一個zone和多個用於負載均衡的RGW實例。從K版本開始，ceph為RGW提供了Multi-Size的配置選項

1.2 多區域相關術語

區域（zone）: 一個ceph集群可以包含多個區域，一個區域只屬於一個集群，一個區域可以有多個RGW

區域組（zonegroup）：由一個或多個區域組成，包含一個主區域（master zone），其他區域稱為Secondary Zone，區域組內的所有區域之間同步數據

域（realm）: 同一個或多個區域組組成，包含一個主區域組，其他都次區域組。域中的所有rados網關都從位於主區域組和主區域中的rados網關拉取配置

注意： master zone group中的master zone處理所元數據更新，因此創建用戶、bucket等操作都必須經由master zone

1.3 多區域網關配置架構

single-zone：一個realm中只有一個zonegroup和一個zone，可以有多個RGW

multi-zone：一個relam中只有一個zonegroup，但是有多個zone。一個realm中存儲的數據復制到該zonegroup中的所有zone中

multi-zonegroup：一個realm中有多個zonegroup，每個zonegroup中又有一個或多個zone

multi-realms：多個realm

 

兩個集群配置兩個zone，實現兩個zone之間的數據同步：

periods和Epochs：

每個zone有關聯的period，每個period有關聯的epoch。

period用於跟蹤zone、zone group和realm的配置狀態

epoch用於跟蹤zone period的配置變更的版本號

每個period具有唯一ID，含有zone配置，並且知道其先前period的id

在更改zone的配置時，需要更新zone的當前period

1.4 多區域同步流程

RGW在所有zone group集合之間同步元數據和數據操作。元數據操作與bucket相關：創建、刪除、啟用和禁用版本控制、管理用戶。meta master位於master zone group中的master zone，負責管理元數據更新

多區域配置后處於活躍狀態時，RGW會在master和secondary區域之間執行一次初始的完整同步。隨后的更新是增量更新

當RGW將數據寫入zone group的任意zone時，它會在該zone group的所有其他zone之間同步這一數據

當RGW同步數據時，所有活躍的網關會更新數據日志並通知其他網關

當RGW網關因用戶操作而同步元數據時，主網關會更新元數據日志並通知其他RGW網關

兩個集群配置兩個zone，實現兩個zone之間的數據同步：periods和Epochs：每個zone有關聯的period，每個period有關聯的epoch。period用於跟蹤zone、zone group和realm的配置狀態epoch用於跟蹤zone period的配置變更的版本號每個period具有唯一ID，含有zone配置，並且知道其先前period的id在更改zone的配置時，需要更新zone的

當前period多區域同步流程：RGW在所有zone group集合之間同步元數據和數據操作。元數據操作與bucket相關：創建、刪除、啟用和禁用版本控制、管理用戶。meta master位於master zone group中的master zone，負責管理元數據更新多區域配置后處於活躍狀態時，RGW會在master和secondary區域之間執行一次初始的完整同步。隨后

的更新是增量更新當RGW將數據寫入zone group的任意zone時，它會在該zone group的所有其他zone之間同步這一數據當RGW同步數據時，所有活躍的網關會更新數據日志並通知其他網關當RGW網關因用戶操作而同步元數據時，主網關會更新元數據日志並通知其他RGW網關

二、配置

2.1 查看

[root@ceph5 ~]# radosgw-admin realm list

[root@ceph5 ~]# radosgw-admin zonegroup list

[root@ceph5 ~]# radosgw-admin zone list

[root@ceph5 ~]# radosgw-admin zone list --rgw-zonegroup default

2.2 創建realm

[root@ceph5 ~]# radosgw-admin realm create --rgw-realm hubei  --default

2.3 創建zonegroup

[root@ceph5 ~]# radosgw-admin zonegroup create --rgw-realm hubei --rgw-zonegroup  fancheng --default  --master

2.4 修改名

[root@ceph5 ~]# radosgw-admin zonegroup rename  --rgw-realm hubei --rgw-zonegroup fancheng --zonegroup-new-name xiangyang

[root@ceph5 ~]# radosgw-admin zonegroup  get --rgw-zonegroup xiangyang

2.5 刪除一個zonegroup

[root@ceph5 ~]# radosgw-admin realm remove --rgw-realm hubei --rgw-zonegroup xiangyang

[root@ceph5 ~]# radosgw-admin zonegroup list

[root@ceph5 ~]# radosgw-admin zonegroup delete --rgw-zonegroup xiangyang

[root@ceph5 ~]# radosgw-admin zonegroup list

2.6 重建一個zonegroup

[root@ceph5 ~]# radosgw-admin zonegroup  create --rgw-realm hubei  --rgw-zonegroup xiangyang  --default --master

{
    "id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
    "name": "xiangyang",
    "api_name": "xiangyang",
    "is_master": "true",
    "endpoints": [],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "",
    "zones": [],
    "placement_targets": [],
    "default_placement": "",
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}

2.7 創建一個zone

[root@ceph5 ~]# radosgw-admin zone create --rgw-zonegroup xiangyang --rgw-zone  fancheng --default --endpoints http://ceph5.lab.example.com

{
    "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "name": "fancheng",
    "domain_root": "fancheng.rgw.meta:root",
    "control_pool": "fancheng.rgw.control",
    "gc_pool": "fancheng.rgw.log:gc",
    "lc_pool": "fancheng.rgw.log:lc",
    "log_pool": "fancheng.rgw.log",
    "intent_log_pool": "fancheng.rgw.log:intent",
    "usage_log_pool": "fancheng.rgw.log:usage",
    "reshard_pool": "fancheng.rgw.log:reshard",
    "user_keys_pool": "fancheng.rgw.meta:users.keys",
    "user_email_pool": "fancheng.rgw.meta:users.email",
    "user_swift_pool": "fancheng.rgw.meta:users.swift",
    "user_uid_pool": "fancheng.rgw.meta:users.uid",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "fancheng.rgw.buckets.index",
                "data_pool": "fancheng.rgw.buckets.data",
                "data_extra_pool": "fancheng.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}

2.8 為zone創建一個系統賬號

[root@ceph5 ~]# radosgw-admin  user  create --uid syncuser  --display-name  "sync user" --system

{
    "user_id": "syncuser",
    "display_name": "sync user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "syncuser",
            "access_key": "42Y4FBWY5VDCRLG9GDM8",
            "secret_key": "BiGRHcqL03RFoFWAbymFZj6xDQAXYFVb60cIzvav"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "system": "true",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

2.9 修改zone綁定這個用戶

[root@ceph5 ~]# radosgw-admin zone modify --rgw-zonegroup  xiangyang --rgw-zone  fancheng --access 42Y4FBWY5VDCRLG9GDM8 --secret BiGRHcqL03RFoFWAbymFZj6xDQAXYFVb60cIzvav

{
    "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "name": "fancheng",
    "domain_root": "fancheng.rgw.meta:root",
    "control_pool": "fancheng.rgw.control",
    "gc_pool": "fancheng.rgw.log:gc",
    "lc_pool": "fancheng.rgw.log:lc",
    "log_pool": "fancheng.rgw.log",
    "intent_log_pool": "fancheng.rgw.log:intent",
    "usage_log_pool": "fancheng.rgw.log:usage",
    "reshard_pool": "fancheng.rgw.log:reshard",
    "user_keys_pool": "fancheng.rgw.meta:users.keys",
    "user_email_pool": "fancheng.rgw.meta:users.email",
    "user_swift_pool": "fancheng.rgw.meta:users.swift",
    "user_uid_pool": "fancheng.rgw.meta:users.uid",
    "system_key": {
        "access_key": "",
        "secret_key": "BiGRHcqL03RFoFWAbymFZj6xDQAXYFVb60cIzvav"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "fancheng.rgw.buckets.index",
                "data_pool": "fancheng.rgw.buckets.data",
                "data_extra_pool": "fancheng.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}

 [root@ceph5 ~]# radosgw-admin zonegroup  get --rgw-zonegroup  xiangyang

{
    "id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
    "name": "xiangyang",
    "api_name": "xiangyang",
    "is_master": "true",
    "endpoints": [],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "zones": [
        {
            "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
            "name": "fancheng",
            "endpoints": [
                "http://ceph5.lab.example.com"
            ],
            "log_meta": "false",
            "log_data": "false",
            "bucket_index_max_shards": 0,
            "read_only": "false",
            "tier_type": "",
            "sync_from_all": "true",
            "sync_from": []
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}

2.10 查看用戶

[root@ceph5 ~]# radosgw-admin user list

[root@ceph5 ~]# radosgw-admin user list --rgw-zonegroup default --rgw-zone default

2.11 實施刷盤操作,更新period

[root@ceph5 ~]#  radosgw-admin period update --commit

{
    "id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
    "epoch": 1,
    "predecessor_uuid": "55fafff6-9c4c-4d54-8801-ad48861221a1",
    "sync_status": [],
    "period_map": {
        "id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
        "zonegroups": [
            {
                "id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
                "name": "xiangyang",
                "api_name": "xiangyang",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
                "zones": [
                    {
                        "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
                        "name": "fancheng",
                        "endpoints": [
                            "http://ceph5.lab.example.com"
                        ],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": []
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
            }
        ],
        "short_zone_ids": [
            {
                "key": "b337a562-5921-46fc-aad2-e70e99454e5f",
                "val": 557238113
            }
        ]
    },
    "master_zonegroup": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
    "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
    "realm_name": "hubei",
    "realm_epoch": 2
}

[root@ceph5 ~]# ceph osd pool ls

 2.12 更新配置文件並重啟

[root@ceph5 ~]# vim /etc/ceph/backup.conf 

[root@ceph5 ~]# systemctl restart [e-mail]5

為當前zone創建用戶

[root@ceph5 ~]# radosgw-admin user create --uid joy --display-name "Joy Ning" --subuser joy:swift

{
    "user_id": "joy",
    "display_name": "Joy Ning",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [
        {
            "id": "joy:swift",
            "permissions": "<none>"
        }
    ],
    "keys": [
        {
            "user": "joy",
            "access_key": "4YGLC3480T3Z5ZRY3UHG",
            "secret_key": "UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB"
        }
    ],
    "swift_keys": [
        {
            "user": "joy:swift",
            "secret_key": "ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL"
        }
    ],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}
[root@ceph5 ~]# radosgw-admin user create --uid joy --display-name "Joy Ning"
[root@ceph5 ~]# radosgw-admin subuser create --uid joy --display-name "Joy Ning" --subuser joy:swift

 三、測試

3.1 修改為full

 [root@ceph5 ~]# radosgw-admin subuser modify --uid joy --display-name "Joy Ning" --subuser joy:swift --access full

{
    "user_id": "joy",
    "display_name": "Joy Ning",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [
        {
            "id": "joy:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "joy",
            "access_key": "4YGLC3480T3Z5ZRY3UHG",
            "secret_key": "UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB"
        }
    ],
    "swift_keys": [
        {
            "user": "joy:swift",
            "secret_key": "ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL"
        }
    ],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

3.2 客戶端測試

[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL list
[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL post testswift
[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL upload  testswift /etc/ceph/ceph.client.rbd.keyring 
etc/ceph/ceph.client.rbd.keyring

3.3 服務端

出現元數據和數據的池

[root@ceph5 ~]# ceph osd pool ls

[root@ceph5 ~]# rados -p fancheng.rgw.buckets.data ls --cluster backup

四 、ceph2配置secondary域

4.1 拉取realm

[root@ceph2 ~]# radosgw-admin realm pull --url http://ceph5.lab.example.com  --access-key 4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB

2019-03-26 11:37:03.055756 7f1974483c40  1 error read_lastest_epoch .rgw.root:periods.1c6ccdef-ca02-44b5-b212-ba68acbd6aad.latest_epoch
2019-03-26 11:37:03.075671 7f1974483c40  1 Set the period's master zonegroup 5acbb712-0f7f-4108-8d93-ea75a19e33b3 as the default
{
    "id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
    "name": "hubei",
    "current_period": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
    "epoch": 2
}

 [root@ceph2 ~]# radosgw-admin realm list

[root@ceph2 ~]# radosgw-admin zonegroup get --rgw-realm hubei

{
    "id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
    "name": "xiangyang",
    "api_name": "xiangyang",
    "is_master": "true",
    "endpoints": [],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "zones": [
        {
            "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
            "name": "fancheng",
            "endpoints": [
                "http://ceph5.lab.example.com"
            ],
            "log_meta": "false",
            "log_data": "false",
            "bucket_index_max_shards": 0,
            "read_only": "false",
            "tier_type": "",
            "sync_from_all": "true",
            "sync_from": []
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}

4.2 拉取period

[root@ceph2 ~]# radosgw-admin period pull --url  http://ceph5.lab.example.com --access-key 4YGLC3480T3Z5ZRY3UHG  --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB

2019-03-26 11:47:05.406374 7f3dc85c7c40  1 found existing latest_epoch 1 >= given epoch 1, returning r=-17
{
    "id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
    "epoch": 1,
    "predecessor_uuid": "55fafff6-9c4c-4d54-8801-ad48861221a1",
    "sync_status": [],
    "period_map": {
        "id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
        "zonegroups": [
            {
                "id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
                "name": "xiangyang",
                "api_name": "xiangyang",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
                "zones": [
                    {
                        "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
                        "name": "fancheng",
                        "endpoints": [
                            "http://ceph5.lab.example.com"
                        ],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": []
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
            }
        ],
        "short_zone_ids": [
            {
                "key": "b337a562-5921-46fc-aad2-e70e99454e5f",
                "val": 557238113
            }
        ]
    },
    "master_zonegroup": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
    "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
    "realm_name": "hubei",
    "realm_epoch": 2
}

4.3 創建Secondary Zone

[root@ceph2 ~]# radosgw-admin zone create --rgw-zonegroup  xiangyang --rgw-zone  xiantao  --endpoints http://ceph2.lab.example.com  --default --access-key 4YGLC3480T3Z5ZRY3UHG  --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB

2019-03-26 11:49:47.924379 7f7d2758cc40  0 failed reading obj info from .rgw.root:zone_info.b337a562-5921-46fc-aad2-e70e99454e5f: (2) No such file or directory
2019-03-26 11:49:47.924419 7f7d2758cc40  0 WARNING: could not read zone params for zone id=b337a562-5921-46fc-aad2-e70e99454e5f name=fancheng
{
    "id": "b012d15d-a83c-4553-9ec3-09bf45d4a67b",
    "name": "xiantao",
    "domain_root": "xiantao.rgw.meta:root",
    "control_pool": "xiantao.rgw.control",
    "gc_pool": "xiantao.rgw.log:gc",
    "lc_pool": "xiantao.rgw.log:lc",
    "log_pool": "xiantao.rgw.log",
    "intent_log_pool": "xiantao.rgw.log:intent",
    "usage_log_pool": "xiantao.rgw.log:usage",
    "reshard_pool": "xiantao.rgw.log:reshard",
    "user_keys_pool": "xiantao.rgw.meta:users.keys",
    "user_email_pool": "xiantao.rgw.meta:users.email",
    "user_swift_pool": "xiantao.rgw.meta:users.swift",
    "user_uid_pool": "xiantao.rgw.meta:users.uid",
    "system_key": {
        "access_key": "4YGLC3480T3Z5ZRY3UHG",
        "secret_key": "UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "xiantao.rgw.buckets.index",
                "data_pool": "xiantao.rgw.buckets.data",
                "data_extra_pool": "xiantao.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}

[root@ceph2 ~]# radosgw-admin zonegroup get --rgw-realm  hubei --rgw-zonegroup

parse error setting 'rgw_zonegroup' to '' (Option --rgw-zonegroup requires an argument.
)

{
    "id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
    "name": "xiangyang",
    "api_name": "xiangyang",
    "is_master": "true",
    "endpoints": [],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "zones": [
        {
            "id": "b012d15d-a83c-4553-9ec3-09bf45d4a67b",
            "name": "xiantao",
            "endpoints": [
                "http://ceph2.lab.example.com"
            ],
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false",
            "tier_type": "",
            "sync_from_all": "true",
            "sync_from": []
        },
        {
            "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
            "name": "fancheng",
            "endpoints": [
                "http://ceph5.lab.example.com"
            ],
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false",
            "tier_type": "",
            "sync_from_all": "true",
            "sync_from": []
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}

[root@ceph5 ~]# radosgw-admin zone modify  --rgw-zonegroup  xiangyang --rgw-zone  fancheng --endpoints http://ceph5.lab.example.com

{
    "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "name": "fancheng",
    "domain_root": "fancheng.rgw.meta:root",
    "control_pool": "fancheng.rgw.control",
    "gc_pool": "fancheng.rgw.log:gc",
    "lc_pool": "fancheng.rgw.log:lc",
    "log_pool": "fancheng.rgw.log",
    "intent_log_pool": "fancheng.rgw.log:intent",
    "usage_log_pool": "fancheng.rgw.log:usage",
    "reshard_pool": "fancheng.rgw.log:reshard",
    "user_keys_pool": "fancheng.rgw.meta:users.keys",
    "user_email_pool": "fancheng.rgw.meta:users.email",
    "user_swift_pool": "fancheng.rgw.meta:users.swift",
    "user_uid_pool": "fancheng.rgw.meta:users.uid",
    "system_key": {
        "access_key": "",
        "secret_key": "BiGRHcqL03RFoFWAbymFZj6xDQAXYFVb60cIzvav"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "fancheng.rgw.buckets.index",
                "data_pool": "fancheng.rgw.buckets.data",
                "data_extra_pool": "fancheng.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}

4.4 更新period

[root@ceph5 ~]#  radosgw-admin period update --commit

{
    "id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
    "epoch": 2,
    "predecessor_uuid": "55fafff6-9c4c-4d54-8801-ad48861221a1",
    "sync_status": [],
    "period_map": {
        "id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
        "zonegroups": [
            {
                "id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
                "name": "xiangyang",
                "api_name": "xiangyang",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
                "zones": [
                    {
                        "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
                        "name": "fancheng",
                        "endpoints": [
                            "http://ceph5.lab.example.com"
                        ],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": []
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
            }
        ],
        "short_zone_ids": [
            {
                "key": "b337a562-5921-46fc-aad2-e70e99454e5f",
                "val": 557238113
            }
        ]
    },
    "master_zonegroup": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
    "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
    "realm_name": "hubei",
    "realm_epoch": 2
}

[root@ceph2 ~]# ceph auth get-or-create client.rgw.ceph2  mon 'allow rwx' osd 'allow rwx' -o /etc/ceph/ceph.client.radosgw.keyring --cluster ceph

[root@ceph2 ~]# ceph osd pool ls

發現沒有成功，重新拉取一遍

4.5 刪除relam

[root@ceph2 ~]# radosgw-admin zonegroup delete --rgw-zonegroup xiangyang
[root@ceph2 ~]# radosgw-admin realm delete --rgw-realm hubei

[root@ceph2 ~]# radosgw-admin zone delete --rgw-zone xiantao

[root@ceph2 ~]# radosgw-admin realm list

 4.6 重新拉取

[root@ceph2 ~]# radosgw-admin realm pull --url http://ceph5.lab.example.com --access-key  4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB
2019-03-26 13:28:24.324288 7f00bbc65c40  1 Set the period's master zonegroup 5acbb712-0f7f-4108-8d93-ea75a19e33b3 as the default
{
    "id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
    "name": "hubei",
    "current_period": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
    "epoch": 2
}

[root@ceph2 ~]# radosgw-admin period pull --url http://ceph5.lab.example.com --access-key  4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB
{
    "id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
    "epoch": 5,
    "predecessor_uuid": "55fafff6-9c4c-4d54-8801-ad48861221a1",
    "sync_status": [],
    "period_map": {
        "id": "1c6ccdef-ca02-44b5-b212-ba68acbd6aad",
        "zonegroups": [
            {
                "id": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
                "name": "xiangyang",
                "api_name": "xiangyang",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
                "zones": [
                    {
                        "id": "b337a562-5921-46fc-aad2-e70e99454e5f",
                        "name": "fancheng",
                        "endpoints": [
                            "http://ceph5.lab.example.com"
                        ],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": []
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
            }
        ],
        "short_zone_ids": [
            {
                "key": "b337a562-5921-46fc-aad2-e70e99454e5f",
                "val": 557238113
            }
        ]
    },
    "master_zonegroup": "5acbb712-0f7f-4108-8d93-ea75a19e33b3",
    "master_zone": "b337a562-5921-46fc-aad2-e70e99454e5f",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba",
    "realm_name": "hubei",
    "realm_epoch": 2
}

2019-03-26 13:28:54.816633 7efd7ebf8c40  1 found existing latest_epoch 5 >= given epoch 5, returning r=-17
[root@ceph2 ~]#  radosgw-admin zone create --rgw-zonegroup xiangyang --rgw-zone xiantao --endpoints http://ceph2.lab.example.com --default --access-key 4YGLC3480T3Z5ZRY3UHG --secret UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB
2019-03-26 13:30:01.076567 7fb699f77c40  0 failed reading obj info from .rgw.root:zone_info.b337a562-5921-46fc-aad2-e70e99454e5f: (2) No such file or directory
2019-03-26 13:30:01.076625 7fb699f77c40  0 WARNING: could not read zone params for zone id=b337a562-5921-46fc-aad2-e70e99454e5f name=fancheng
{
    "id": "09ece789-b2fc-4379-8c60-d37811074216",
    "name": "xiantao",
    "domain_root": "xiantao.rgw.meta:root",
    "control_pool": "xiantao.rgw.control",
    "gc_pool": "xiantao.rgw.log:gc",
    "lc_pool": "xiantao.rgw.log:lc",
    "log_pool": "xiantao.rgw.log",
    "intent_log_pool": "xiantao.rgw.log:intent",
    "usage_log_pool": "xiantao.rgw.log:usage",
    "reshard_pool": "xiantao.rgw.log:reshard",
    "user_keys_pool": "xiantao.rgw.meta:users.keys",
    "user_email_pool": "xiantao.rgw.meta:users.email",
    "user_swift_pool": "xiantao.rgw.meta:users.swift",
    "user_uid_pool": "xiantao.rgw.meta:users.uid",
    "system_key": {
        "access_key": "4YGLC3480T3Z5ZRY3UHG",
        "secret_key": "UuHLN9nlTofwez8Nz0RVJ60Vl6v6zGVJtjLP04pB"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "xiantao.rgw.buckets.index",
                "data_pool": "xiantao.rgw.buckets.data",
                "data_extra_pool": "xiantao.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": "d4668fc2-ceed-4eb2-a5e7-a70c2aa7deba"
}
[root@ceph2 ~]# cd /etc/ceph/
[root@ceph2 ceph]# rm -rf ceph.client.radosgw.keyring 
[root@ceph2 ceph]#  ceph auth get-or-create client.rgw.ceph2 mon 'allow rwx' osd 'allow rwx' -o /etc/ceph/ceph.client.radosgw.keyring --cluster ceph
[root@ceph2 ceph]# ceph osd  pool ls
testpool
rbd
rbdmirror
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
xiantao.rgw.control
xiantao.rgw.meta
xiantao.rgw.log

五、遷移Single-Zone至Multi-Zone

5.1 查看

[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL list 

 

[root@ceph1 ~]# swift -A http://ceph5.lab.example.com/auth/v1.0 -U joy:swift -K ofbO1nZ1vqUHQwdNuyUd6zLLYhlbiDxMCILfbJoL list testswift

5.2 ceph5把default的zone添加到一個realm

[root@ceph5 ~]#  radosgw-admin realm list

 

[root@ceph5 ~]# radosgw-admin zonegroup list

5.3 為default創建一個新的zonegroup

[root@ceph5 ~]# radosgw-admin realm create --rgw-realm realmnew

5.4 修改default的realm，設置為master

[root@ceph5 ~]# radosgw-admin zonegroup modify --rgw-realm realmnew --rgw-zonegroup default --master

{
    "id": "e80133e1-a513-44f5-ba90-e25b6c987b26",
    "name": "default",
    "api_name": "",
    "is_master": "true",
    "endpoints": [],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
    "zones": [
        {
            "id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
            "name": "default",
            "endpoints": [],
            "log_meta": "false",
            "log_data": "false",
            "bucket_index_max_shards": 0,
            "read_only": "false",
            "tier_type": "",
            "sync_from_all": "true",
            "sync_from": []
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "de918d45-d763-416d-af0a-0350b1339ca1"
}

 [root@ceph5 ~]# radosgw-admin zone modify --rgw-zonegroup default --rgw-zone default --master --endpoints http://ceph5.lab.example.com

{
    "id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.log:gc",
    "lc_pool": "default.rgw.log:lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",
    "user_email_pool": "default.rgw.meta:users.email",
    "user_swift_pool": "default.rgw.meta:users.swift",
    "user_uid_pool": "default.rgw.meta:users.uid",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "data_pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": ""
}

5.5 創建一個系統用戶

[root@ceph5 ~]# radosgw-admin user create --uid syncuser1 --display-name "sync user" --rgw-zonegroup default --rgw-zone default

{
    "user_id": "syncuser1",
    "display_name": "sync user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "syncuser1",
            "access_key": "MIVNCAI762F49VPRAFDF",
            "secret_key": "kHJSrkBxpVQJsCOy2sV4P9ElmzTkjcPX81R6hycR"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

5.6 將master-zone與系統用戶關聯

 [root@ceph5 ~]#  radosgw-admin zone modify --rgw-zonegroup default --rgw-zone default --master --endpoints http://ceph5.lab.example.com --access-key MIVNCAI762F49VPRAFDF --secret kHJSrkBxpVQJsCOy2sV4P9ElmzTkjcPX81R6hycR

{
    "id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.log:gc",
    "lc_pool": "default.rgw.log:lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",
    "user_email_pool": "default.rgw.meta:users.email",
    "user_swift_pool": "default.rgw.meta:users.swift",
    "user_uid_pool": "default.rgw.meta:users.uid",
    "system_key": {
        "access_key": "MIVNCAI762F49VPRAFDF",
        "secret_key": "kHJSrkBxpVQJsCOy2sV4P9ElmzTkjcPX81R6hycR"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "data_pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": ""
}

5.7 提交新的配置 

[root@ceph5 ~]#  radosgw-admin period update --commit --rgw-zonegroup default --rgw-zone default --rgw-realm realmnew

2019-03-26 13:50:39.137059 7fd1dd9cec40  1 Set the period's master zonegroup e80133e1-a513-44f5-ba90-e25b6c987b26 as the default
{
    "id": "152a15c4-d0f7-4bd1-93db-fc1f8655b741",
    "epoch": 1,
    "predecessor_uuid": "0daf85ed-732b-4f0f-8f58-a3f8ef1e996e",
    "sync_status": [],
    "period_map": {
        "id": "152a15c4-d0f7-4bd1-93db-fc1f8655b741",
        "zonegroups": [
            {
                "id": "e80133e1-a513-44f5-ba90-e25b6c987b26",
                "name": "default",
                "api_name": "",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
                "zones": [
                    {
                        "id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
                        "name": "default",
                        "endpoints": [
                            "http://ceph5.lab.example.com"
                        ],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": []
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "de918d45-d763-416d-af0a-0350b1339ca1"
            }
        ],
        "short_zone_ids": [
            {
                "key": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
                "val": 422091396
            }
        ]
    },
    "master_zonegroup": "e80133e1-a513-44f5-ba90-e25b6c987b26",
    "master_zone": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "de918d45-d763-416d-af0a-0350b1339ca1",
    "realm_name": "realmnew",
    "realm_epoch": 2
}

5.8 修改配置文件使生效

[root@ceph5 ~]# vim /etc/ceph/backup.conf

fsid = 51dda18c-7545-4edb-8ba9-27330ead81a7
mon_initial_members = ceph5
mon_host = 172.25.250.14
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 172.25.250.0/24
cluster_network = 172.25.250.0/24
[mgr]
mgr modules = dashboard
[client.rgw.ceph5]
host = ceph5
keyring = /etc/ceph/backup.client.rgw.ceph5.keyring
rgw_frontends = civetweb port=80 num_threads=100
log = /var/log/ceph/$cluster.$name.log
rgw_dns_name = ceph5.lab.example.com
rgw_zone = default
rgw_region = realmnew
rgw_zonegroup = default

[root@ceph5 ~]# systemctl restart [e-mail]5

[root@ceph5 ~]# ps -ef|grep rados

---

博主聲明：本文的內容來源主要來自譽天教育晏威老師，由本人實驗完成操作驗證，需要的博友請聯系譽天教育（http://www.yutianedu.com/），獲得官方同意或者晏老師（https://www.cnblogs.com/breezey/）本人同意即可轉載，謝謝！