20210821第二天：Ceph賬號管理（普通用戶掛載）、mds高可用

下面主要內容：

1. 用戶權限管理和授權流程
2. 用普通用戶掛載rbd和cephfs
3. mds高可用
   • 多mds active
   • 多mds active加standby

 

一、Ceph的用戶權限管理和授權流程

一般系統的身份認真無非三點：賬號、角色和認真鑒權，Ceph 的用戶可以是一個具體的人或系統角色（e.g. 應用程序），Ceph 管理員通過創建用戶並設置權限來控制誰可以訪問、操作 Ceph Cluster、Pool 或 Objects 等資源。

 

 Ceph 的用戶類型可以分為以下幾類：

• 客戶端用戶
  • 操作用戶（e.g. client.admin）
  • 應用程序用戶（e.g. client.cinder）
• 其他用戶
  • Ceph 守護進程用戶（e.g. mds.ceph-node1、osd.0）

用戶命名遵循 <TYPE.ID> 的命名規則，其中 Type 有 mon,osd,client 三者，L 版本以后添加了 mgr 類型。

NOTE：為 Ceph 守護進程創建用戶，是因為 MON、OSD、MDS 等守護進程同樣遵守 CephX 協議，但它們不能屬於真正意義上的客戶端。

Ceph 用戶的密鑰：相當於用戶的密碼，本質是一個唯一字符串。e.g. key: AQB3175c7fuQEBAAMIyROU5o2qrwEghuPwo68g==

Ceph 用戶的授權（Capabilities, caps）：即用戶的權限，通常在創建用戶的同時進行授權。只有在授權之后，用戶才可以使用權限范圍內的 MON、OSD、MDS 的功能，也可以通過授權限制用戶對 Ceph 集群數據或命名空間的訪問范圍。以 client.admin 和 client.jack為例：

client.admin
    key: AQBErhthY4YdIhAANKTOMAjkzpKkHSkXSoNpaQ==
    # 允許訪問 MDS
    caps: [mds] allow *
    # 允許訪問 Mgr
    caps: [mgr] allow *
    # 允許訪問 MON
    caps: [mon] allow *
    # 允許訪問 OSD
    caps: [osd] allow *

client.tom
    key: AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    # 允許對 MON 讀的權限
    caps: [mon] allow r
    # 允許對rbd-data資源池進行讀寫
    caps: [osd] allow rwx pool=rbd-date

授權類型：

• allow：在守護進程進行訪問設置之前就已經具有特定權限，常見於管理員和守護進程用戶。
• r：授予用戶讀的權限，讀取集群各個組件（MON/OSD/MDS/CRUSH/PG）的狀態，但是不能修改。
• w：授予用戶寫對象的權限，與 r 配合使用，修改集群的各個組件的狀態，可以執行組件的各個動作指令。
• x：授予用戶調用類方法的能力，僅僅和 ceph auth 操作相關。
• class-read：授予用戶調用類讀取方法的能力，是 x 的子集。
• class-write：授予用戶調用類寫入方法的能力，是 x 的子集。
• *：授予用戶 rwx 權限。
• profile osd：授權用戶以 OSD 身份連接到其它 OSD 或 MON，使得 OSD 能夠處理副本心跳和狀態匯報。
• profile mds：授權用戶以 MDS 身份連接其它 MDS 或 MON。
• profile bootstrap-osd：授權用戶引導 OSD 守護進程的能力，通常授予部署工具（e.g. ceph-deploy），讓它們在引導 OSD 時就有增加密鑰的權限了。
• profile bootstrap-mds：授權用戶引導 MDS 守護進程的能力。同上。

NOTE：可見 Ceph 客戶端不直接訪問 Objects，而是必須要經過 OSD 守護進程的交互。

用戶管理常規操作

Ceph 用戶管理指令 auth：

root@node02:~# ceph auth --help
...
auth add <entity> [<caps>...]                                                                            add auth info for <entity> from input file, or random key if no input is given, and/or any caps 
                                                                                                          specified in the command
auth caps <entity> <caps>...                                                                             update caps for <name> from caps specified in the command
auth export [<entity>]                                                                                   write keyring for requested entity, or master keyring if none given
auth get <entity>                                                                                        write keyring file with requested key
auth get-key <entity>                                                                                    display requested key
auth get-or-create <entity> [<caps>...]                                                                  add auth info for <entity> from input file, or random key if no input given, and/or any caps specified 
                                                                                                          in the command
auth get-or-create-key <entity> [<caps>...]                                                              get, or add, key for <name> from system/caps pairs specified in the command.  If key already exists, any 
                                                                                                          given caps must match the existing caps for that key.
auth import                                                                                              auth import: read keyring file from -i <file>
auth ls                                                                                                  list authentication state
auth print-key <entity>                                                                                  display requested key
auth print_key <entity>                                                                                  display requested key
auth rm <entity>                                                                                         remove all caps for <name>

獲取指定用戶的權限信息：

root@node02:~# ceph auth get client.admin
[client.admin]
    key = AQBErhthY4YdIhAANKTOMAjkzpKkHSkXSoNpaQ==
    caps mds = "allow *"
    caps mgr = "allow *"
    caps mon = "allow *"
    caps osd = "allow *"
exported keyring for client.admin
root@node02:~# 

新建用戶：

# 授權的格式為：`{守護進程類型} 'allow {權限}'`

# 創建 client.jack 用戶，並授權可讀 MON、可讀寫 OSD Pool livepool
ceph auth add client.jack mon 'allow r' osd 'allow rw pool=rbd-data'

# 獲取或創建 client.tom，若創建，則授權可讀 MON，可讀寫 OSD Pool livepool
ceph auth get-or-create client.tom mon 'allow r' osd 'allow rw pool=rbd-data'

# 獲取或創建 client.jerry，若創建，則授權可讀 MON，可讀寫  OSD Pool livepool，輸出用戶秘鑰環文件
ceph auth get-or-create client.jerry mon 'allow r' osd 'allow rw pool=rbd-data' -o jerry.keyring
ceph auth get-or-create-key client.ringo mon 'allow r' osd 'allow rw pool=rbd-data' -o ringo.key

刪除用戶：

ceph auth del {TYPE}.{ID}

CephX 認證系統

Ceph 提供了兩種身份認證方式：None 和 CephX。前者表示客戶端不需要通過密鑰訪問即可訪問 Ceph 存儲集群，顯然這種方式是不被推薦的。所以我們一般會啟用 CephX 認證系統，通過編輯 ceph.conf 開啟：

[global]
...
# 表示存儲集群（mon,osd,mds）相互之間需要通過 keyring 認證
auth_cluster_required = cephx
# 表示客戶端（比如gateway）到存儲集群（mon,osd,mds）需要通過 keyring 認證
auth_service_required = cephx
# 表示存儲集群（mon,osd,mds）到客戶端（e.g. gateway）需要通過 keyring 認證
auth_client_required = cephx

　　CephX 的本質是一種對稱加密協議，加密算法為 AES，用於識別用戶的身份、對用戶在客戶端上的操作進行認證，以此防止中間人攻擊、數據篡改等網絡安全問題。應用 CephX 的前提是創建一個用戶，當我們使用上述指令創建一個用戶時，MON 會將用戶的密鑰返回給客戶端，同時自己也會保存一份副本。這樣客戶端和 MON 就共享了一份密鑰。CephX 使用共享密鑰的方式進行認證，客戶端和 MON 集群都會持有一份用戶密鑰，它提供了一個相互認證的機制，MON 集群確定客戶端持有用戶密鑰，而客戶端確定 MON 集群持有用戶密鑰的的副本。

 

身份認證原理

 

1. 用戶通過客戶端向 MON 發起請求。
2. 客戶端將用戶名傳遞到 MON。
3. MON 對用戶名進行檢查，若用戶存在，則通過加密用戶密鑰生成一個 session key 並返回客戶端。
4. 客戶端通過共享密鑰解密 session key，只有擁有相同用戶密鑰環文件的客戶端可以完成解密。
5. 客戶端得到 session key 后，客戶端持有 session key 再次向 MON 發起請求
6. MON 生成一個 ticket，同樣使用用戶密鑰進行加密，然后發送給客戶端。
7. 客戶端同樣通過共享密鑰解密得到 ticket。
8. 往后，客戶端持有 ticket 向 MON、OSD 發起請求。

 這就是共享密鑰認證的好處，客戶端、MON、OSD、MDS 共同持有用戶的密鑰，只要客戶端與 MON 完成驗證之后，客戶端就可以與任意服務進行交互。並且只要客戶端擁有任意用戶的密鑰環文件，客戶端就可以執行特定用戶所具有權限的所有操作。當我們執行 ceph -s 時候，實際上執行的是 ceph -s --conf /etc/ceph/ceph.conf --name client.admin --keyring /etc/ceph/ceph.client.admin.keyring。客戶端會從下列默認路徑查看 admin 用戶的 keyring：

 

/etc/ceph/ceph.client.admin.keyring
/etc/ceph/ceph.keyring
/etc/ceph/keyring
/etc/ceph/keyring.bin

使用 ceph-authtool 進行密鑰環管理

NOTE：密鑰環文件 .keyring 是 Ceph 用戶的載體，當客戶端擁有了密鑰環文件就相當於可以通過對應的用戶來執行客戶端操作。

MON、OSD、admin client 的初始密鑰環文件：

• /var/lib/ceph/mon/ceph-$hostname/keyring
• /var/lib/ceph/osd/ceph-$hostname/keyring
• /etc/ceph/ceph.client.admin.keyring

指定密鑰環（admin 用戶）來執行集群操作：

ceph -n client.admin --keyring=/etc/ceph/ceph.client.admin.keyring health
rbd create -p rbd volume01 --size 1G --image-feature layering --conf /etc/ceph/ceph.conf --name client.admin --keyring /etc/ceph/ceph.client.admin.keyring

查看用戶密鑰：

ceph auth print-key {TYPE}.{ID}

將用戶密鑰導入到密鑰環：

ceph auth get client.admin -o /etc/ceph/ceph.client.admin.keyring

創建用戶並生產密鑰環文件：

ceph-authtool -n client.user1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.keyring
ceph-authtool -C /etc/ceph/ceph.keyring -n client.user1 --cap osd 'allow rwx' --cap mon 'allow rwx' --gen-key

修改用戶授權：

ceph-authtool /etc/ceph/ceph.keyring -n client.user1 --cap osd 'allow rwx' --cap mon 'allow rwx'

從密鑰環文件導入一個用戶:

ceph auth import -i /path/to/keyring

將用戶導出到一個密鑰環文件：

ceph auth get client.admin -o /etc/ceph/ceph.client.admin.keyring

不廢話了，下面是我本次作業的情況：

1、創建塊、cephfs和創建普通用戶掛載

1.1、創建rbd-data存儲池並創建rbdtest1.img塊：

root@node01:~# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL    USED  RAW USED  %RAW USED
hdd    240 GiB  240 GiB  41 MiB    41 MiB       0.02
TOTAL  240 GiB  240 GiB  41 MiB    41 MiB       0.02
 
--- POOLS ---
POOL                   ID  PGS   STORED  OBJECTS    USED  %USED  MAX AVAIL
device_health_metrics   1    1      0 B        0     0 B      0     76 GiB
mypool1                 2   32  3.1 KiB        3  21 KiB      0     76 GiB
root@node01:~# 
root@node01:~# ceph osd pool create rbd-data 32 32
pool 'rbd-data' created
root@node02:~# ceph osd lspools
1 device_health_metrics
2 mypool1
3 rbd-data
root@node02:~# 
root@node01:~# ceph osd pool ls
device_health_metrics
mypool1
rbd-data
root@node01:~# ceph osd pool application enable rbd-data rbd
enabled application 'rbd' on pool 'rbd-data'
root@node01:~# rb
rbash            rbd              rbdmap           rbd-replay       rbd-replay-many  rbd-replay-prep  
root@node01:~# rbd pool init -p rbd-data
root@node01:~# rbd create rbdtest1.img --pool rbd-data --size 1G
root@node01:~# rbd ls --pool rbd-data
rbdtest1.img
root@node01:~# rbd info rbd-data/rbdtest1.img
rbd image 'rbdtest1.img':
    size 1 GiB in 256 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: fb9a70e1dad6
    block_name_prefix: rbd_data.fb9a70e1dad6
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Tue Aug 24 20:17:40 2021
    access_timestamp: Tue Aug 24 20:17:40 2021
    modify_timestamp: Tue Aug 24 20:17:40 2021
root@node01:~# rbd feature disable rbd-data/rbdtest1.img exclusive-lock object-map fast-diff  deep-flatten
root@node01:~# rbd info rbd-data/rbdtest1.img
rbd image 'rbdtest1.img':
    size 1 GiB in 256 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: fb9a70e1dad6
    block_name_prefix: rbd_data.fb9a70e1dad6
    format: 2
    features: layering
    op_features: 
    flags: 
    create_timestamp: Tue Aug 24 20:17:40 2021
    access_timestamp: Tue Aug 24 20:17:40 2021
    modify_timestamp: Tue Aug 24 20:17:40 2021
root@node01:~# 

 下面是具體主要步驟命令：

root@node01:~# ceph osd pool create rbd-data 32 32
root@node01:~# ceph osd pool application enable rbd-data rbd
root@node01:~# rbd pool init -p rbd-data
root@node01:~# rbd create rbdtest1.img --pool rbd-data --size 1G
root@node01:~# rbd ls --pool rbd-data
root@node01:~# rbd info rbd-data/rbdtest1.img
root@node01:~# rbd feature disable rbd-data/rbdtest1.img exclusive-lock object-map fast-diff  deep-flatten
root@node01:~# rbd info rbd-data/rbdtest1.img

1.2、創建刪除tom、Jerry等賬號的情況：

root@node01:~# ceph auth add client.tom mon 'allow r' osd 'allow rwx pool=rbd-date'
added key for client.tom
root@node01:~# ceph auth get client.tom
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
exported keyring for client.tom
root@node01:~# 
root@node01:~# 
root@node01:~# ceph auth get
get                get-key            get-or-create      get-or-create-key  
root@node01:~# ceph auth get-or-create client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# 
root@node01:~# ceph auth get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.jerry
root@node01:~# 
root@node01:~# ceph auth get-or-create-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# ceph auth get-or-create-key client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# ceph auth get-or-create-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# 
root@node01:~# ceph auth print-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==root@node01:~# 
root@node01:~# 
root@node01:~# 
root@node01:~# ceph auth  get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.jerry
root@node01:~# ceph auth caps client.jerry mon 'allow rw' osd 'allow r pool=rbd-data'
updated caps for client.jerry
root@node01:~# ceph auth get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow rw"
    caps osd = "allow r pool=rbd-data"
exported keyring for client.jerry
root@node01:~# ceph auth  del client.jerry
updated
root@node01:~# ceph auth get client.jerry
Error ENOENT: failed to find client.jerry in keyring
root@node01:~# 

 下面是具體主要步驟命令：

root@node01:~# ceph auth add client.tom mon 'allow r' osd 'allow rwx pool=rbd-date'
root@node01:~# ceph auth get client.tom
root@node01:~# ceph auth get-or-create client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
root@node01:~# ceph auth get-or-create-key client.jerry
root@node01:~# ceph auth get-or-create-key client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
root@node01:~# ceph auth print-key client.jerry
root@node01:~# ceph auth caps client.jerry mon 'allow rw' osd 'allow r pool=rbd-data'

1.3、通過秘鑰環文件備份與恢復用

root@node01:~# ceph auth get-or-create client.user1 mon 'allow r' osd 'allow rwx pool=rbd-data'
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
root@node01:~# ceph auth get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# ls
ceph-deploy  cluster  snap
root@node01:~# ceph-authtool --help
usage: ceph-authtool keyringfile [OPTIONS]...
where the options are:
  -l, --list                    will list all keys and capabilities present in
                                the keyring
  -p, --print-key               will print an encoded key for the specified
                                entityname. This is suitable for the
                                'mount -o secret=..' argument
  -C, --create-keyring          will create a new keyring, overwriting any
                                existing keyringfile
  -g, --gen-key                 will generate a new secret key for the
                                specified entityname
  --gen-print-key               will generate a new secret key without set it
                                to the keyringfile, prints the secret to stdout
  --import-keyring FILE         will import the content of a given keyring
                                into the keyringfile
  -n NAME, --name NAME          specify entityname to operate on
  -a BASE64, --add-key BASE64   will add an encoded key to the keyring
  --cap SUBSYSTEM CAPABILITY    will set the capability for given subsystem
  --caps CAPSFILE               will set all of capabilities associated with a
                                given key, for all subsystems
  --mode MODE                   will set the desired file mode to the keyring
                                e.g: '0644', defaults to '0600'
root@node01:~# ceph-authtool -C ceph.client.user1.keyring
creating ceph.client.user1.keyring
root@node01:~# cat ceph.client.user1.keyring 
root@node01:~# file ceph.client.user1.keyring 
ceph.client.user1.keyring: empty
root@node01:~# 
root@node01:~# ceph auth  get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# ceph auth  get client.user1 -o ceph.client.user1.keyring 
exported keyring for client.user1
root@node01:~# cat ceph.client.user1.keyring 
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
root@node01:~# 

從 keyring 文件恢復用戶認證信息:


root@node01:~# 
root@node01:~# ceph auth del client.user1
updated
root@node01:~# ceph auth get client.user1
Error ENOENT: failed to find client.user1 in keyring
root@node01:~# 
root@node01:~# ceph auth import -i ceph.client.user1.keyring 
imported keyring
root@node01:~# 
root@node01:~# ceph auth get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# 

 下面是具體主要步驟命令：

root@node01:~# ceph auth get-or-create client.user1 mon 'allow r' osd 'allow rwx pool=rbd-data'
root@node01:~# ceph auth get client.user1
root@node01:~# ceph-authtool -C ceph.client.user1.keyring
root@node01:~# ceph auth  get client.user1 -o ceph.client.user1.keyring 

從keyring 文件恢復用戶認證信息:
root@node01:~# ceph auth del client.user1
root@node01:~# ceph auth get client.user1
root@node01:~# ceph auth import -i ceph.client.user1.keyring 
root@node01:~# ceph auth get client.user1

再加一點：一個配置文件中多個賬號情況：

root@node01:~# ceph-authtool -C ceph.client.tom.keyring
creating ceph.client.tom.keyring
root@node01:~# ceph auth get client.tom -o ceph.client.tom.keyring 
exported keyring for client.tom
root@node01:~# cat ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# ceph-authtool -l ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# 
root@node01:~# ceph-authtool ceph.client.user1.keyring --import-keyring ceph.client.tom.keyring 
importing contents of ceph.client.tom.keyring into ceph.client.user1.keyring
root@node01:~# echo $?
0
root@node01:~# cat ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# 
root@node01:~# cat ceph.client.user1.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
root@node01:~# 

1.4、普通用戶掛載RBD情況：

服務上同步配置文件到客戶端：
root@node01:~# ls
ceph.client.tom.keyring  ceph.client.user1.keyring  ceph-deploy  client.client.user1.keyring  client.user1.keyring  cluster  snap
root@node01:~# scp ceph.client.user1.keyring ceph-deploy/ceph.conf root@node04:/etc/ceph/
root@node04's password: 
ceph.client.user1.keyring                                                                                                                                                       100%  244   263.4KB/s   00:00    
ceph.conf                                                                                                                                                                       100%  265    50.5KB/s   00:00    
root@node01:~#

客戶端操作

root@node04:/etc/netplan# apt install ceph-common
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  ibverbs-providers libbabeltrace1 libcephfs2 libdw1 libgoogle-perftools4 libibverbs1 libjaeger libleveldb1d liblttng-ust-ctl4 liblttng-ust0 liblua5.3-0 libnl-route-3-200 liboath0 librabbitmq4 librados2
  libradosstriper1 librbd1 librdkafka1 librdmacm1 librgw2 libsnappy1v5 libtcmalloc-minimal4 python3-ceph-argparse python3-ceph-common python3-cephfs python3-prettytable python3-rados python3-rbd python3-rgw
Suggested packages:
  ceph-base ceph-mds
The following NEW packages will be installed:
  ceph-common ibverbs-providers libbabeltrace1 libcephfs2 libdw1 libgoogle-perftools4 libibverbs1 libjaeger libleveldb1d liblttng-ust-ctl4 liblttng-ust0 liblua5.3-0 libnl-route-3-200 liboath0 librabbitmq4
  librados2 libradosstriper1 librbd1 librdkafka1 librdmacm1 librgw2 libsnappy1v5 libtcmalloc-minimal4 python3-ceph-argparse python3-ceph-common python3-cephfs python3-prettytable python3-rados python3-rbd
  python3-rgw
0 upgraded, 30 newly installed, 0 to remove and 79 not upgraded.
Need to get 37.3 MB of archives.
After this operation, 152 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific focal/main amd64 libjaeger amd64 16.2.5-1focal [3,780 B]

中間太長了，我就刪除了點

Setting up ceph-common (16.2.5-1focal) ...
Adding group ceph....done
Adding system user ceph....done
Setting system user ceph properties....done
chown: cannot access '/var/log/ceph/*.log*': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target.
Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service.
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
root@node04:/etc/netplan# ls /etc/ceph/
rbdmap
root@node04:/etc/netplan# ls /etc/ceph/
ceph.client.user1.keyring  ceph.conf  rbdmap
root@node04:/etc/netplan# cd /etc/ceph/
root@node04:/etc/ceph# ll
total 20
drwxr-xr-x  2 root root 4096 Aug 24 21:10 ./
drwxr-xr-x 96 root root 4096 Aug 24 21:09 ../
-rw-------  1 root root  244 Aug 24 21:10 ceph.client.user1.keyring
-rw-r--r--  1 root root  265 Aug 24 21:10 ceph.conf
-rw-r--r--  1 root root   92 Jul  8 22:16 rbdmap
root@node04:/etc/ceph# ceph --user user1 -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 4225 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 70m)
    mgr: node01(active, since 70m), standbys: node02, node03
    osd: 6 osds: 6 up (since 70m), 6 in (since 7d)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 7 objects, 54 B
    usage:   44 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean
 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --user user1 -p rbd-data map rbdtest1.img
/dev/rbd0
rbd: --user is deprecated, use --id
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd0                      252:0    0    1G  0 disk 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
rbd: warning: image already mapped as /dev/rbd0
/dev/rbd1
root@node04:/etc/ceph# rbd --id user1 unmap rbd-data/rbdtest1.img
rbd: rbd-data/rbdtest1.img: mapped more than once, unmapping /dev/rbd0 only
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd1                      252:16   0    1G  0 disk 
root@node04:/etc/ceph# rbd --id user1 unmap rbd-data/rbdtest1.img
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
root@node04:/etc/ceph# 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
/dev/rbd0
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd0                      252:0    0    1G  0 disk 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0              isize=512    agcount=8, agsize=32768 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@node04:/etc/ceph# mount /dev/rbd0 /mnt/
root@node04:/etc/ceph# cp /etc/passwd /mnt/
root@node04:/etc/ceph# cd /mnt/
root@node04:/mnt# ls
passwd
root@node04:/mnt# tail passwd 
uuidd:x:107:112::/run/uuidd:/usr/sbin/nologin
tcpdump:x:108:113::/nonexistent:/usr/sbin/nologin
landscape:x:109:115::/var/lib/landscape:/usr/sbin/nologin
pollinate:x:110:1::/var/cache/pollinate:/bin/false
usbmux:x:111:46:usbmux daemon,,,:/var/lib/usbmux:/usr/sbin/nologin
sshd:x:112:65534::/run/sshd:/usr/sbin/nologin
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin
vmuser:x:1000:1000:vmuser:/home/vmuser:/bin/bash
lxd:x:998:100::/var/snap/lxd/common/lxd:/bin/false
ceph:x:64045:64045:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin
root@node04:/mnt# 

 下面是具體主要步驟命令：

服務上操作同步配置文件到客戶端：
root@node01:~# scp ceph.client.user1.keyring ceph-deploy/ceph.conf root@node04:/etc/ceph/

客戶端操作：
root@node04:/etc/netplan# apt install ceph-common
root@node04:/etc/netplan# ls /etc/ceph/
root@node04:/etc/ceph# ceph --user user1 -s
root@node04:/etc/ceph# rbd --user user1 -p rbd-data map rbdtest1.img（這台命令少用吧，用下面的）
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
root@node04:/etc/ceph# mkfs.xfs /dev/rbd0
root@node04:/etc/ceph# mount /dev/rbd0 /mnt/

1.5、普通用戶掛載CephFS情況：

root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 4986 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 83m)
    mgr: node01(active, since 83m), standbys: node02, node03
    osd: 6 osds: 6 up (since 83m), 6 in (since 7d)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 18 objects, 14 MiB
    usage:   132 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean
 
root@node01:~/ceph-deploy# ceph osd lspools
1 device_health_metrics
2 mypool1
3 rbd-data
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph osd pool create cephfs-metadata 16 16
pool 'cephfs-metadata' created
root@node01:~/ceph-deploy# ceph osd pool create cephfs-data 32 32
pool 'cephfs-data' created
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5126 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 85m)
    mgr: node01(active, since 85m), standbys: node02, node03
    osd: 6 osds: 6 up (since 85m), 6 in (since 7d)
 
  data:
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs new mycephfs ceph-metadata cephfs-data
Error ENOENT: pool 'ceph-metadata' does not exist
root@node01:~/ceph-deploy# ceph fs new mycephfs cephfs-metadata cephfs-data
new fs with metadata pool 4 and data pool 5
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            1 slow ops, oldest one blocked for 5221 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 87m)
    mgr: node01(active, since 87m), standbys: node02, node03
    mds: 0/0 daemons up
    osd: 6 osds: 6 up (since 87m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:0
root@node01:~/ceph-deploy# ceph mds status mycephfs
no valid command found; 10 closest matches:
mds metadata [<who>]
mds count-metadata <property>
mds versions
mds compat show
mds ok-to-stop <ids>...
mds fail <role_or_gid>
mds repaired <role>
mds rm <gid:int>
mds compat rm_compat <feature:int>
mds compat rm_incompat <feature:int>
Error EINVAL: invalid command
root@node01:~/ceph-deploy# ceph fs status mycephfs
mycephfs - 0 clients
========
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata     0   75.9G  
  cephfs-data      data       0   75.9G  
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            1 slow ops, oldest one blocked for 5291 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 88m)
    mgr: node01(active, since 88m), standbys: node02, node03
    mds: 0/0 daemons up
    osd: 6 osds: 6 up (since 88m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:0
root@node01:~/ceph-deploy# ls
ceph.bootstrap-mds.keyring  ceph.bootstrap-mgr.keyring  ceph.bootstrap-osd.keyring  ceph.bootstrap-rgw.keyring  ceph.client.admin.keyring  ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring
root@node01:~/ceph-deploy# ceph-de
ceph-debugpack  ceph-dencoder   ceph-deploy     
root@node01:~/ceph-deploy# ceph-deploy mds create node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f9007af8a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f9007acddd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node01', 'node01')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node01:node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node01
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node01][WARNIN] mds keyring does not exist yet, creating one
[node01][DEBUG ] create a keyring file
[node01][DEBUG ] create path if it doesn't exist
[node01][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node01 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node01/keyring
[node01][INFO  ] Running command: systemctl enable ceph-mds@node01
[node01][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/[e-mail] → /lib/systemd/system/ceph-mds@.service.
[node01][INFO  ] Running command: systemctl start ceph-mds@node01
[node01][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active}
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth add client.jack mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data'
added key for client.jack
root@node01:~/ceph-deploy# ceph auth get client.jack
[client.jack]
    key = AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==
    caps mds = "allow rw"
    caps mon = "allow r"
    caps osd = "allow rwx pool=cephfs-data"
exported keyring for client.jack
root@node01:~/ceph-deploy# ceph auth get client.admin
[client.admin]
    key = AQBErhthY4YdIhAANKTOMAjkzpKkHSkXSoNpaQ==
    caps mds = "allow *"
    caps mgr = "allow *"
    caps mon = "allow *"
    caps osd = "allow *"
exported keyring for client.admin
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth get client.jack -o ceph.client.jack.keyring
exported keyring for client.jack
root@node01:~/ceph-deploy# cat ceph.client.jack.keyring
[client.jack]
    key = AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==
    caps mds = "allow rw"
    caps mon = "allow r"
    caps osd = "allow rwx pool=cephfs-data"
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# scp ceph.client.jack.keyring node04:/etc/ceph/
root@node04's password: 
ceph.client.jack.keyring                                                                                                                                                        100%  148   118.7KB/s   00:00    
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth get-or-create-key cliet.jack
Error EINVAL: bad entity name
root@node01:~/ceph-deploy# ceph auth print-key cliet.jack
Error EINVAL: invalid entity_auth cliet.jack
root@node01:~/ceph-deploy# ceph auth print-key client.jack 
AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==root@node01:~/ceph-deploy# ceph auth print-key client.jack > jack.key
root@node01:~/ceph-deploy# scp jack.key node04:/etc/ceph/
root@node04's password: 
jack.key                                                                                                                                                                        100%   40    26.3KB/s   00:00    
root@node01:~/ceph-deploy#


root@node04:~# ceph --user jack -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5671 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 94m)
    mgr: node01(active, since 95m), standbys: node02, node03
    mds: 1/1 daemons up
    osd: 6 osds: 6 up (since 94m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 40 objects, 14 MiB
    usage:   138 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node04:~# ceph --id jack -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5676 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 94m)
    mgr: node01(active, since 95m), standbys: node02, node03
    mds: 1/1 daemons up
    osd: 6 osds: 6 up (since 94m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 40 objects, 14 MiB
    usage:   138 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node04:~# 
root@node04:~# mkdir /data
root@node04:~# 
root@node04:~# mount -t ceph node01:6789,node02:6789,node03:6789:/ /data/ -o name=jack,secretfile=/etc/ceph/jack.key
root@node04:~# df -hT
Filesystem                                                    Type      Size  Used Avail Use% Mounted on
udev                                                          devtmpfs  936M     0  936M   0% /dev
tmpfs                                                         tmpfs     196M  1.2M  195M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv                             ext4       20G  4.5G   15G  25% /
tmpfs                                                         tmpfs     980M     0  980M   0% /dev/shm
tmpfs                                                         tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                                                         tmpfs     980M     0  980M   0% /sys/fs/cgroup
/dev/sda2                                                     ext4      976M  106M  804M  12% /boot
/dev/loop0                                                    squashfs   56M   56M     0 100% /snap/core18/2128
/dev/loop1                                                    squashfs   56M   56M     0 100% /snap/core18/1944
/dev/loop2                                                    squashfs   32M   32M     0 100% /snap/snapd/10707
/dev/loop3                                                    squashfs   70M   70M     0 100% /snap/lxd/19188
tmpfs                                                         tmpfs     196M     0  196M   0% /run/user/0
/dev/loop4                                                    squashfs   33M   33M     0 100% /snap/snapd/12704
/dev/loop5                                                    squashfs   71M   71M     0 100% /snap/lxd/21029
/dev/rbd0                                                     xfs      1014M   40M  975M   4% /mnt
192.168.11.210:6789,192.168.11.220:6789,192.168.11.230:6789:/ ceph       76G     0   76G   0% /data
root@node04:~# cd /data/
root@node04:/data# echo 'test content' > testfile.txt
root@node04:/data# cat testfile.txt 
test content
root@node04:/data# 

 下面是具體主要步驟命令：

服務器端操作：
root@node01:~/ceph-deploy# ceph-deploy mds create node01
root@node01:~/ceph-deploy# ceph osd pool create cephfs-metadata 16 16
root@node01:~/ceph-deploy# ceph osd pool create cephfs-data 32 32
root@node01:~/ceph-deploy# ceph fs new mycephfs cephfs-metadata cephfs-data

root@node01:~/ceph-deploy# ceph auth add client.jack mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data'
root@node01:~/ceph-deploy# ceph auth get client.jack -o ceph.client.jack.keyring

root@node01:~/ceph-deploy# scp ceph.client.jack.keyring node04:/etc/ceph/
root@node01:~/ceph-deploy# ceph auth print-key client.jack > jack.key
root@node01:~/ceph-deploy# scp jack.key node04:/etc/ceph/

客戶端操作：
root@node04:~# ceph --user jack -s
root@node04:~# mkdir /data
root@node04:~# mount -t ceph node01:6789,node02:6789,node03:6789:/ /data/ -o name=jack,secretfile=/etc/ceph/jack.key

 

 2、CephFS的mds高可用情況：

 本次實驗的節點情況：

192.168.11.210    node01
192.168.11.220    node02
192.168.11.230    node03
192.168.11.240    node04  這個節點就是新加的作為上面客戶端和這里的mds

2.1、開始創建多個mds吧

root@node01:~/ceph-deploy# ceph-deploy mds create node02
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node02
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7febd49fda50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7febd49d2dd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node02', 'node02')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node02:node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node02
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node02][WARNIN] mds keyring does not exist yet, creating one
[node02][DEBUG ] create a keyring file
[node02][DEBUG ] create path if it doesn't exist
[node02][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node02 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node02/keyring
[node02][INFO  ] Running command: systemctl enable ceph-mds@node02
[node02][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/[e-mail] → /lib/systemd/system/ceph-mds@.service.
[node02][INFO  ] Running command: systemctl start ceph-mds@node02
[node02][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 1 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph-deploy mds create node03
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f1d42589a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f1d4255edd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node03', 'node03')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node03:node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node03
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node03][WARNIN] mds keyring does not exist yet, creating one
[node03][DEBUG ] create a keyring file
[node03][DEBUG ] create path if it doesn't exist
[node03][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node03 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node03/keyring
[node03][INFO  ] Running command: systemctl enable ceph-mds@node03
[node03][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/[e-mail] → /lib/systemd/system/ceph-mds@.service.
[node03][INFO  ] Running command: systemctl start ceph-mds@node03
[node03][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 1 up:standby
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 2 up:standby
root@node01:~/ceph-deploy# ceph-deploy mds create node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f72374d4a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f72374a9dd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node04', 'node04')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node04:node04
root@node04's password: 
bash: python2: command not found
[ceph_deploy.mds][ERROR ] connecting to host: node04 resulted in errors: IOError cannot send (already closed?)
[ceph_deploy][ERROR ] GenericError: Failed to create 1 MDSs

root@node01:~/ceph-deploy# ceph-deploy mds create node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4fa90baa50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f4fa908fdd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node04', 'node04')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node04:node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node04
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node04][WARNIN] mds keyring does not exist yet, creating one
[node04][DEBUG ] create a keyring file
[node04][DEBUG ] create path if it doesn't exist
[node04][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node04 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node04/keyring
[node04][INFO  ] Running command: systemctl enable ceph-mds@node04
[node04][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/[e-mail] → /lib/systemd/system/ceph-mds@.service.
[node04][INFO  ] Running command: systemctl start ceph-mds@node04
[node04][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 3 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 3 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   108k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
   node02    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs get mycephfs
Filesystem 'mycephfs' (1)
fs_name    mycephfs
epoch    5
flags    12
created    2021-08-24T21:27:30.730136+0800
modified    2021-08-24T21:29:55.774998+0800
tableserver    0
root    0
session_timeout    60
session_autoclose    300
max_file_size    1099511627776
required_client_features    {}
last_failure    0
last_failure_osd_epoch    0
compat    compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds    1
in    0
up    {0=65409}
failed    
damaged    
stopped    
data_pools    [5]
metadata_pool    4
inline_data    disabled
balancer    
standby_count_wanted    1
[mds.node01{0:65409} state up:active seq 2 addr [v2:192.168.11.210:6810/3443284017,v1:192.168.11.210:6811/3443284017]]
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs set mycephfs max_mds 2
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs get mycephfs
Filesystem 'mycephfs' (1)
fs_name    mycephfs
epoch    12
flags    12
created    2021-08-24T21:27:30.730136+0800
modified    2021-08-24T21:44:44.248039+0800
tableserver    0
root    0
session_timeout    60
session_autoclose    300
max_file_size    1099511627776
required_client_features    {}
last_failure    0
last_failure_osd_epoch    0
compat    compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds    2
in    0,1
up    {0=65409,1=64432}
failed    
damaged    
stopped    
data_pools    [5]
metadata_pool    4
inline_data    disabled
balancer    
standby_count_wanted    1
[mds.node01{0:65409} state up:active seq 2 addr [v2:192.168.11.210:6810/3443284017,v1:192.168.11.210:6811/3443284017]]
[mds.node02{1:64432} state up:active seq 41 addr [v2:192.168.11.220:6808/4242415336,v1:192.168.11.220:6809/4242415336]]
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 

 下面是具體主要步驟命令：

部署節點上（node01）上添加另外三個mds：
root@node01:~/ceph-deploy# ceph-deploy mds create node02
root@node01:~/ceph-deploy# ceph-deploy mds create node03
root@node01:~/ceph-deploy# ceph-deploy mds create node04
查看狀態：
root@node01:~/ceph-deploy# ceph mds stat
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   108k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
   node02    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 
root@node01:~/ceph-deploy# ceph fs get mycephfs
root@node01:~/ceph-deploy# ceph fs set mycephfs max_mds 2
root@node01:~/ceph-deploy# ceph fs status

此時兩主，兩備，任何一個主出現問題，兩個備節點中的一個會自動頂上去：

root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)

重啟node01節點

root@node01:~/ceph-deploy# systemctl restart ceph
ceph-crash.service       [e-mail]  [e-mail]  [e-mail]  [e-mail]       ceph-osd.target          ceph.service             
ceph-fuse.target         ceph-mds.target          ceph-mgr.target          ceph-mon.target          [e-mail]       ceph-radosgw.target      ceph.target              
root@node01:~/ceph-deploy# systemctl restart ceph-m
[e-mail]  ceph-mds.target          [e-mail]  ceph-mgr.target          [e-mail]  ceph-mon.target          
root@node01:~/ceph-deploy# systemctl restart ceph-mds
[e-mail]  ceph-mds.target          
root@node01:~/ceph-deploy# systemctl restart [e-mail] 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s     1      4      2      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 

如果想做成一對一的綁定——node01和node02互為主備，node03和node04互為主備，下面這樣整一下吧（主要是配置文件）：

root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node04
mds_standby_replay = true
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node01
mds_standby_replay = true
root@node01:~/ceph-deploy# ceph-deploy config push node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy config push node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4fdbb800f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f4fdbbd6350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to config 1 hosts

root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf config push node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f201b0280f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f201b07e350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf config push node02 node02 node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push node02 node02 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f549232a0f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node02', 'node02', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f5492380350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart [e-mail] 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph-deploy admin node01 node02 node03 node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy admin node01 node02 node03 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f9fa57ccf50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01', 'node02', 'node03', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f9fa58a54d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node04
root@node04's password: 
Permission denied, please try again.
root@node04's password: 忘了做免密同步，不過也沒上面，就輸入一下密碼即可
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy][ERROR ] GenericError: Failed to configure 1 admin hosts

root@node01:~/ceph-deploy# ceph-deploy admin node01 node02 node03 node04 --overwrite-conf 
usage: ceph-deploy [-h] [-v | -q] [--version] [--username USERNAME]
                   [--overwrite-conf] [--ceph-conf CEPH_CONF]
                   COMMAND ...
ceph-deploy: error: unrecognized arguments: --overwrite-conf
root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf admin node01 node02 node03 node04 
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf admin node01 node02 node03 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f41f97a5f50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01', 'node02', 'node03', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f41f987e4d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ls
ceph.bootstrap-mds.keyring  ceph.bootstrap-osd.keyring  ceph.client.admin.keyring  ceph.conf             ceph.mon.keyring
ceph.bootstrap-mgr.keyring  ceph.bootstrap-rgw.keyring  ceph.client.jack.keyring   ceph-deploy-ceph.log  jack.key
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s     1      4      2      0   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# cat ceph.c
cat: ceph.c: No such file or directory
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node01
mds_standby_replay = true
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart [e-mail] 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node04  Reqs:    0 /s    10     13     12      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 

好了就這樣吧！！！

 

================================ 下面是我的整個過程的log ===================================

 

root@node01:~# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL    USED  RAW USED  %RAW USED
hdd    240 GiB  240 GiB  41 MiB    41 MiB       0.02
TOTAL  240 GiB  240 GiB  41 MiB    41 MiB       0.02
 
--- POOLS ---
POOL                   ID  PGS   STORED  OBJECTS    USED  %USED  MAX AVAIL
device_health_metrics   1    1      0 B        0     0 B      0     76 GiB
mypool1                 2   32  3.1 KiB        3  21 KiB      0     76 GiB
root@node01:~# 
root@node01:~# ceph osd pool create rbd-data 32 32
pool 'rbd-data' created
root@node02:~# ceph osd lspools
1 device_health_metrics
2 mypool1
3 rbd-data
root@node02:~# 
root@node01:~# ceph osd pool ls
device_health_metrics
mypool1
rbd-data
root@node01:~# ceph osd pool application enable rbd-data rbd
enabled application 'rbd' on pool 'rbd-data'
root@node01:~# rb
rbash            rbd              rbdmap           rbd-replay       rbd-replay-many  rbd-replay-prep  
root@node01:~# rbd pool init -p rbd-data
root@node01:~# rbd create rbdtest1.img --pool rbd-data --size 1G
root@node01:~# rbd ls --pool rbd-data
rbdtest1.img
root@node01:~# rbd info rbd-data/rbdtest1.img
rbd image 'rbdtest1.img':
    size 1 GiB in 256 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: fb9a70e1dad6
    block_name_prefix: rbd_data.fb9a70e1dad6
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Tue Aug 24 20:17:40 2021
    access_timestamp: Tue Aug 24 20:17:40 2021
    modify_timestamp: Tue Aug 24 20:17:40 2021
root@node01:~# rbd feature disable rbd-data/rbdtest1.img exclusive-lock object-map fast-diff  deep-flatten
root@node01:~# rbd info rbd-data/rbdtest1.img
rbd image 'rbdtest1.img':
    size 1 GiB in 256 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: fb9a70e1dad6
    block_name_prefix: rbd_data.fb9a70e1dad6
    format: 2
    features: layering
    op_features: 
    flags: 
    create_timestamp: Tue Aug 24 20:17:40 2021
    access_timestamp: Tue Aug 24 20:17:40 2021
    modify_timestamp: Tue Aug 24 20:17:40 2021
root@node01:~# 







root@node01:~# ceph auth add client.tom mon 'allow r' osd 'allow rwx pool=rbd-date'
added key for client.tom
root@node01:~# ceph auth get client.tom
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
exported keyring for client.tom
root@node01:~# 
root@node01:~# 
root@node01:~# ceph auth get
get                get-key            get-or-create      get-or-create-key  
root@node01:~# ceph auth get-or-create client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# 
root@node01:~# ceph auth get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.jerry
root@node01:~# 
root@node01:~# ceph auth get-or-create-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# ceph auth get-or-create-key client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# ceph auth get-or-create-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# 
root@node01:~# ceph auth print-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==root@node01:~# 
root@node01:~# 
root@node01:~# 
root@node01:~# ceph auth  get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.jerry
root@node01:~# ceph auth caps client.jerry mon 'allow rw' osd 'allow r pool=rbd-data'
updated caps for client.jerry
root@node01:~# ceph auth get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow rw"
    caps osd = "allow r pool=rbd-data"
exported keyring for client.jerry
root@node01:~# ceph auth  del client.jerry
updated
root@node01:~# ceph auth get client.jerry
Error ENOENT: failed to find client.jerry in keyring
root@node01:~# 



5.6.1:通 通 過 秘 鑰 環 文 件 備 份 與 恢 復 用

root@node01:~# ceph auth get-or-create client.user1 mon 'allow r' osd 'allow rwx pool=rbd-data'
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
root@node01:~# ceph auth get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# ls
ceph-deploy  cluster  snap
root@node01:~# ceph-authtool --help
usage: ceph-authtool keyringfile [OPTIONS]...
where the options are:
  -l, --list                    will list all keys and capabilities present in
                                the keyring
  -p, --print-key               will print an encoded key for the specified
                                entityname. This is suitable for the
                                'mount -o secret=..' argument
  -C, --create-keyring          will create a new keyring, overwriting any
                                existing keyringfile
  -g, --gen-key                 will generate a new secret key for the
                                specified entityname
  --gen-print-key               will generate a new secret key without set it
                                to the keyringfile, prints the secret to stdout
  --import-keyring FILE         will import the content of a given keyring
                                into the keyringfile
  -n NAME, --name NAME          specify entityname to operate on
  -a BASE64, --add-key BASE64   will add an encoded key to the keyring
  --cap SUBSYSTEM CAPABILITY    will set the capability for given subsystem
  --caps CAPSFILE               will set all of capabilities associated with a
                                given key, for all subsystems
  --mode MODE                   will set the desired file mode to the keyring
                                e.g: '0644', defaults to '0600'
root@node01:~# ceph-authtool -C ceph.client.user1.keyring
creating ceph.client.user1.keyring
root@node01:~# cat ceph.client.user1.keyring 
root@node01:~# file ceph.client.user1.keyring 
ceph.client.user1.keyring: empty
root@node01:~# 
root@node01:~# ceph auth  get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# ceph auth  get client.user1 -o ceph.client.user1.keyring 
exported keyring for client.user1
root@node01:~# cat ceph.client.user1.keyring 
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
root@node01:~# 

從 keyring 文件恢復用戶認證信息:


root@node01:~# 
root@node01:~# ceph auth del client.user1
updated
root@node01:~# ceph auth get client.user1
Error ENOENT: failed to find client.user1 in keyring
root@node01:~# 
root@node01:~# ceph auth import -i ceph.client.user1.keyring 
imported keyring
root@node01:~# 
root@node01:~# ceph auth get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# 





root@node01:~# ceph-authtool -C ceph.client.tom.keyring
creating ceph.client.tom.keyring
root@node01:~# ceph auth get client.tom -o ceph.client.tom.keyring 
exported keyring for client.tom
root@node01:~# cat ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# ceph-authtool -l ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# 
root@node01:~# ceph-authtool ceph.client.user1.keyring --import-keyring ceph.client.tom.keyring 
importing contents of ceph.client.tom.keyring into ceph.client.user1.keyring
root@node01:~# echo $?
0
root@node01:~# cat ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# 
root@node01:~# cat ceph.client.user1.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
root@node01:~# 





普通用戶掛載
同步配置文件：
root@node01:~# ls
ceph.client.tom.keyring  ceph.client.user1.keyring  ceph-deploy  client.client.user1.keyring  client.user1.keyring  cluster  snap
root@node01:~# scp ceph.client.user1.keyring ceph-deploy/ceph.conf root@node04:/etc/ceph/
root@node04's password: 
ceph.client.user1.keyring                                                                                                                                                       100%  244   263.4KB/s   00:00    
ceph.conf                                                                                                                                                                       100%  265    50.5KB/s   00:00    
root@node01:~#

客戶端操作

root@node04:/etc/netplan# apt install ceph-common
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  ibverbs-providers libbabeltrace1 libcephfs2 libdw1 libgoogle-perftools4 libibverbs1 libjaeger libleveldb1d liblttng-ust-ctl4 liblttng-ust0 liblua5.3-0 libnl-route-3-200 liboath0 librabbitmq4 librados2
  libradosstriper1 librbd1 librdkafka1 librdmacm1 librgw2 libsnappy1v5 libtcmalloc-minimal4 python3-ceph-argparse python3-ceph-common python3-cephfs python3-prettytable python3-rados python3-rbd python3-rgw
Suggested packages:
  ceph-base ceph-mds
The following NEW packages will be installed:
  ceph-common ibverbs-providers libbabeltrace1 libcephfs2 libdw1 libgoogle-perftools4 libibverbs1 libjaeger libleveldb1d liblttng-ust-ctl4 liblttng-ust0 liblua5.3-0 libnl-route-3-200 liboath0 librabbitmq4
  librados2 libradosstriper1 librbd1 librdkafka1 librdmacm1 librgw2 libsnappy1v5 libtcmalloc-minimal4 python3-ceph-argparse python3-ceph-common python3-cephfs python3-prettytable python3-rados python3-rbd
  python3-rgw
0 upgraded, 30 newly installed, 0 to remove and 79 not upgraded.
Need to get 37.3 MB of archives.
After this operation, 152 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific focal/main amd64 libjaeger amd64 16.2.5-1focal [3,780 B]

中間太長了，我就刪除了點

Setting up ceph-common (16.2.5-1focal) ...
Adding group ceph....done
Adding system user ceph....done
Setting system user ceph properties....done
chown: cannot access '/var/log/ceph/*.log*': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target.
Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service.
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
root@node04:/etc/netplan# ls /etc/ceph/
rbdmap
root@node04:/etc/netplan# ls /etc/ceph/
ceph.client.user1.keyring  ceph.conf  rbdmap
root@node04:/etc/netplan# cd /etc/ceph/
root@node04:/etc/ceph# ll
total 20
drwxr-xr-x  2 root root 4096 Aug 24 21:10 ./
drwxr-xr-x 96 root root 4096 Aug 24 21:09 ../
-rw-------  1 root root  244 Aug 24 21:10 ceph.client.user1.keyring
-rw-r--r--  1 root root  265 Aug 24 21:10 ceph.conf
-rw-r--r--  1 root root   92 Jul  8 22:16 rbdmap
root@node04:/etc/ceph# ceph --user user1 -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 4225 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 70m)
    mgr: node01(active, since 70m), standbys: node02, node03
    osd: 6 osds: 6 up (since 70m), 6 in (since 7d)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 7 objects, 54 B
    usage:   44 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean
 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --user user1 -p rbd-data map rbdtest1.img
/dev/rbd0
rbd: --user is deprecated, use --id
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd0                      252:0    0    1G  0 disk 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
rbd: warning: image already mapped as /dev/rbd0
/dev/rbd1
root@node04:/etc/ceph# rbd --id user1 unmap rbd-data/rbdtest1.img
rbd: rbd-data/rbdtest1.img: mapped more than once, unmapping /dev/rbd0 only
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd1                      252:16   0    1G  0 disk 
root@node04:/etc/ceph# rbd --id user1 unmap rbd-data/rbdtest1.img
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
root@node04:/etc/ceph# 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
/dev/rbd0
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd0                      252:0    0    1G  0 disk 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0              isize=512    agcount=8, agsize=32768 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@node04:/etc/ceph# mount /dev/rbd0 /mnt/
root@node04:/etc/ceph# cp /etc/passwd /mnt/
root@node04:/etc/ceph# cd /mnt/
root@node04:/mnt# ls
passwd
root@node04:/mnt# tail passwd 
uuidd:x:107:112::/run/uuidd:/usr/sbin/nologin
tcpdump:x:108:113::/nonexistent:/usr/sbin/nologin
landscape:x:109:115::/var/lib/landscape:/usr/sbin/nologin
pollinate:x:110:1::/var/cache/pollinate:/bin/false
usbmux:x:111:46:usbmux daemon,,,:/var/lib/usbmux:/usr/sbin/nologin
sshd:x:112:65534::/run/sshd:/usr/sbin/nologin
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin
vmuser:x:1000:1000:vmuser:/home/vmuser:/bin/bash
lxd:x:998:100::/var/snap/lxd/common/lxd:/bin/false
ceph:x:64045:64045:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin
root@node04:/mnt# 



cephfs掛載：
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 4986 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 83m)
    mgr: node01(active, since 83m), standbys: node02, node03
    osd: 6 osds: 6 up (since 83m), 6 in (since 7d)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 18 objects, 14 MiB
    usage:   132 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean
 
root@node01:~/ceph-deploy# ceph osd lspools
1 device_health_metrics
2 mypool1
3 rbd-data
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph osd pool create cephfs-metadata 16 16
pool 'cephfs-metadata' created
root@node01:~/ceph-deploy# ceph osd pool create cephfs-data 32 32
pool 'cephfs-data' created
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5126 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 85m)
    mgr: node01(active, since 85m), standbys: node02, node03
    osd: 6 osds: 6 up (since 85m), 6 in (since 7d)
 
  data:
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs new mycephfs ceph-metadata cephfs-data
Error ENOENT: pool 'ceph-metadata' does not exist
root@node01:~/ceph-deploy# ceph fs new mycephfs cephfs-metadata cephfs-data
new fs with metadata pool 4 and data pool 5
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            1 slow ops, oldest one blocked for 5221 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 87m)
    mgr: node01(active, since 87m), standbys: node02, node03
    mds: 0/0 daemons up
    osd: 6 osds: 6 up (since 87m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:0
root@node01:~/ceph-deploy# ceph mds status mycephfs
no valid command found; 10 closest matches:
mds metadata [<who>]
mds count-metadata <property>
mds versions
mds compat show
mds ok-to-stop <ids>...
mds fail <role_or_gid>
mds repaired <role>
mds rm <gid:int>
mds compat rm_compat <feature:int>
mds compat rm_incompat <feature:int>
Error EINVAL: invalid command
root@node01:~/ceph-deploy# ceph fs status mycephfs
mycephfs - 0 clients
========
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata     0   75.9G  
  cephfs-data      data       0   75.9G  
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            1 slow ops, oldest one blocked for 5291 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 88m)
    mgr: node01(active, since 88m), standbys: node02, node03
    mds: 0/0 daemons up
    osd: 6 osds: 6 up (since 88m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:0
root@node01:~/ceph-deploy# ls
ceph.bootstrap-mds.keyring  ceph.bootstrap-mgr.keyring  ceph.bootstrap-osd.keyring  ceph.bootstrap-rgw.keyring  ceph.client.admin.keyring  ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring
root@node01:~/ceph-deploy# ceph-de
ceph-debugpack  ceph-dencoder   ceph-deploy     
root@node01:~/ceph-deploy# ceph-deploy mds create node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f9007af8a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f9007acddd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node01', 'node01')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node01:node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node01
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node01][WARNIN] mds keyring does not exist yet, creating one
[node01][DEBUG ] create a keyring file
[node01][DEBUG ] create path if it doesn't exist
[node01][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node01 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node01/keyring
[node01][INFO  ] Running command: systemctl enable ceph-mds@node01
[node01][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/[e-mail] → /lib/systemd/system/ceph-mds@.service.
[node01][INFO  ] Running command: systemctl start ceph-mds@node01
[node01][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active}
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth add client.jack mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data'
added key for client.jack
root@node01:~/ceph-deploy# ceph auth get client.jack
[client.jack]
    key = AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==
    caps mds = "allow rw"
    caps mon = "allow r"
    caps osd = "allow rwx pool=cephfs-data"
exported keyring for client.jack
root@node01:~/ceph-deploy# ceph auth get client.admin
[client.admin]
    key = AQBErhthY4YdIhAANKTOMAjkzpKkHSkXSoNpaQ==
    caps mds = "allow *"
    caps mgr = "allow *"
    caps mon = "allow *"
    caps osd = "allow *"
exported keyring for client.admin
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth get client.jack -o ceph.client.jack.keyring
exported keyring for client.jack
root@node01:~/ceph-deploy# cat ceph.client.jack.keyring
[client.jack]
    key = AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==
    caps mds = "allow rw"
    caps mon = "allow r"
    caps osd = "allow rwx pool=cephfs-data"
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# scp ceph.client.jack.keyring node04:/etc/ceph/
root@node04's password: 
ceph.client.jack.keyring                                                                                                                                                        100%  148   118.7KB/s   00:00    
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth get-or-create-key cliet.jack
Error EINVAL: bad entity name
root@node01:~/ceph-deploy# ceph auth print-key cliet.jack
Error EINVAL: invalid entity_auth cliet.jack
root@node01:~/ceph-deploy# ceph auth print-key client.jack 
AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==root@node01:~/ceph-deploy# ceph auth print-key client.jack > jack.key
root@node01:~/ceph-deploy# scp jack.key node04:/etc/ceph/
root@node04's password: 
jack.key                                                                                                                                                                        100%   40    26.3KB/s   00:00    
root@node01:~/ceph-deploy#


root@node04:~# ceph --user jack -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5671 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 94m)
    mgr: node01(active, since 95m), standbys: node02, node03
    mds: 1/1 daemons up
    osd: 6 osds: 6 up (since 94m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 40 objects, 14 MiB
    usage:   138 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node04:~# ceph --id jack -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5676 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 94m)
    mgr: node01(active, since 95m), standbys: node02, node03
    mds: 1/1 daemons up
    osd: 6 osds: 6 up (since 94m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 40 objects, 14 MiB
    usage:   138 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node04:~# 
root@node04:~# mkdir /data
root@node04:~# 
root@node04:~# mount -t ceph node01:6789,node02:6789,node03:6789:/ /data/ -o name=jack,secretfile=/etc/ceph/jack.key
root@node04:~# df -hT
Filesystem                                                    Type      Size  Used Avail Use% Mounted on
udev                                                          devtmpfs  936M     0  936M   0% /dev
tmpfs                                                         tmpfs     196M  1.2M  195M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv                             ext4       20G  4.5G   15G  25% /
tmpfs                                                         tmpfs     980M     0  980M   0% /dev/shm
tmpfs                                                         tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                                                         tmpfs     980M     0  980M   0% /sys/fs/cgroup
/dev/sda2                                                     ext4      976M  106M  804M  12% /boot
/dev/loop0                                                    squashfs   56M   56M     0 100% /snap/core18/2128
/dev/loop1                                                    squashfs   56M   56M     0 100% /snap/core18/1944
/dev/loop2                                                    squashfs   32M   32M     0 100% /snap/snapd/10707
/dev/loop3                                                    squashfs   70M   70M     0 100% /snap/lxd/19188
tmpfs                                                         tmpfs     196M     0  196M   0% /run/user/0
/dev/loop4                                                    squashfs   33M   33M     0 100% /snap/snapd/12704
/dev/loop5                                                    squashfs   71M   71M     0 100% /snap/lxd/21029
/dev/rbd0                                                     xfs      1014M   40M  975M   4% /mnt
192.168.11.210:6789,192.168.11.220:6789,192.168.11.230:6789:/ ceph       76G     0   76G   0% /data
root@node04:~# cd /data/
root@node04:/data# echo 'test content' > testfile.txt
root@node04:/data# cat testfile.txt 
test content
root@node04:/data# 

上面是用戶創建及掛載，下面是：mds高可用情況：

root@node01:~/ceph-deploy# ceph-deploy mds create node02
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node02
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7febd49fda50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7febd49d2dd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node02', 'node02')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node02:node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node02
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node02][WARNIN] mds keyring does not exist yet, creating one
[node02][DEBUG ] create a keyring file
[node02][DEBUG ] create path if it doesn't exist
[node02][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node02 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node02/keyring
[node02][INFO  ] Running command: systemctl enable ceph-mds@node02
[node02][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/[e-mail] → /lib/systemd/system/ceph-mds@.service.
[node02][INFO  ] Running command: systemctl start ceph-mds@node02
[node02][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 1 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph-deploy mds create node03
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f1d42589a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f1d4255edd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node03', 'node03')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node03:node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node03
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node03][WARNIN] mds keyring does not exist yet, creating one
[node03][DEBUG ] create a keyring file
[node03][DEBUG ] create path if it doesn't exist
[node03][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node03 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node03/keyring
[node03][INFO  ] Running command: systemctl enable ceph-mds@node03
[node03][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/[e-mail] → /lib/systemd/system/ceph-mds@.service.
[node03][INFO  ] Running command: systemctl start ceph-mds@node03
[node03][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 1 up:standby
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 2 up:standby
root@node01:~/ceph-deploy# ceph-deploy mds create node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f72374d4a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f72374a9dd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node04', 'node04')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node04:node04
root@node04's password: 
bash: python2: command not found
[ceph_deploy.mds][ERROR ] connecting to host: node04 resulted in errors: IOError cannot send (already closed?)
[ceph_deploy][ERROR ] GenericError: Failed to create 1 MDSs

root@node01:~/ceph-deploy# ceph-deploy mds create node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4fa90baa50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f4fa908fdd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node04', 'node04')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node04:node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node04
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node04][WARNIN] mds keyring does not exist yet, creating one
[node04][DEBUG ] create a keyring file
[node04][DEBUG ] create path if it doesn't exist
[node04][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node04 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node04/keyring
[node04][INFO  ] Running command: systemctl enable ceph-mds@node04
[node04][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/[e-mail] → /lib/systemd/system/ceph-mds@.service.
[node04][INFO  ] Running command: systemctl start ceph-mds@node04
[node04][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 3 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 3 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   108k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
   node02    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs get mycephfs
Filesystem 'mycephfs' (1)
fs_name    mycephfs
epoch    5
flags    12
created    2021-08-24T21:27:30.730136+0800
modified    2021-08-24T21:29:55.774998+0800
tableserver    0
root    0
session_timeout    60
session_autoclose    300
max_file_size    1099511627776
required_client_features    {}
last_failure    0
last_failure_osd_epoch    0
compat    compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds    1
in    0
up    {0=65409}
failed    
damaged    
stopped    
data_pools    [5]
metadata_pool    4
inline_data    disabled
balancer    
standby_count_wanted    1
[mds.node01{0:65409} state up:active seq 2 addr [v2:192.168.11.210:6810/3443284017,v1:192.168.11.210:6811/3443284017]]
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs set mycephfs max_mds 2
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs get mycephfs
Filesystem 'mycephfs' (1)
fs_name    mycephfs
epoch    12
flags    12
created    2021-08-24T21:27:30.730136+0800
modified    2021-08-24T21:44:44.248039+0800
tableserver    0
root    0
session_timeout    60
session_autoclose    300
max_file_size    1099511627776
required_client_features    {}
last_failure    0
last_failure_osd_epoch    0
compat    compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds    2
in    0,1
up    {0=65409,1=64432}
failed    
damaged    
stopped    
data_pools    [5]
metadata_pool    4
inline_data    disabled
balancer    
standby_count_wanted    1
[mds.node01{0:65409} state up:active seq 2 addr [v2:192.168.11.210:6810/3443284017,v1:192.168.11.210:6811/3443284017]]
[mds.node02{1:64432} state up:active seq 41 addr [v2:192.168.11.220:6808/4242415336,v1:192.168.11.220:6809/4242415336]]
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 



root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart ceph
ceph-crash.service       [e-mail]  [e-mail]  [e-mail]  [e-mail]       ceph-osd.target          ceph.service             
ceph-fuse.target         ceph-mds.target          ceph-mgr.target          ceph-mon.target          [e-mail]       ceph-radosgw.target      ceph.target              
root@node01:~/ceph-deploy# systemctl restart ceph-m
[e-mail]  ceph-mds.target          [e-mail]  ceph-mgr.target          [e-mail]  ceph-mon.target          
root@node01:~/ceph-deploy# systemctl restart ceph-mds
[e-mail]  ceph-mds.target          
root@node01:~/ceph-deploy# systemctl restart [e-mail] 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s     1      4      2      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node04
mds_standby_replay = true
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node01
mds_standby_replay = true
root@node01:~/ceph-deploy# ceph-deploy config push node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy config push node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4fdbb800f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f4fdbbd6350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to config 1 hosts

root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf config push node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f201b0280f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f201b07e350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf config push node02 node02 node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push node02 node02 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f549232a0f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node02', 'node02', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f5492380350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart [e-mail] 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph-deploy admin node01 node02 node03 node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy admin node01 node02 node03 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f9fa57ccf50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01', 'node02', 'node03', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f9fa58a54d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node04
root@node04's password: 
Permission denied, please try again.
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy][ERROR ] GenericError: Failed to configure 1 admin hosts

root@node01:~/ceph-deploy# ceph-deploy admin node01 node02 node03 node04 --overwrite-conf 
usage: ceph-deploy [-h] [-v | -q] [--version] [--username USERNAME]
                   [--overwrite-conf] [--ceph-conf CEPH_CONF]
                   COMMAND ...
ceph-deploy: error: unrecognized arguments: --overwrite-conf
root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf admin node01 node02 node03 node04 
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf admin node01 node02 node03 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f41f97a5f50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01', 'node02', 'node03', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f41f987e4d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ls
ceph.bootstrap-mds.keyring  ceph.bootstrap-osd.keyring  ceph.client.admin.keyring  ceph.conf             ceph.mon.keyring
ceph.bootstrap-mgr.keyring  ceph.bootstrap-rgw.keyring  ceph.client.jack.keyring   ceph-deploy-ceph.log  jack.key
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s     1      4      2      0   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# cat ceph.c
cat: ceph.c: No such file or directory
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node01
mds_standby_replay = true
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart [e-mail] 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node04  Reqs:    0 /s    10     13     12      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy#