文章索引：
一、Ceph組件：
1.OSD（Object Storage Daemon）
功能：Ceph OSDs（對象存儲守護程序ceph-osd）：提供數據存儲，操作系統上的一個磁盤就是一個OSD守護程序，用於處理ceph集群數據復制、回復、重新平衡，並通過檢查其他Ceph OSD守護程序的心跳來向Ceph監視器和管理器提供一些監視信息，實現冗余和高可用性至少需要3個Ceph OSD。
2.Mon （monitor）：ceph的監視器
功能：一個主機上運行的一個守護進程，用於維護集群狀態映射（maintains maps of the cluster state），如ceph集群中有多少存儲池、每個存儲池有多少個PG以及存儲池的PG的映射關系等，一個ceph集群至少有一個Mon（1,3,5,7...），Ceph守護程序相互協調所需的關鍵集群狀態有：monitor map，manager map，the OSD map，the MDS map和the CRUSH map。
3.Mgr（Manager）管理器
功能：一個主機上運行的一個守護進程，Ceph Manager守護程序負責跟蹤運行時，指標和Ceph集群的當前狀態，包括存儲利用率，當前性能指標和系統負載。還托管基於Python的模塊來管理和公開Ceph集群信息，包括基於Web的Ceph儀表板和REST API。高可用至少需要兩個管理器。
二、Ceph數據讀寫流程：
- 計算文件到對象的映射，得到oid（object id）= ino+non：
- ino：iNode number （INO），File的元數據序列號，File的唯一id
- ono：object number （ONO），File切分產生的某個object的序號，默認以4M切分一個塊大小
- 通過hash算法計算出文件對應的pool中的PG：
通過一致性HASH計算object到PG，Object --> PG映射的hash（oid）&mask --> pgid
- 通過CRUSH把對象映射到PG中的OSD
通過CRUSH算法計算PG到OSD，PG --> OSD映射：[CRUSH(pgid)->(osd1,osd2,osd3)]
- PG中的主OSD將對象寫入到硬盤
- 主OSD將數據同步到備份OSD，並等待備份OSD返回確認
- 主OSD將寫入完成返回給客戶端。
說明：
Pool：存儲池、分區，存儲池的大小取決於底層的存儲空間。
PG（placement group）：一個pool內部可以有多個PG存在，Pool和PG都是抽象的邏輯概念，一個pool中有多少個PG可以通過公式計算。
OSD（Object storage Daemon，對象存儲設備）：每一塊磁盤都是一個osd，一個主機由一個或多個osd組成。
ceph集群部署好之后，要先創建存儲池才能向ceph寫入數據，文件在向ceph保存之前要先進行一致性hash計算，計算后會把文件保存在某個對應的PG中，此文件一定屬於某個pool的一個PG，在通過PG保存在OSD上。數據對象在寫到主OSD之后再同步到從OSD以實現數據的高可用。
三、部署ceph集群
| 服務器角色 | 系統版本 | IP地址 | 基本配置及分區大小 |
| ceph-deploy | Ubuntu 1804 | 10.0.0.100/192.168.0.100 | 2C2G/120G |
| ceph-mon1 | Ubuntu 1804 | 10.0.0.101/192.168.0.101 | 2C2G/120G |
| ceph-mon2 | Ubuntu 1804 | 10.0.0.102/192.168.0.102 | 2C2G/120G |
| ceph-mon3 | Ubuntu 1804 | 10.0.0.103/192.168.0.103 | 2C2G/120G |
| ceph-mgr1 | Ubuntu 1804 | 10.0.0.104/192.168.0.104 | 2C2G/120G |
| ceph-mgr2 | Ubuntu 1804 | 10.0.0.105/192.168.0.105 | 2C2G/120G |
| ceph-node1 | Ubuntu 1804 | 10.0.0.106/192.168.0.106 | 2C2G/120G+100*5 |
| ceph-node2 | Ubuntu 1804 | 10.0.0.107/192.168.0.107 | 2C2G/120G+100*5 |
| ceph-node3 | Ubuntu 1804 | 10.0.0.108/192.168.0.108 | 2C2G/120G+100*5 |
| ceph-node4 | Ubuntu 1804 | 10.0.0.109/192.168.0.109 | 2C2G/120G+100*5 |
環境簡介：
1、一個服務器用語部署ceph集群即安裝ceph-deploy，也可以和cepy-mgr等復用。
10.0.0.100/192.168.0.100
2、三台服務器作為ceph集群Mon監控服務器，每台服務器可以和ceph集群的cluster網絡通信
10.0.0.101/192.168.0.101
10.0.0.102/192.168.0.102
10.0.0.103/192.168.0.103
3、兩個ceph-mgr管理器，可以和ceph集群的cluster網絡通信
10.0.0.104/192.168.0.104
10.0.0.105/192.168.0.105
4、四台服務器作為ceph集群OSD存儲服務器，每台服務器支持兩個網絡，public網絡針對客戶端使用，cluster網絡用於集群管理及數據同步，每台3塊以上硬盤
10.0.0.106/192.168.0.106
10.0.0.107/192.168.0.107
10.0.0.108/192.168.0.108
10.0.0.109/192.168.0.109
#各存儲服務器磁盤划分：
/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf #100G
5、創建一個普通用戶，能夠通過sudo執行特權命令，配置主機名解析，ceph集群部署過程中需要對各主機配置不同的主機名，另外如果是centos系統則需要關閉各服務器的防火牆和selinux。
1、更改主機名
# cat /etc/hostname Ubuntu1804 # hostnamectl set-hostname ceph-deploy.example.lcoal #hostnamectl set-hostname 更改后的主機名 # cat /etc/hostname ceph-deploy.example.lcoal
2、更改網卡名稱為eth*
方法一：安裝Ubuntu系統界面時傳遞內核參數：net.ifnames=0 biosdevname=0
方法二：如果沒有在安裝系統之前傳遞內核參數將網卡名稱更改為eth*，可以通過如下方式更改（需重啟Ubuntu系統）：
3、配置root遠程登錄
默認情況下，Ubuntu不允許root用戶遠程ssh，需添加root密碼並編輯/etc/ssh/sshd_config文件：
$ sudo vim /etc/ssh/sshd_config 32 #PermitRootLogin prohibit-password 33 PermitRootLogin yes #允許root登錄 101 #UseDNS no 102 UseDNS no #關閉DNS解析
$ sudo su - root #切換至root用戶 # passwd #設置root密碼 Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully # systemctl restart sshd #重啟ssh服務
4、各節點配置服務器網卡，例如ceph-deploy：
root@ceph-deploy:~# cat /etc/netplan/01-netcfg.yaml # This file describes the network interfaces available on your system # For more information, see netplan(5). network: version: 2 renderer: networkd ethernets: eth0: dhcp4: no dhcp6: no addresses: [10.0.0.100/24] gateway4: 10.0.0.2 nameservers: addresses: [10.0.0.2, 114.114.114.114, 8.8.8.8] eth1: dhcp4: no dhcp6: no addresses: [192.168.0.100/24] root@ceph-deploy:~# ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.0.0.100 netmask 255.255.255.0 broadcast 10.0.0.255 inet6 fe80::20c:29ff:fe65:a300 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:65:a3:00 txqueuelen 1000 (Ethernet) RX packets 2057 bytes 172838 (172.8 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1575 bytes 221983 (221.9 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.100 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::20c:29ff:fe65:a30a prefixlen 64 scopeid 0x20<link> ether 00:0c:29:65:a3:0a txqueuelen 1000 (Ethernet) RX packets 2 bytes 486 (486.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 14 bytes 1076 (1.0 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 182 bytes 14992 (14.9 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 182 bytes 14992 (14.9 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 root@ceph-deploy:~# ping -c 1 -i 1 www.baidu.com
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=1 ttl=128 time=6.67 ms
5、配置apt倉庫
https://mirrors.aliyun.com/ceph/ #阿里雲鏡像倉庫
http://mirrors.163.com/ceph/ #網易鏡像倉庫
https://mirrors.tuna.tsinghua.edu.cn/ceph/ #清華大學鏡像源倉庫
$ wget -q -O- 'https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc' | sudo apt-key add - #導入key文件 OK $ sudo echo "deb https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-pacific bionic main" >> /etc/apt/sources.list $ cat /etc/apt/sources.list # 默認注釋了源碼鏡像以提高 apt update 速度，如有需要可自行取消注釋 deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic main #echo追加該條內容
$ sudo apt update
1、創建cephadmin用戶：
推薦使用指定的普通用戶部署、運行ceph集群，普通用戶只要能以交互式方式使用sudo命令執行一些特權命令即可，新版的ceph-deploy可以指定包含root在內的只要可以執行sudo命令的用戶，不過仍推薦使用普通用戶，如：cephuser、cephadmin這樣的用戶去管理ceph集群。
在包含ceph-deploy節點的存儲節點、Mon節點和mgr節點創建cephadmin用戶。
groupadd -r -g 2022 cephadmin && useradd -r -m -s /bin/bash -u 2022 -g 2022 cephadmin && echo cephadmin:123.com | chpasswd
各服務允許cephadmin用戶使用sudo執行特權命令：
echo "cephadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
2、配置免秘鑰登錄：
在ceph-deploy節點配置允許以非交互的方式登錄到各ceph node/mon/mgr節點，及在ceph-deploy節點生成密鑰對，然后分發公鑰到各被管理節點：
cephadmin@ceph-deploy:~$ ssh-keygen #生成ssh秘鑰對 Generating public/private rsa key pair. Enter file in which to save the key (/home/cephadmin/.ssh/id_rsa): Created directory '/home/cephadmin/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/cephadmin/.ssh/id_rsa. Your public key has been saved in /home/cephadmin/.ssh/id_rsa.pub. The key fingerprint is: SHA256:0+vL5tnFkcEzFiGCmKTzR7G58KHrbUB9qBiaqtYsSi4 cephadmin@ceph-deploy The key's randomart image is: +---[RSA 2048]----+ | ..o... . o. | | .o .+ . o . | | o ..=. * | | .o.=o+. . = | | o +o.S.. o | | o . oo . . . . | | oo .. . o | |Eo o . ..o.o . | |B.. ...o*.. | +----[SHA256]-----+ cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.100 #分發公鑰至各管理節點（包括自身） cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.101 cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.102 cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.103 cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.104 cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.105 cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.106 cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.107 cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.108 cephadmin@ceph-deploy:~$ ssh-copy-id cephadmin@10.0.0.109
3、各節點配置域名解析：
# cat >> /etc/hosts << EOF 10.0.0.100 ceph-deploy 10.0.0.101 ceph-mon1 10.0.0.102 ceph-mon2 10.0.0.103 ceph-mon3 10.0.0.104 ceph-mgr1 10.0.0.105 ceph-mgr2 10.0.0.106 ceph-node1 10.0.0.107 ceph-node2 10.0.0.108 ceph-node3 10.0.0.109 ceph-node4 EOF
4、在各節點安裝Python2：
# apt -y install python2.7 #安裝Python2.7 # ln -sv /usr/bin/python2.7 /usr/bin/python2 #創建軟鏈接
5、安裝ceph部署工具
在ceph部署服務器安裝部署工具ceph-deploy
cephadmin@ceph-deploy:~$ apt-cache madison ceph-deploy ceph-deploy | 2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages ceph-deploy | 2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main i386 Packages ceph-deploy | 1.5.38-0ubuntu1 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe amd64 Packages ceph-deploy | 1.5.38-0ubuntu1 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe i386 Packages cephadmin@ceph-deploy:~$ sudo apt -y install ceph-deploy
6、初始化Mon節點
在ceph-deploy管理節點初始化Mon節點，Mon節點也需要有cluster network，否則初始化會報錯；
cephadmin@ceph-deploy:~$ mkdir ceph-cluster cephadmin@ceph-deploy:~$ cd ceph-cluster/ cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy new --cluster-network 192.168.0.0/24 --public-network 10.0.0.0/24 ceph-mon1
驗證初始化：
cephadmin@ceph-deploy:~/ceph-cluster$ ll total 20 drwxrwxr-x 2 cephadmin cephadmin 4096 Aug 18 15:26 ./ drwxr-xr-x 6 cephadmin cephadmin 4096 Aug 18 15:20 ../ -rw-rw-r-- 1 cephadmin cephadmin 259 Aug 18 15:26 ceph.conf #自動生成的配置文件 -rw-rw-r-- 1 cephadmin cephadmin 3892 Aug 18 15:26 ceph-deploy-ceph.log #初始化日志 -rw------- 1 cephadmin cephadmin 73 Aug 18 15:26 ceph.mon.keyring #用於ceph Mon節點內部通訊認證的秘鑰環文件 cephadmin@ceph-deploy:~/ceph-cluster$ cat ceph.conf [global] fsid = 0d11d338-a480-40da-8520-830423b22c3e #ceph的集群ID public_network = 10.0.0.0/24 cluster_network = 192.168.0.0/24 mon_initial_members = ceph-mon1 #可以用逗號做分割添加多個mon節點 mon_host = 10.0.0.101 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx
配置Mon節點並生成同步秘鑰
在各Mon節點安裝組件ceph-mon，並初始化Mon節點，Mon節點可以后期橫向擴容
root@ceph-mon1:~# apt -y install ceph-mon cephadmin@ceph-deploy:~/ceph-cluster$ pwd /home/cephadmin/ceph-cluster cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy mon create-initial [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy mon create-initial [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] subcommand : create-initial [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f0903be4fa0> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] func : <function mon at 0x7f0903bc8ad0> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] keyrings : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-mon1 [ceph_deploy.mon][DEBUG ] detecting platform for host ceph-mon1 ... [ceph-mon1][DEBUG ] connection detected need for sudo [ceph-mon1][DEBUG ] connected to host: ceph-mon1 [ceph-mon1][DEBUG ] detect platform information from remote host [ceph-mon1][DEBUG ] detect machine type [ceph-mon1][DEBUG ] find the location of an executable [ceph_deploy.mon][INFO ] distro info: Ubuntu 18.04 bionic [ceph-mon1][DEBUG ] determining if provided host has same hostname in remote [ceph-mon1][DEBUG ] get remote short hostname [ceph-mon1][DEBUG ] deploying mon to ceph-mon1 [ceph-mon1][DEBUG ] get remote short hostname [ceph-mon1][DEBUG ] remote hostname: ceph-mon1 [ceph-mon1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph-mon1][DEBUG ] create the mon path if it does not exist [ceph-mon1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-mon1/done [ceph-mon1][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-mon1/done [ceph-mon1][INFO ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-mon1.mon.keyring [ceph-mon1][DEBUG ] create the monitor keyring file [ceph-mon1][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i ceph-mon1 --keyring /var/lib/ceph/tmp/ceph-ceph-mon1.mon.keyring --setuser 64045 --setgroup 64045 [ceph-mon1][INFO ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-mon1.mon.keyring [ceph-mon1][DEBUG ] create a done file to avoid re-doing the mon deployment [ceph-mon1][DEBUG ] create the init path if it does not exist [ceph-mon1][INFO ] Running command: sudo systemctl enable ceph.target [ceph-mon1][INFO ] Running command: sudo systemctl enable ceph-mon@ceph-mon1 [ceph-mon1][WARNIN] Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@ceph-mon1.service → /lib/systemd/system/ceph-mon@.service. [ceph-mon1][INFO ] Running command: sudo systemctl start ceph-mon@ceph-mon1 [ceph-mon1][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon1.asok mon_status [ceph-mon1][DEBUG ] ******************************************************************************** [ceph-mon1][DEBUG ] status for monitor: mon.ceph-mon1 [ceph-mon1][DEBUG ] { [ceph-mon1][DEBUG ] "election_epoch": 3, [ceph-mon1][DEBUG ] "extra_probe_peers": [], [ceph-mon1][DEBUG ] "feature_map": { [ceph-mon1][DEBUG ] "mon": [ [ceph-mon1][DEBUG ] { [ceph-mon1][DEBUG ] "features": "0x3f01cfb9fffdffff", [ceph-mon1][DEBUG ] "num": 1, [ceph-mon1][DEBUG ] "release": "luminous" [ceph-mon1][DEBUG ] } [ceph-mon1][DEBUG ] ] [ceph-mon1][DEBUG ] }, [ceph-mon1][DEBUG ] "features": { [ceph-mon1][DEBUG ] "quorum_con": "4540138297136906239", [ceph-mon1][DEBUG ] "quorum_mon": [ [ceph-mon1][DEBUG ] "kraken", [ceph-mon1][DEBUG ] "luminous", [ceph-mon1][DEBUG ] "mimic", [ceph-mon1][DEBUG ] "osdmap-prune", [ceph-mon1][DEBUG ] "nautilus", [ceph-mon1][DEBUG ] "octopus", [ceph-mon1][DEBUG ] "pacific", [ceph-mon1][DEBUG ] "elector-pinging" [ceph-mon1][DEBUG ] ], [ceph-mon1][DEBUG ] "required_con": "2449958747317026820", [ceph-mon1][DEBUG ] "required_mon": [ [ceph-mon1][DEBUG ] "kraken", [ceph-mon1][DEBUG ] "luminous", [ceph-mon1][DEBUG ] "mimic", [ceph-mon1][DEBUG ] "osdmap-prune", [ceph-mon1][DEBUG ] "nautilus", [ceph-mon1][DEBUG ] "octopus", [ceph-mon1][DEBUG ] "pacific", [ceph-mon1][DEBUG ] "elector-pinging" [ceph-mon1][DEBUG ] ] [ceph-mon1][DEBUG ] }, [ceph-mon1][DEBUG ] "monmap": { [ceph-mon1][DEBUG ] "created": "2021-08-18T07:55:40.349602Z", [ceph-mon1][DEBUG ] "disallowed_leaders: ": "", [ceph-mon1][DEBUG ] "election_strategy": 1, [ceph-mon1][DEBUG ] "epoch": 1, [ceph-mon1][DEBUG ] "features": { [ceph-mon1][DEBUG ] "optional": [], [ceph-mon1][DEBUG ] "persistent": [ [ceph-mon1][DEBUG ] "kraken", [ceph-mon1][DEBUG ] "luminous", [ceph-mon1][DEBUG ] "mimic", [ceph-mon1][DEBUG ] "osdmap-prune", [ceph-mon1][DEBUG ] "nautilus", [ceph-mon1][DEBUG ] "octopus", [ceph-mon1][DEBUG ] "pacific", [ceph-mon1][DEBUG ] "elector-pinging" [ceph-mon1][DEBUG ] ] [ceph-mon1][DEBUG ] }, [ceph-mon1][DEBUG ] "fsid": "0d11d338-a480-40da-8520-830423b22c3e", [ceph-mon1][DEBUG ] "min_mon_release": 16, [ceph-mon1][DEBUG ] "min_mon_release_name": "pacific", [ceph-mon1][DEBUG ] "modified": "2021-08-18T07:55:40.349602Z", [ceph-mon1][DEBUG ] "mons": [ [ceph-mon1][DEBUG ] { [ceph-mon1][DEBUG ] "addr": "10.0.0.101:6789/0", [ceph-mon1][DEBUG ] "crush_location": "{}", [ceph-mon1][DEBUG ] "name": "ceph-mon1", [ceph-mon1][DEBUG ] "priority": 0, [ceph-mon1][DEBUG ] "public_addr": "10.0.0.101:6789/0", [ceph-mon1][DEBUG ] "public_addrs": { [ceph-mon1][DEBUG ] "addrvec": [ [ceph-mon1][DEBUG ] { [ceph-mon1][DEBUG ] "addr": "10.0.0.101:3300", [ceph-mon1][DEBUG ] "nonce": 0, [ceph-mon1][DEBUG ] "type": "v2" [ceph-mon1][DEBUG ] }, [ceph-mon1][DEBUG ] { [ceph-mon1][DEBUG ] "addr": "10.0.0.101:6789", [ceph-mon1][DEBUG ] "nonce": 0, [ceph-mon1][DEBUG ] "type": "v1" [ceph-mon1][DEBUG ] } [ceph-mon1][DEBUG ] ] [ceph-mon1][DEBUG ] }, [ceph-mon1][DEBUG ] "rank": 0, [ceph-mon1][DEBUG ] "weight": 0 [ceph-mon1][DEBUG ] } [ceph-mon1][DEBUG ] ], [ceph-mon1][DEBUG ] "stretch_mode": false [ceph-mon1][DEBUG ] }, [ceph-mon1][DEBUG ] "name": "ceph-mon1", [ceph-mon1][DEBUG ] "outside_quorum": [], [ceph-mon1][DEBUG ] "quorum": [ [ceph-mon1][DEBUG ] 0 [ceph-mon1][DEBUG ] ], [ceph-mon1][DEBUG ] "quorum_age": 1, [ceph-mon1][DEBUG ] "rank": 0, [ceph-mon1][DEBUG ] "state": "leader", [ceph-mon1][DEBUG ] "stretch_mode": false, [ceph-mon1][DEBUG ] "sync_provider": [] [ceph-mon1][DEBUG ] } [ceph-mon1][DEBUG ] ******************************************************************************** [ceph-mon1][INFO ] monitor: mon.ceph-mon1 is running [ceph-mon1][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon1.asok mon_status [ceph_deploy.mon][INFO ] processing monitor mon.ceph-mon1 [ceph-mon1][DEBUG ] connection detected need for sudo [ceph-mon1][DEBUG ] connected to host: ceph-mon1 [ceph-mon1][DEBUG ] detect platform information from remote host [ceph-mon1][DEBUG ] detect machine type [ceph-mon1][DEBUG ] find the location of an executable [ceph-mon1][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-mon1.asok mon_status [ceph_deploy.mon][INFO ] mon.ceph-mon1 monitor has reached quorum! [ceph_deploy.mon][INFO ] all initial monitors are running and have formed quorum [ceph_deploy.mon][INFO ] Running gatherkeys... [ceph_deploy.gatherkeys][INFO ] Storing keys in temp directory /tmp/tmpqCeuN6 [ceph-mon1][DEBUG ] connection detected need for sudo [ceph-mon1][DEBUG ] connected to host: ceph-mon1 [ceph-mon1][DEBUG ] detect platform information from remote host [ceph-mon1][DEBUG ] detect machine type [ceph-mon1][DEBUG ] get remote short hostname [ceph-mon1][DEBUG ] fetch remote file [ceph-mon1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.ceph-mon1.asok mon_status [ceph-mon1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-mon1/keyring auth get client.admin [ceph-mon1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-mon1/keyring auth get client.bootstrap-mds [ceph-mon1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-mon1/keyring auth get client.bootstrap-mgr [ceph-mon1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-mon1/keyring auth get client.bootstrap-osd [ceph-mon1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-mon1/keyring auth get client.bootstrap-rgw [ceph_deploy.gatherkeys][INFO ] Storing ceph.client.admin.keyring [ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mds.keyring [ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mgr.keyring [ceph_deploy.gatherkeys][INFO ] keyring 'ceph.mon.keyring' already exists [ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring [ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring [ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmpqCeuN6
驗證Mon節點
驗證在mon節點已經安裝並啟動了ceph-mon服務，並且后期在ceph-deploy節點初始化目錄會生成一些bootstrap ceph mds/mgr/osd/rgw等服務的keyring認證文件，這些初始化文件擁有對ceph集群的最高特權，所以一定要保存好。
root@ceph-mon1:~# ps -ef | grep ceph-mon ceph 6688 1 0 15:55 ? 00:00:00 /usr/bin/ceph-mon -f --cluster ceph --id ceph-mon1 --setuser ceph --setgroup ceph root 7252 2514 0 16:00 pts/0 00:00:00 grep --color=auto ceph-mon
7、配置manager節點
部署ceph-mgr節點：
mgr節點需要讀取ceph的配置文件，及/etc/ceph目錄中的配置文件
#初始化ceph-mgr節點： root@ceph-mgr1:~# apt -y install ceph-mgr cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy mgr create ceph-mgr1 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy mgr create ceph-mgr1 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] mgr : [('ceph-mgr1', 'ceph-mgr1')] [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] subcommand : create [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f8d17024c30> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] func : <function mgr at 0x7f8d17484150> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts ceph-mgr1:ceph-mgr1 The authenticity of host 'ceph-mgr1 (10.0.0.104)' can't be established. ECDSA key fingerprint is SHA256:Y7Y9tQOTjbM8RnmDHvT8eJBzIu8ZPdaBkG9jBg8bifA. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ceph-mgr1' (ECDSA) to the list of known hosts. [ceph-mgr1][DEBUG ] connection detected need for sudo [ceph-mgr1][DEBUG ] connected to host: ceph-mgr1 [ceph-mgr1][DEBUG ] detect platform information from remote host [ceph-mgr1][DEBUG ] detect machine type [ceph_deploy.mgr][INFO ] Distro info: Ubuntu 18.04 bionic [ceph_deploy.mgr][DEBUG ] remote host will use systemd [ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-mgr1 [ceph-mgr1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph-mgr1][WARNIN] mgr keyring does not exist yet, creating one [ceph-mgr1][DEBUG ] create a keyring file [ceph-mgr1][DEBUG ] create path recursively if it doesn't exist [ceph-mgr1][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-mgr1 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-mgr1/keyring [ceph-mgr1][INFO ] Running command: sudo systemctl enable ceph-mgr@ceph-mgr1 [ceph-mgr1][WARNIN] Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@ceph-mgr1.service → /lib/systemd/system/ceph-mgr@.service. [ceph-mgr1][INFO ] Running command: sudo systemctl start ceph-mgr@ceph-mgr1 [ceph-mgr1][INFO ] Running command: sudo systemctl enable ceph.target
驗證ceph-mgr節點：
root@ceph-mgr1:~# ps -ef | grep ceph-mgr ceph 8128 1 8 17:09 ? 00:00:03 /usr/bin/ceph-mgr -f --cluster ceph --id ceph-mgr1 --setuser ceph --setgroup ceph root 8326 2396 0 17:10 pts/0 00:00:00 grep --color=auto ceph-mgr
8、分發admin秘鑰：
在ceph-deploy節點把配置文件和admin秘鑰拷貝至ceph集群需要執行ceph管理命令的節點，從而不需要后期通過ceph命令對ceph集群進行管理配置的時候每次都需要指定ceph-mon節點地址和ceph.client.admin.keyring文件，另外各ceph-mon節點也需要同步ceph的集群配置文件及認證文件。
在ceph-deploy節點管理集群：
root@ceph-deploy:~# apt -y install ceph-common #安裝ceph公共組件，安裝ceph-common需要使用root root@cepn-node1:~# apt -y install ceph-common root@cepn-node2:~# apt -y install ceph-common root@cepn-node3:~# apt -y install ceph-common root@cepn-node4:~# apt -y install ceph-common cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy admin ceph-deploy ceph-node1 ceph-node2 ceph-node3 ceph-node4 #分發admin秘鑰 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy admin ceph-deploy [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4ba41a4190> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] client : ['ceph-deploy'] [ceph_deploy.cli][INFO ] func : <function admin at 0x7f4ba4aa5a50> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-deploy [ceph-deploy][DEBUG ] connection detected need for sudo [ceph-deploy][DEBUG ] connected to host: ceph-deploy [ceph-deploy][DEBUG ] detect platform information from remote host [ceph-deploy][DEBUG ] detect machine type [ceph-deploy][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy admin ceph-node1 ceph-node2 ceph-node3 ceph-node4 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc78eac3190> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] client : ['ceph-node1', 'ceph-node2', 'ceph-node3', 'ceph-node4'] [ceph_deploy.cli][INFO ] func : <function admin at 0x7fc78f3c4a50> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node1 The authenticity of host 'ceph-node1 (10.0.0.106)' can't be established. ECDSA key fingerprint is SHA256:Y7Y9tQOTjbM8RnmDHvT8eJBzIu8ZPdaBkG9jBg8bifA. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ceph-node1' (ECDSA) to the list of known hosts. [ceph-node1][DEBUG ] connection detected need for sudo [ceph-node1][DEBUG ] connected to host: ceph-node1 [ceph-node1][DEBUG ] detect platform information from remote host [ceph-node1][DEBUG ] detect machine type [ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node2 The authenticity of host 'ceph-node2 (10.0.0.107)' can't be established. ECDSA key fingerprint is SHA256:Y7Y9tQOTjbM8RnmDHvT8eJBzIu8ZPdaBkG9jBg8bifA. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ceph-node2' (ECDSA) to the list of known hosts. [ceph-node2][DEBUG ] connection detected need for sudo [ceph-node2][DEBUG ] connected to host: ceph-node2 [ceph-node2][DEBUG ] detect platform information from remote host [ceph-node2][DEBUG ] detect machine type [ceph-node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node3 The authenticity of host 'ceph-node3 (10.0.0.108)' can't be established. ECDSA key fingerprint is SHA256:Y7Y9tQOTjbM8RnmDHvT8eJBzIu8ZPdaBkG9jBg8bifA. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ceph-node3' (ECDSA) to the list of known hosts. [ceph-node3][DEBUG ] connection detected need for sudo [ceph-node3][DEBUG ] connected to host: ceph-node3 [ceph-node3][DEBUG ] detect platform information from remote host [ceph-node3][DEBUG ] detect machine type [ceph-node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-node4 The authenticity of host 'ceph-node4 (10.0.0.109)' can't be established. ECDSA key fingerprint is SHA256:Y7Y9tQOTjbM8RnmDHvT8eJBzIu8ZPdaBkG9jBg8bifA. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ceph-node4' (ECDSA) to the list of known hosts. [ceph-node4][DEBUG ] connection detected need for sudo [ceph-node4][DEBUG ] connected to host: ceph-node4 [ceph-node4][DEBUG ] detect platform information from remote host [ceph-node4][DEBUG ] detect machine type [ceph-node4][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
ceph節點驗證秘鑰：
到ceph-node節點驗證key文件
root@cepn-node1:~# ll /etc/ceph/ total 20 drwxr-xr-x 2 root root 4096 Aug 18 16:38 ./ drwxr-xr-x 91 root root 4096 Aug 18 16:27 ../ -rw------- 1 root root 151 Aug 18 16:38 ceph.client.admin.keyring -rw-r--r-- 1 root root 259 Aug 18 16:38 ceph.conf -rw-r--r-- 1 root root 92 Jul 8 22:17 rbdmap -rw------- 1 root root 0 Aug 18 16:38 tmp4MDGPp
認證文件的屬主和屬組為了安全考慮，默認設置為root用戶和root組，如果需要cephadmin用戶也能執行ceph命令，需要對cephadmin用戶進行授權
cephadmin@ceph-deploy:~/ceph-cluster$ sudo setfacl -m u:cephadmin:rw /etc/ceph/ceph.client.admin.keyring root@cepn-node1:~# setfacl -m u:cephadmin:rw /etc/ceph/ceph.client.admin.keyring root@cepn-node2:~# setfacl -m u:cephadmin:rw /etc/ceph/ceph.client.admin.keyring root@cepn-node3:~# setfacl -m u:cephadmin:rw /etc/ceph/ceph.client.admin.keyring root@cepn-node4:~# setfacl -m u:cephadmin:rw /etc/ceph/ceph.client.admin.keyring
測試ceph命令：
cephadmin@ceph-deploy:~/ceph-cluster$ ceph -s cluster: id: 0d11d338-a480-40da-8520-830423b22c3e health: HEALTH_WARN mon is allowing insecure global_id reclaim #需要禁用非安全模式通信 OSD count 0 < osd_pool_default_size 3 #集群的OSD數量小於3 services: mon: 1 daemons, quorum ceph-mon1 (age 98m) mgr: ceph-mgr1(active, since 24m) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: cephadmin@ceph-deploy:~/ceph-cluster$ ceph config set mon auth_allow_insecure_global_id_reclaim false #禁用非安全模式通信 cephadmin@ceph-deploy:~/ceph-cluster$ ceph -s cluster: id: 0d11d338-a480-40da-8520-830423b22c3e health: HEALTH_WARN OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum ceph-mon1 (age 100m) mgr: ceph-mgr1(active, since 26m) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: cephadmin@ceph-deploy:~/ceph-cluster$
9、初始化node節點過程
添加OSD之前，需要對node節點安裝基本環境：
cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy install --no-adjust-repos --nogpgcheck ceph-node1 ceph-node2 ceph-node3 ceph-node4
--no-adjust-repos install packages without modifying source repos
--nogpgcheck install packages without gpgcheck
擦除硬盤
cephadmin@ceph-deploy:~/ceph-cluster
$ ceph-deploy disk list ceph-node1 ceph-node2 ceph-node3 ceph-node4
#
列出遠端存儲node節點的磁盤信息
使用ceph-deploy disk zap 擦除各caph node數據磁盤
cephadmin@ceph-deploy:~/ceph-cluster$ cat EraseDisk.sh #!/bin/bash # for i in {1..4}; do for d in {b..f}; do ceph-deploy disk zap ceph-node$i /dev/sd$d done done cephadmin@ceph-deploy:~/ceph-cluster$ bash -n EraseDisk.sh cephadmin@ceph-deploy:~/ceph-cluster$ bash EraseDisk.sh
添加主機的磁盤OSD：
數據分類保存方式：
- Data：即ceph保存的對象數據
- Block：rocks DB數據，即元數據
- block-wal：數據庫的wal日志
添加OSD（OSD的ID從0開始順序使用）：
cephadmin@ceph-deploy:~/ceph-cluster$ cat CreateDisk.sh #!/bin/bash # for i in {1..4}; do for d in {b..f}; do ceph-deploy osd create ceph-node$i --data /dev/sd$d done done cephadmin@ceph-deploy:~/ceph-cluster$ bash -n CreateDisk.sh cephadmin@ceph-deploy:~/ceph-cluster$ bash CreateDisk.sh
10、驗證ceph集群：
cephadmin@ceph-deploy:~/ceph-cluster$ ceph -s cluster: id: 0d11d338-a480-40da-8520-830423b22c3e health: HEALTH_OK services: mon: 1 daemons, quorum ceph-mon1 (age 2h) mgr: ceph-mgr1(active, since 88m) osd: 20 osds: 20 up (since 55s), 20 in (since 63s) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 150 MiB used, 2.0 TiB / 2.0 TiB avail pgs: 1 active+clean
11、測試上傳與下載數據：
存取數據時，客戶端必須首先連接至RADOS集群上某存儲池，然后根據對象名稱由相關的CRUSH規格完成數據對象尋址，於是，為了測試集群的數據存取功能，這里首先先創建一個用於測試的存儲池mypool，並設定其PG數量為32個。
$ ceph -h #一個更底層的命令 $ rados -h #
創建pool
[ceph@ceph-deploy ceph-cluster]$ ceph osd pool create mypool 32 32 #32個PG和32種PGD組合 pool 'mypool' created cephadmin@ceph-deploy:~/ceph-cluster$ ceph pg ls-by-pool mypool | awk '{print $1,$2,$15}' #驗證PG與PGP組合 PG OBJECTS ACTING 2.0 0 [8,10,3]p8 2.1 0 [15,0,13]p15 2.2 0 [5,1,15]p5 2.3 0 [17,5,14]p17 2.4 0 [1,12,18]p1 2.5 0 [12,4,8]p12 2.6 0 [1,13,19]p1 2.7 0 [6,17,2]p6 2.8 0 [16,13,0]p16 2.9 0 [4,9,19]p4 2.a 0 [11,4,18]p11 2.b 0 [13,7,17]p13 2.c 0 [12,0,5]p12 2.d 0 [12,19,3]p12 2.e 0 [2,13,19]p2 2.f 0 [11,17,8]p11 2.10 0 [15,13,0]p15 2.11 0 [16,6,1]p16 2.12 0 [10,3,9]p10 2.13 0 [17,6,3]p17 2.14 0 [8,13,17]p8 2.15 0 [19,1,11]p19 2.16 0 [8,12,17]p8 2.17 0 [6,14,2]p6 2.18 0 [18,9,12]p18 2.19 0 [3,6,13]p3 2.1a 0 [6,14,2]p6 2.1b 0 [11,7,17]p11 2.1c 0 [10,7,1]p10 2.1d 0 [15,10,7]p15 2.1e 0 [3,13,15]p3 2.1f 0 [4,7,14]p4 * NOTE: afterwards cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd tree #查看osd與存儲服務器的對應關系 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.95374 root default -5 0.48843 host ceph-node2 5 hdd 0.09769 osd.5 up 1.00000 1.00000 6 hdd 0.09769 osd.6 up 1.00000 1.00000 7 hdd 0.09769 osd.7 up 1.00000 1.00000 8 hdd 0.09769 osd.8 up 1.00000 1.00000 9 hdd 0.09769 osd.9 up 1.00000 1.00000 -7 0.48843 host ceph-node3 10 hdd 0.09769 osd.10 up 1.00000 1.00000 11 hdd 0.09769 osd.11 up 1.00000 1.00000 12 hdd 0.09769 osd.12 up 1.00000 1.00000 13 hdd 0.09769 osd.13 up 1.00000 1.00000 14 hdd 0.09769 osd.14 up 1.00000 1.00000 -9 0.48843 host ceph-node4 15 hdd 0.09769 osd.15 up 1.00000 1.00000 16 hdd 0.09769 osd.16 up 1.00000 1.00000 17 hdd 0.09769 osd.17 up 1.00000 1.00000 18 hdd 0.09769 osd.18 up 1.00000 1.00000 19 hdd 0.09769 osd.19 up 1.00000 1.00000 -3 0.48843 host cepn-node1 0 hdd 0.09769 osd.0 up 1.00000 1.00000 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2 hdd 0.09769 osd.2 up 1.00000 1.00000 3 hdd 0.09769 osd.3 up 1.00000 1.00000 4 hdd 0.09769 osd.4 up 1.00000 1.00000 cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool ls #查看當前存儲池 device_health_metrics mypool cephadmin@ceph-deploy:~/ceph-cluster$ rados lspools #查看當前存儲池 device_health_metrics mypool
當前的ceph環境還沒有部署使用塊存儲和文件系統使用ceph，也沒有使用對象存儲的客戶端，但是ceph的rados命令可以實現訪問ceph對象存儲的功能：
上傳文件：
cephadmin@ceph-deploy:~/ceph-cluster$ sudo rados put msg1 /var/log/syslog --pool=mypool #把文件上傳至mypool並指定對象ID為msg1
列出文件：
cephadmin@ceph-deploy:~/ceph-cluster$ rados ls --pool=mypool
msg1
文件信息：
ceph osd map 命令：獲取到存儲池中數據對象的具體位置信息
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd map mypool msg1
osdmap e131 pool 'mypool' (2) object 'msg1' -> pg 2.c833d430 (2.10) -> up ([15,13,0], p15) acting ([15,13,0], p15)
2.c844d430：表示文件放在了存儲池ID為2的c844d430的PG上
2.10：表示數據存儲在ID為2的存儲池、ID為10的PG中
[15,13,0],p15：OSD編號，主OSD為15，活動的OSD 15,13,0，三個OSD表示數據存放一共3個副本，PG中的OSD是ceph的crush算法計算出三份數據保存在哪些OSD中
下載文件：
cephadmin@ceph-deploy:~/ceph-cluster$ sudo rados get msg1 --pool=mypool /opt/my.txt cephadmin@ceph-deploy:~/ceph-cluster$ ll /opt/ total 1840 drwxr-xr-x 2 root root 4096 Aug 20 15:23 ./ drwxr-xr-x 23 root root 4096 Aug 18 18:32 ../ -rw-r--r-- 1 root root 1873597 Aug 20 15:23 my.txt
#驗證下載文件： cephadmin@ceph-deploy:~/ceph-cluster$ head /opt/my.txt Aug 18 18:33:40 ceph-deploy systemd-modules-load[484]: Inserted module 'iscsi_tcp' Aug 18 18:33:40 ceph-deploy systemd-modules-load[484]: Inserted module 'ib_iser' Aug 18 18:33:40 ceph-deploy systemd[1]: Starting Flush Journal to Persistent Storage... Aug 18 18:33:40 ceph-deploy systemd[1]: Started Load/Save Random Seed. Aug 18 18:33:40 ceph-deploy systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling. Aug 18 18:33:40 ceph-deploy systemd[1]: Started udev Kernel Device Manager. Aug 18 18:33:40 ceph-deploy systemd[1]: Started Set the console keyboard layout. Aug 18 18:33:40 ceph-deploy systemd[1]: Reached target Local File Systems (Pre). Aug 18 18:33:40 ceph-deploy systemd[1]: Reached target Local File Systems. Aug 18 18:33:40 ceph-deploy systemd[1]: Starting Tell Plymouth To Write Out Runtime Data...
修改文件：
cephadmin@ceph-deploy:~/ceph-cluster$ sudo rados put msg1 /etc/passwd --pool=mypool cephadmin@ceph-deploy:~/ceph-cluster$ sudo rados get msg1 --pool=mypool /opt/my1.txt
#驗證修改后的文件 cephadmin@ceph-deploy:~/ceph-cluster$ tail /opt/my1.txt _apt:x:104:65534::/nonexistent:/usr/sbin/nologin lxd:x:105:65534::/var/lib/lxd/:/bin/false uuidd:x:106:110::/run/uuidd:/usr/sbin/nologin dnsmasq:x:107:65534:dnsmasq,,,:/var/lib/misc:/usr/sbin/nologin landscape:x:108:112::/var/lib/landscape:/usr/sbin/nologin sshd:x:109:65534::/run/sshd:/usr/sbin/nologin pollinate:x:110:1::/var/cache/pollinate:/bin/false wang:x:1000:1000:wang,,,:/home/wang:/bin/bash cephadmin:x:2022:2022::/home/cephadmin:/bin/bash ceph:x:64045:64045:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin
刪除文件
cephadmin@ceph-deploy:~/ceph-cluster$ sudo rados rm msg1 --pool=mypool cephadmin@ceph-deploy:~/ceph-cluster$ rados ls --pool=mypool cephadmin@ceph-deploy:~/ceph-cluster$
12、擴展ceph集群實現高可用：
主要是擴展ceph集群的mon節點以及mgr節點以實現集群高可用。
擴展ceph-mon節點：
Ceph-mon是原生具備自選舉以實現高可用機制的ceph服務，節點數量通常是奇數。
root@ceph-mon2:~# apt -y install ceph-mon root@ceph-mon3:~# apt -y install ceph-mon cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy mon add ceph-mon2 cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy mon add ceph-mon3
驗證ceph-mon狀態：
cephadmin@ceph-deploy:~/ceph-cluster$ ceph quorum_status --format json-pretty { "election_epoch": 14, "quorum": [ 0, 1, 2 ], "quorum_names": [ "ceph-mon1", "ceph-mon2", "ceph-mon3" ], "quorum_leader_name": "ceph-mon1", #當前的leader "quorum_age": 304, "features": { "quorum_con": "4540138297136906239", "quorum_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ] }, "monmap": { "epoch": 3, "fsid": "0d11d338-a480-40da-8520-830423b22c3e", "modified": "2021-08-20T07:39:56.803507Z", "created": "2021-08-18T07:55:40.349602Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, #當前節點等級 "name": "ceph-mon1", #當前節點名稱 "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.0.0.101:3300", "nonce": 0 }, { "type": "v1", "addr": "10.0.0.101:6789", "nonce": 0 } ] }, "addr": "10.0.0.101:6789/0", #監聽地址 "public_addr": "10.0.0.101:6789/0", #監聽地址 "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "ceph-mon2", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.0.0.102:3300", "nonce": 0 }, { "type": "v1", "addr": "10.0.0.102:6789", "nonce": 0 } ] }, "addr": "10.0.0.102:6789/0", "public_addr": "10.0.0.102:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 2, "name": "ceph-mon3", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.0.0.103:3300", "nonce": 0 }, { "type": "v1", "addr": "10.0.0.103:6789", "nonce": 0 } ] }, "addr": "10.0.0.103:6789/0", "public_addr": "10.0.0.103:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] } } cephadmin@ceph-deploy:~/ceph-cluster$ ceph -s cluster: id: 0d11d338-a480-40da-8520-830423b22c3e health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 28m) mgr: ceph-mgr1(active, since 6h) osd: 20 osds: 20 up (since 6h), 20 in (since 45h) data: pools: 2 pools, 33 pgs objects: 0 objects, 0 B usage: 165 MiB used, 2.0 TiB / 2.0 TiB avail pgs: 33 active+clean
擴展mgr節點
root@ceph-mgr2:~# apt -y install ceph-mgr cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy mgr create ceph-mgr2
驗證mgr節點狀態
cephadmin@ceph-deploy:~/ceph-cluster$ ceph -s cluster: id: 0d11d338-a480-40da-8520-830423b22c3e health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 33m) mgr: ceph-mgr1(active, since 6h), standbys: ceph-mgr2 osd: 20 osds: 20 up (since 6h), 20 in (since 45h) data: pools: 2 pools, 33 pgs objects: 0 objects, 0 B usage: 165 MiB used, 2.0 TiB / 2.0 TiB avail pgs: 33 active+clean
四、塊設備RBD
RBD（RADOS Block Devices）即為塊存儲的一種，RBD通過librbd庫與OSD進行交互，RBD為KVM等虛擬化技術和雲服務（如OpenStack和CloudStack）提供高性能和無線可拓展性的存儲后端，這些系統依賴libvirt和QWMU使用程序與RBD進行集成，客戶端基於librbd庫即可將RADOS存儲集群用作塊設備，用於rbd的存儲池需要先啟用rbd功能並進行初始化。
-
創建RBD
$ ceph osd pool create <pool> [<pg_num:int>] [<pgp_num:int>] [replicated|erasure] #創建存儲池命令格式
cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool create myrbd1 64 64 #創建存儲池，指定pg和pgd的數量，pgp是對存在於pg的數據進行組合存儲，pgp通常等於pg的值 pool 'myrbd1' created cephadmin@ceph-deploy:~/ceph-cluster$ ceph osd pool application enable myrbd1 rbd #對存儲池啟用RBD功能 enabled application 'rbd' on pool 'myrbd1' cephadmin@ceph-deploy:~/ceph-cluster$ rbd pool init -p myrbd1 #通過RBD命令對存儲池初始化
-
創建並驗證img：
rbd存儲池並不能直接用於塊設備，而是需要事先在其中按需創建映像（image），並把映像文件作為塊設備使用，rbd命令可用於創建、查看及刪除塊設備所在的映像（image），以及克隆映像、創建快照、將映像回滾到快照和查看快照等管理操作。
cephadmin@ceph-deploy:~/ceph-cluster$ rbd create myimg1 --size 5G --pool myrbd1 cephadmin@ceph-deploy:~/ceph-cluster$ rbd create myimg2 --size 3G --pool myrbd1 --image-format 2 --image-feature layering #centos系統內核較低無法掛載使用，因此只開啟部分特性。 #除了layering 其他特性需要高版本內核支持 cephadmin@ceph-deploy:~/ceph-cluster$ rbd ls --pool myrbd1 #列出指定的pool中所有的img myimg1 myimg2 cephadmin@ceph-deploy:~/ceph-cluster$ rbd --image myimg1 --pool myrbd1 info #查看指定rbd的信息 rbd image 'myimg1': size 5 GiB in 1280 objects order 22 (4 MiB objects) snapshot_count: 0 id: 38ee810c4674 block_name_prefix: rbd_data.38ee810c4674 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Fri Aug 20 18:08:52 2021 access_timestamp: Fri Aug 20 18:08:52 2021 modify_timestamp: Fri Aug 20 18:08:52 2021 cephadmin@ceph-deploy:~/ceph-cluster$ rbd --image myimg2 --pool myrbd1 info rbd image 'myimg2': size 3 GiB in 768 objects order 22 (4 MiB objects) snapshot_count: 0 id: 38f7fea54b29 block_name_prefix: rbd_data.38f7fea54b29 format: 2 features: layering op_features: flags: create_timestamp: Fri Aug 20 18:09:51 2021 access_timestamp: Fri Aug 20 18:09:51 2021 modify_timestamp: Fri Aug 20 18:09:51 2021
-
客戶端使用塊存儲：
1、查看當前ceph狀態：
cephadmin@ceph-deploy:~/ceph-cluster$ ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 2.0 TiB 2.0 TiB 169 MiB 169 MiB 0 TOTAL 2.0 TiB 2.0 TiB 169 MiB 169 MiB 0 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 633 GiB mypool 2 32 0 B 0 0 B 0 633 GiB myrbd1 3 64 405 B 7 48 KiB 0 633 GiB
2、在客戶端安裝ceph-common：
[root@ceph-client1 ~]# yum install epel-release #配置yum源 [root@ceph-client1 ~]# yum -y install https://mirrors.aliyun.com/ceph/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm [root@ceph-client1 ~]# yum -y install ceph-common
#從部署服務器同步認證文件
cephadmin@ceph-deploy:~/ceph-cluster$ scp ceph.conf ceph.client.admin.keyring 10.0.0.71:/etc/ceph
3、客戶端映射img：
[root@ceph-client1 ~]# rbd -p myrbd1 map myimg2 /dev/rbd0 [root@ceph-client1 ~]# rbd -p myrbd1 map myimg1 rbd: sysfs write failed RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable myrbd1/myimg1 object-map fast-diff deep-flatten". In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (6) No such device or address [root@ceph-client1 ~]# rbd feature disable myrbd1/myimg1 object-map fast-diff deep-flatten [root@ceph-client1 ~]# rbd -p myrbd1 map myimg1 /dev/rbd1
4、客戶端驗證RBD：
[root@ceph-client1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 200G 0 disk ├─sda1 8:1 0 1G 0 part /boot ├─sda2 8:2 0 100G 0 part / ├─sda3 8:3 0 50G 0 part /data ├─sda4 8:4 0 1K 0 part └─sda5 8:5 0 4G 0 part [SWAP] sr0 11:0 1 1024M 0 rom rbd0 253:0 0 3G 0 disk rbd1 253:16 0 5G 0 disk
5、客戶端格式化磁盤並掛載使用：
[root@ceph-client1 ~]# mkfs.ext4 /dev/rbd0 mke2fs 1.42.9 (28-Dec-2013) Discarding device blocks: done Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=1024 blocks, Stripe width=1024 blocks 196608 inodes, 786432 blocks 39321 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=805306368 24 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done [root@ceph-client1 ~]# mkfs.xfs /dev/rbd1 Discarding blocks...Done. meta-data=/dev/rbd1 isize=512 agcount=8, agsize=163840 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=1310720, imaxpct=25 = sunit=1024 swidth=1024 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@ceph-client1 ~]# mount /dev/rbd0 /mnt/ [root@ceph-client1 ~]# df -TH Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 943M 0 943M 0% /dev tmpfs tmpfs 954M 0 954M 0% /dev/shm tmpfs tmpfs 954M 11M 944M 2% /run tmpfs tmpfs 954M 0 954M 0% /sys/fs/cgroup /dev/sda2 xfs 108G 5.2G 103G 5% / /dev/sda3 xfs 54G 175M 54G 1% /data /dev/sda1 xfs 1.1G 150M 915M 15% /boot tmpfs tmpfs 191M 0 191M 0% /run/user/0 /dev/rbd0 ext4 3.2G 9.5M 3.0G 1% /mnt [root@ceph-client1 ~]# mkdir /data [root@ceph-client1 ~]# mount /dev/rbd1 /data/ [root@ceph-client1 ~]# df -TH Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 943M 0 943M 0% /dev tmpfs tmpfs 954M 0 954M 0% /dev/shm tmpfs tmpfs 954M 10M 944M 2% /run tmpfs tmpfs 954M 0 954M 0% /sys/fs/cgroup /dev/sda2 xfs 108G 5.2G 103G 5% / /dev/rbd1 xfs 5.4G 35M 5.4G 1% /data /dev/sda1 xfs 1.1G 150M 915M 15% /boot tmpfs tmpfs 191M 0 191M 0% /run/user/0 /dev/rbd0 ext4 3.2G 9.5M 3.0G 1% /mnt
6、客戶端驗證：
[root@ceph-client1 data]# dd if=/dev/zero of=/data/ceph-test-file bs=1MB count=300 300+0 records in 300+0 records out 300000000 bytes (300 MB) copied, 1.61684 s, 186 MB/s [root@ceph-client1 data]# file /data/ceph-test-file /data/ceph-test-file: data [root@ceph-client1 data]# ll -h /data/ceph-test-file -rw-r--r--. 1 root root 287M Aug 23 12:27 /data/ceph-test-file
7、ceph驗證數據：
cephadmin@cepn-node1:~$ ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 2.0 TiB 2.0 TiB 2.3 GiB 2.3 GiB 0.11 TOTAL 2.0 TiB 2.0 TiB 2.3 GiB 2.3 GiB 0.11 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 632 GiB myrbd1 2 64 363 MiB 115 1.1 GiB 0.06 632 GiB