Ceph集群
集群組件
Ceph集群包括Ceph OSD,Ceph Monitor兩種守護進程。
Ceph OSD(Object Storage Device): 功能是存儲數據,處理數據的復制、恢復、回填、再均衡,並通過檢查其他OSD守護進程的心跳來向Ceph Monitors提供一些監控信息。
Ceph Monitor: 是一個監視器,監視Ceph集群狀態和維護集群中的各種關系。
Ceph存儲集群至少需要一個Ceph Monitor和兩個 OSD 守護進程。
集群環境准備
集群圖
准備工作
准備四台服務器，需要可以上外網，其中三台作為ceph集群 一台作為cehp客戶端，除了client外每台加一個磁盤（/dev/sdb），無需分區
環境查看
配置主機名及hosts（四台主機主機名不一樣hosts一樣）
# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.101 node1 192.168.1.102 node2 192.168.1.103 node3 192.168.1.104 client
關閉防火牆及selinux
設置ntpdate時間同步並設置在crontab內
ntpdate time1.aliyun.com
配置ceph的yum源
yum install epel-release -y
四台主機均設置，如果需要設置其他版本的yum源可以打開頁面選擇
http://mirrors.aliyun.com/ceph
# cat /etc/yum.repos.d/ceph.repo [ceph] name=ceph baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/x86_64/ enabled=1 gpgcheck=0 priority=1 [ceph-noarch] name=cephnoarch baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/noarch/ enabled=1 gpgcheck=0 priority=1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/SRPMS enabled=1 gpgcheck=0 priority=1
更新yum源
yum update
注意：需要更新否則后面會報錯
配置ssh免密登錄
以node1位部署節點在node1配置使node1可以免密登錄四台主機
ssh-keygen -t rsa ssh-copy-id node1 ssh-copy-id node2 ssh-copy-id node3 ssh-copy-id client
在node1上ssh其他主機無需輸入密碼代表配置ssh成功
在node1上安裝部署工具（其他節點不用安裝）
yum install ceph-deploy -y
建立一個集群配置目錄
注意：后面大部分操作在此目錄進行
mkdir /etc/ceph cd /etc/ceph
創建集群
[root@node1 ~]# ceph-deploy new node1
過程如下
[root@localhost ceph]# ceph-deploy new node1 [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.30): /usr/bin/ceph-deploy new node1 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] func : <function new at 0x7f8fe7885ed8> [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f8fe789ce18> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] ssh_copykey : True [ceph_deploy.cli][INFO ] mon : ['node1'] [ceph_deploy.cli][INFO ] public_network : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] cluster_network : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] fsid : None [ceph_deploy.new][DEBUG ] Creating new cluster named ceph [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [node1][DEBUG ] connected to host: node1 [node1][DEBUG ] detect platform information from remote host [node1][DEBUG ] detect machine type [node1][DEBUG ] find the location of an executable [node1][INFO ] Running command: /usr/sbin/ip link show [node1][INFO ] Running command: /usr/sbin/ip addr show [node1][DEBUG ] IP addresses found: ['192.168.1.101'] [ceph_deploy.new][DEBUG ] Resolving host node1 [ceph_deploy.new][DEBUG ] Monitor node1 at 192.168.1.101 [ceph_deploy.new][DEBUG ] Monitor initial members are ['node1'] [ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.1.101'] [ceph_deploy.new][DEBUG ] Creating a random mon key... [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring... [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
會在當前配置文件目錄生成以下配置文件
#ls ceph.conf ceph-deploy-ceph.log ceph.mon.keyring
說明
說明: ceph.conf 集群配置文件 ceph-deploy-ceph.log 使用ceph-deploy部署的日志記錄 ceph.mon.keyring mon的驗證key文件 監控需要的令牌
集群節點安裝ceph
node1 node2 node3 client都需要安裝
yum install ceph ceph-radosgw -y
查看版本
# ceph -v ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable)
注意： 如果公網OK,並且網速好的話,可以用`ceph-deploy install node1 node2 node3`命令來安裝,但網速不好的話會比較坑
所以這里我們選擇直接用准備好的本地ceph源,然后`yum install ceph ceph-radosgw -y`安裝即可。
客戶端client安裝ceph-common
[root@client ~]# yum install ceph-common -y
好像上一步已經在client安裝了cehp-common
創建mon監控
解決public網絡用於監控
修改配置文件在[global]配置端添加下面一句
public network = 192.168.1.0/24
監控節點初始化,並同步配置到所有節點(node1,node2,node3,不包括client）
ceph-deploy mon create-initial
如果執行報錯
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.mon][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors
則在修改配置public network = 192.168.1.0/24之前執行一次ceph-deploy mon create-initial修改配置之后再次執行即可
查看監控狀態
[root@node1 ceph]# ceph health HEALTH_OK
將配置信息同步到所有節點
ceph-deploy admin node1 node2 node3
在node2 node3多了幾個配置文件
[root@node2 ceph]# ll 總用量 12 -rw------- 1 root root 151 8月 3 14:46 ceph.client.admin.keyring -rw-r--r-- 1 root root 322 8月 3 14:46 ceph.conf -rw-r--r-- 1 root root 92 4月 24 01:07 rbdmap -rw------- 1 root root 0 8月 1 16:46 tmpDX_w1Z
查看狀態
[root@localhost ceph]# ceph -s
cluster:
id: a9aa277d-7192-4687-b384-9787b73ece71
health: HEALTH_OK #集群監控狀態OK
services:
mon: 1 daemons, quorum node1 (age 3m) #1個mon
mgr: no daemons active
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
為了防止mon單點故障，你可以加多個mon節點(建議奇數個，因為有quorum仲裁投票）
ceph-deploy mon add node2 ceph-deploy mon add node3
再次查看
[root@localhost ceph]# ceph -s
cluster:
id: a9aa277d-7192-4687-b384-9787b73ece71
health: HEALTH_OK
services:
mon: 3 daemons, quorum node1,node2,node3 (age 0.188542s) #3個mon了
mgr: no daemons active
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
#### 監控到時間不同步的解決方法
ceph集群對時間同步要求非常高, 即使你已經將ntpd服務開啟,但仍然可能有`clock skew deteted`相關警告**
請做如下嘗試:
1, 在ceph集群所有節點上(`node1`,`node2`,`node3`)不使用ntpd服務,直接使用crontab同步
# systemctl stop ntpd
# systemctl disable ntpd
# crontab -e
*/10 * * * * ntpdate ntp1.aliyun.com 每5或10分鍾同步1次公網的任意時間服務器
2, 調大時間警告的閾值
[root@node1 ceph]# vim ceph.conf
[global] 在global參數組里添加以下兩行
mon clock drift allowed = 2 # monitor間的時鍾滴答數(默認0.5秒)
mon clock drift warn backoff = 30 # 調大時鍾允許的偏移量(默認為5)
修改了ceph.conf配置需要同步到所有節點
ceph-deploy --overwrite-conf admin node1 node2 node3
前面第1次同步不需要加--overwrite-conf參數
這次修改ceph.conf再同步就需要加--overwrite-conf參數覆蓋
所有ceph集群節點上重啟ceph-mon.target服務
systemctl restart ceph-mon.target
創建mgr（管理）
eph luminous版本中新增加了一個組件：Ceph Manager Daemon，簡稱ceph-mgr。
該組件的主要作用是分擔和擴展monitor的部分功能，減輕monitor的負擔，讓更好地管理ceph存儲系統。
ceph-deploy mgr create node1
查看
[root@localhost ceph]# ceph -s
cluster:
id: a9aa277d-7192-4687-b384-9787b73ece71
health: HEALTH_WARN
Module 'restful' has failed dependency: No module named 'pecan'
Reduced data availability: 1 pg inactive
OSD count 0 < osd_pool_default_size 3
services:
mon: 3 daemons, quorum node1,node2,node3 (age 10m)
mgr: node1(active, since 2m) #創建了一個mgr節點是node
osd: 0 osds: 0 up, 0 in
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs: 100.000% pgs unknown
1 unknown
添加多個mgr可以實現HA
ceph-deploy mgr create node2 ceph-deploy mgr create node3
查看
[root@localhost ceph]# ceph -s
cluster:
id: a9aa277d-7192-4687-b384-9787b73ece71
health: HEALTH_WARN
Module 'restful' has failed dependency: No module named 'pecan'
Reduced data availability: 1 pg inactive
OSD count 0 < osd_pool_default_size 3
services:
mon: 3 daemons, quorum node1,node2,node3 (age 11m)
mgr: node1(active, since 3m), standbys: node2, node3 #3個mgr其中node1為active node2 3為standbys
osd: 0 osds: 0 up, 0 in
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs: 100.000% pgs unknown
1 unknown
創建OSD（存儲盤）
查看幫助
[root@node1 ceph]# ceph-deploy disk --help [root@node1 ceph]# ceph-deploy osd --help
zap表示干掉磁盤上的數據,相當於格式化
[root@node1 ceph]# ceph-deploy disk zap node1 /dev/sdb [root@node1 ceph]# ceph-deploy disk zap node2 /dev/sdb [root@node1 ceph]# ceph-deploy disk zap node3 /dev/sdb
將磁盤創建為osd
[root@node1 ceph]# ceph-deploy osd create --data /dev/sdb node1 [root@node1 ceph]# ceph-deploy osd create --data /dev/sdb node2 [root@node1 ceph]# ceph-deploy osd create --data /dev/sdb node3
[root@localhost ceph]# ceph-deploy osd create --data /dev/sdb node1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy osd create --data /dev/sdb node1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] bluestore : None
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f718ae904d0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] fs_type : xfs
[ceph_deploy.cli][INFO ] block_wal : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] journal : None
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] host : node1
[ceph_deploy.cli][INFO ] filestore : None
[ceph_deploy.cli][INFO ] func : <function osd at 0x7f718b7115f0>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] zap_disk : False
[ceph_deploy.cli][INFO ] data : /dev/sdb
[ceph_deploy.cli][INFO ] block_db : None
[ceph_deploy.cli][INFO ] dmcrypt : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data device /dev/sdb
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.5.1804 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to node1
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node1][WARNIN] osd keyring does not exist yet, creating one
[node1][DEBUG ] create a keyring file
[node1][DEBUG ] find the location of an executable
[node1][INFO ] Running command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb
[node1][WARNIN] Running command: /usr/bin/ceph-authtool --gen-print-key
[node1][WARNIN] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d39eddff-d29a-426a-ae2b-4c5a4026458d
[node1][WARNIN] Running command: /usr/sbin/vgcreate --force --yes ceph-a1598e52-3eb9-45c4-8416-9c5537241d70 /dev/sdb
[node1][WARNIN] stdout: Physical volume "/dev/sdb" successfully created.
[node1][WARNIN] stdout: Volume group "ceph-a1598e52-3eb9-45c4-8416-9c5537241d70" successfully created
[node1][WARNIN] Running command: /usr/sbin/lvcreate --yes -l 100%FREE -n osd-block-d39eddff-d29a-426a-ae2b-4c5a4026458d ceph-a1598e52-3eb9-45c4-8416-9c5537241d70
[node1][WARNIN] stdout: Logical volume "osd-block-d39eddff-d29a-426a-ae2b-4c5a4026458d" created.
[node1][WARNIN] Running command: /usr/bin/ceph-authtool --gen-print-key
[node1][WARNIN] Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
[node1][WARNIN] Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-a1598e52-3eb9-45c4-8416-9c5537241d70/osd-block-d39eddff-d29a-426a-ae2b-4c5a4026458d
[node1][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
[node1][WARNIN] Running command: /usr/bin/ln -s /dev/ceph-a1598e52-3eb9-45c4-8416-9c5537241d70/osd-block-d39eddff-d29a-426a-ae2b-4c5a4026458d /var/lib/ceph/osd/ceph-0/block
[node1][WARNIN] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
[node1][WARNIN] stderr: 2020-08-04T17:39:52.619+0800 7fa772483700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
[node1][WARNIN] 2020-08-04T17:39:52.619+0800 7fa772483700 -1 AuthRegistry(0x7fa76c058528) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
[node1][WARNIN] stderr: got monmap epoch 3
[node1][WARNIN] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQDnLClfQ6UHKhAAkJ585N2DYxYt5negauu0Mg==
[node1][WARNIN] stdout: creating /var/lib/ceph/osd/ceph-0/keyring
[node1][WARNIN] added entity osd.0 auth(key=AQDnLClfQ6UHKhAAkJ585N2DYxYt5negauu0Mg==)
[node1][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
[node1][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
[node1][WARNIN] Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid d39eddff-d29a-426a-ae2b-4c5a4026458d --setuser ceph --setgroup ceph
[node1][WARNIN] --> ceph-volume lvm prepare successful for: /dev/sdb
[node1][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
[node1][WARNIN] Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-a1598e52-3eb9-45c4-8416-9c5537241d70/osd-block-d39eddff-d29a-426a-ae2b-4c5a4026458d --path /var/lib/ceph/osd/ceph-0 --no-mon-config
[node1][WARNIN] Running command: /usr/bin/ln -snf /dev/ceph-a1598e52-3eb9-45c4-8416-9c5537241d70/osd-block-d39eddff-d29a-426a-ae2b-4c5a4026458d /var/lib/ceph/osd/ceph-0/block
[node1][WARNIN] Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
[node1][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
[node1][WARNIN] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
[node1][WARNIN] Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-d39eddff-d29a-426a-ae2b-4c5a4026458d
[node1][WARNIN] stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-d39eddff-d29a-426a-ae2b-4c5a4026458d.service to /usr/lib/systemd/system/ceph-volume@.service.
[node1][WARNIN] Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
[node1][WARNIN] stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service to /usr/lib/systemd/system/ceph-osd@.service.
[node1][WARNIN] Running command: /usr/bin/systemctl start ceph-osd@0
[node1][WARNIN] --> ceph-volume lvm activate successful for osd ID: 0
[node1][WARNIN] --> ceph-volume lvm create successful for: /dev/sdb
[node1][INFO ] checking OSD status...
[node1][DEBUG ] find the location of an executable
[node1][INFO ] Running command: /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node1 is now ready for osd use.
查看
[root@localhost ceph]# ceph -s
cluster:
id: a9aa277d-7192-4687-b384-9787b73ece71
health: HEALTH_WARN
Module 'restful' has failed dependency: No module named 'pecan'
services:
mon: 3 daemons, quorum node1,node2,node3 (age 22m)
mgr: node1(active, since 14m), standbys: node2, node3
osd: 3 osds: 3 up (since 4s), 3 in (since 4s) #3個osd 3個up啟用狀態 3 in代表3G在使用 每塊硬盤占用1G
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 45 GiB / 48 GiB avail #3G使用 45G空閑 總計48G及三塊16G硬盤相加之和
pgs: 1 active+clean
集群節點的擴容方法
假設再加一個新的集群節點node4
1, 主機名配置和綁定
2, 在node4上`yum install ceph ceph-radosgw -y`安裝軟件
3, 在部署節點node1上同步配置文件給node4. `ceph-deploy admin node4`
4, 按需求選擇在node4上添加mon或mgr或osd等
ceph dashboard
查看集群狀態確認mgr的active節點
[root@node1 ceph]# ceph -s
cluster:
id: 134c6247-2fd4-4669-8c19-d0c9200d90c8
health: HEALTH_OK
services:
mon: 3 daemons, quorum node1,node2,node3
mgr: node1(active), standbys: node2, node3 #確認mgr active節點為node1
osd: 3 osds: 3 up, 3 in
rgw: 1 daemon active
data:
pools: 4 pools, 32 pgs
objects: 189 objects, 1.5 KiB
usage: 3.0 GiB used, 45 GiB / 48 GiB avail
pgs: 32 active+clean
查看開啟及關閉的模塊
[root@node1 mgr-dashboard]# ceph mgr module ls
{
"enabled_modules": [
"balancer",
"crash",
"dashboard",
"iostat",
"restful",
"status"
],
"disabled_modules": [
{
"name": "hello",
"can_run": true,
"error_string": ""
},
{
"name": "influx",
"can_run": false,
"error_string": "influxdb python module not found"
},
{
"name": "localpool",
"can_run": true,
"error_string": ""
},
{
"name": "prometheus",
"can_run": true,
"error_string": ""
},
{
"name": "selftest",
"can_run": true,
"error_string": ""
},
{
"name": "smart",
"can_run": true,
"error_string": ""
},
{
"name": "telegraf",
"can_run": true,
"error_string": ""
},
{
"name": "telemetry",
"can_run": true,
"error_string": ""
},
{
"name": "zabbix",
"can_run": true,
"error_string": ""
}
]
}
開啟dashboard模塊
ceph mgr module enable dashboard
如果開啟報錯
[root@localhost ceph]# ceph mgr module enable dashboard Error ENOENT: all mgr daemons do not support module 'dashboard', pass --force to force enablement
需要在每個開啟mgr的節點安裝ceph-mgr-dashboard
yum install ceph-mgr-dashboard -y
注意：不能僅僅在active節點安裝，需要在standby節點都安裝
創建自簽名證書
[root@node1 ceph]# ceph dashboard create-self-signed-cert Self-signed certificate created
生成密鑰對，並配置給ceph mgr
[root@node1 ceph]# mkdir /etc/mgr-dashboard [root@node1 ceph]# cd /etc/mgr-dashboard/
openssl req -new -nodes -x509 -subj "/O=IT-ceph/CN=cn" -days 3650 -keyout dashboard.key -out dashboard.crt -extensions v3_ca
[root@node1 mgr-dashboard]# ls dashboard.crt dashboard.key
在ceph集群的active mgr節點上(我這里為node1)配置mgr services
[root@node1 mgr-dashboard]# ceph config set mgr mgr/dashboard/server_addr 192.168.1.101 [root@node1 mgr-dashboard]# ceph config set mgr mgr/dashboard/server_port 8080
重啟dashboard模塊,並查看訪問地址
注意：需要重啟不重啟查看的端口是默認的8443端口無法訪問
重啟就是先關disable再開啟enable
[root@node1 mgr-dashboard]# ceph mgr module disable dashboard [root@node1 mgr-dashboard]# ceph mgr module enable dashboard
查看mgr service
# ceph mgr services
{
"dashboard": "https://192.168.1.101:8080/"
}
設置訪問web頁面用戶名和密碼
ceph dashboard set-login-credentials admin admin
通過本機或其它主機訪問 https://ip:8080
Ceph文件存儲
要運行Ceph文件系統，你必須先裝只是帶一個mds的Ceph存儲集群
Ceph.MDS：為Ceph文件存儲類型存放元數據metadata（也就是說Ceph塊存儲和Ceph對象存儲不使用MDS）
創建文件存儲並使用
第1步 在node1部署節點上修改配置/etc/ceph/ceph.conf 增加配置
mon_allow_pool_delete = true
同步配置文件
注意：修改了配置文件才需要同步，沒有修改不需要同步配置文件
ceph-deploy --overwrite-conf admin node1 node2 node3
創建3個mds
[root@node1 ceph]# ceph-deploy mds create node1 node2 node3
第2步: 一個Ceph文件系統需要至少兩個RADOS存儲池，一個用於數據，一個用於元數據。所以我們創建它們
[root@node1 ceph]# ceph osd pool create cephfs_pool 128 pool 'cephfs_pool' created [root@node1 ceph]# ceph osd pool create cephfs_metadata 64 pool 'cephfs_metadata' created
參數解釋
創建pool自定義名為cephfs_pool 用於存儲數據 PG數為128 用於存儲數據所以PG數較大 創建pool自定義名為cephfs_metadata 用於存儲元數據 PG數為64
PG介紹
PG, Placement Groups。CRUSH先將數據分解成一組對象，然后根據對象名稱、復制級別和系統中的PG數等信息執行散列操作，再將結果生成PG ID。可以將PG看做一個邏輯容器，這個容器包含多個對象，同時這個邏輯對象映射之多個OSD上。
如果沒有PG，在成千上萬個OSD上管理和跟蹤數百萬計的對象的復制和傳播是相當困難的。沒有PG這一層，管理海量的對象所消耗的計算資源也是不可想象的。建議每個OSD上配置50~100個PG。
如果定義PG數
一般的
少於5個OSD則PG數為128
5-10個OSD則PG數為512
10-50個OSD則PG數為1024
如果有更多的OSD需要自己理解計算
查看
[root@node1 ceph]# ceph osd pool ls |grep cephfs cephfs_pool cephfs_metadata
查看創建的pool詳細信息
[root@localhost ceph]# ceph osd pool get cephfs_pool all size: 3 min_size: 2 pg_num: 128 pgp_num: 128 crush_rule: replicated_rule hashpspool: true nodelete: false nopgchange: false nosizechange: false write_fadvise_dontneed: false noscrub: false nodeep-scrub: false use_gmt_hitset: 1 fast_read: 0 pg_autoscale_mode: warn
第3步: 創建Ceph文件系統,並確認客戶端訪問的節點
[root@node1 ceph]# ceph fs new cephfs cephfs_metadata cephfs_pool new fs with metadata pool 10 and data pool 9
查看
[root@node1 ceph]# ceph osd pool ls .rgw.root default.rgw.control default.rgw.meta default.rgw.log cephfs_pool cephfs_metadata
[root@node1 ceph]# ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_pool ]
[root@node1 ceph]# ceph mds stat
cephfs-1/1/1 up {0=node3=up:active}, 2 up:standby #這里看到node3為up狀態
metadata保存在node3上
客戶端准備驗證key文件
說明: ceph默認啟用了cephx認證, 所以客戶端的掛載必須要驗證（ceph.conf默認配置文件開啟）
在集群節點(node1,node2,node3)上任意一台查看密鑰字符串
cat /etc/ceph/ceph.client.admin.keyring
輸出
ceph-authtool -p /etc/ceph/ceph.client.admin.keyring >admin.key
把這個文件放在客戶端client /root/admin.key
注意：直接把key復制編輯admin.key文檔可能會在掛載時報錯
客戶端client安裝
yum -y install ceph-fuse
需要安裝否則客戶端不支持
客戶端掛載
可以使用其他node主機名進行掛載，例如node3
mount -t ceph node1:6789:/ /mnt -o name=admin,secretfile=/root/admin.key
查看
注意：如果使用文件掛載報錯可以使用參數secret=秘鑰進行掛載
可以使用兩個客戶端, 同時掛載此文件存儲,可實現同讀同寫
往掛載的硬盤寫數據可以在dashboard查看讀寫監控狀態
dd if=/dev/zero of=/mnt/file1 bs=10M count=1000
刪除文件存儲
在所有掛載了文件存儲的客戶端卸載文件掛載
umount /mnt
停掉所有節點的mds node1 node2 node3
systemctl stop ceph-mds.target
回到集群任意一個節點上(node1,node2,node3其中之一)刪除
如果要客戶端刪除,需要在node1上`ceph-deploy admin client`同步配置才可以
[root@node3 ~]# ceph fs rm cephfs --yes-i-really-mean-it [root@node3 ~]# ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it pool 'cephfs_metadata' removed [root@node3 ~]# ceph osd pool delete cephfs_pool cephfs_pool --yes-i-really-really-mean-it pool 'cephfs_pool' removed
注意：為了安全需要輸入兩次創建的pool名並且加參數--yes-i-really-really-mean-it才能刪除
注意：需要在配置文件添加以下配置，才能刪除
mon_allow_pool_delete = true
如果已經添加配置還提示
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool
則重啟服務ceph-mon.target即可
啟動md是服務 node1 node2 node3節點啟動
systemctl start ceph-mds.target
Ceph快存儲
第一步：在node1上同步配置文件到client
[root@node1 ceph]# cd /etc/ceph [root@node1 ceph]# ceph-deploy admin client
第二步：建立存儲池，並初始化
在客戶端client操作
[root@client ~]# ceph osd pool create rbd_pool 128 pool 'rbd_pool' created
初始化
[root@client ~]# rbd pool init rbd_pool
第三步：創建一個存儲卷（這里卷名為volume1 大小為5000M）
rbd create volume1 --pool rbd_pool --size 5000
查看
[root@client ~]# rbd ls rbd_pool volume1
[root@client ~]# rbd info volume1 -p rbd_pool rbd image 'volume1': #volume1為rbd_image size 4.9 GiB in 1250 objects order 22 (4 MiB objects) id: 60fc6b8b4567 block_name_prefix: rbd_data.60fc6b8b4567 format: 2 #兩種格式1和2 默認是2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Tue Aug 4 09:20:05 2020
第四步：將創建的卷映射成塊設備
因為rbd鏡像的一些特性，OS kernel並不支持，所以映射報錯
[root@client ~]# rbd map rbd_pool/volume1 rbd: sysfs write failed RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable rbd_pool/volume1 object-map fast-diff deep-flatten". In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (6) No such device or address
解決辦法：disable掉相關特性
rbd feature disable rbd_pool/volume1 exclusive-lock object-map fast-diff deep-flatten
再次映射
[root@client ~]# rbd map rbd_pool/volume1 /dev/rbd0
創建了磁盤/dev/rbd0 類似於做了一個軟連接
查看映射
[root@client ~]# rbd showmapped id pool image snap device 0 rbd_pool volume1 - /dev/rbd0
如果需要取消映射可以使用命令
rbd unmap /dev/rbd0
第六步：格式化掛載
mkfs.xfs /dev/rbd0 mount /dev/rbd0 /mnt
查看
[root@client ~]# df -h|tail -1 /dev/rbd0 4.9G 33M 4.9G 1% /mnt
塊擴容與裁減
擴容成8000M
[root@client ~]# rbd resize --size 8000 rbd_pool/volume1 Resizing image: 100% complete...done.
查看並沒有變化
[root@client ~]# df -h|tail -1 /dev/rbd0 4.9G 33M 4.9G 1% /mnt
動態刷新擴容
[root@client ~]# xfs_growfs -d /mnt/
meta-data=/dev/rbd0 isize=512 agcount=8, agsize=160768 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=1280000, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 1280000 to 2048000
注意：該命令和LVM擴容命令一致
再次查看，擴容成功
[root@client ~]# df -h|tail -1 /dev/rbd0 7.9G 33M 7.8G 1% /mnt
塊存儲裁減
不能在線裁減，裁減后需要重新格式化再掛載，如果有數據需要提前備份好數據
裁減為5000M
[root@client ~]# rbd resize --size 5000 rbd_pool/volume1 --allow-shrink Resizing image: 100% complete...done.
卸載，格式化
umount /mnt fdisk -l mkfs.xfs -f /dev/rbd0 mount /dev/rbd0 /mnt/
查看
[root@client ~]# df -h|tail -1 /dev/rbd0 4.9G 33M 4.9G 1% /mnt
刪除塊存儲
#卸載 [root@client ~]# umount /mnt #取消映射 [root@client ~]# rbd unmap /dev/rbd0 #刪除存儲池 [root@client ~]# ceph osd pool delete rbd_pool rbd_pool --yes-i-really-really-mean-it pool 'rbd_pool' removed
Ceph對象存儲
第一步：在node1上創建rgw
[root@node1 ceph]# ceph-deploy rgw create node1
查看,運行端口是7480
lsof -i:7480
第二步：在客戶端測試連接對象網關
在client操作
安裝測試工具
創建一個測試用戶,需要在部署節點使用ceph-deploy admin client同步配置文件給client
[root@client ~]# radosgw-admin user create --uid="testuser" --display-name="First User"|grep -E 'access_key|secret_key'
"access_key": "S859J38AS6WW1CZSB90M",
"secret_key": "PCmfHoAsHw2GIEioSWvN887o02VXesOkX2gJ20fG"
[root@client ~]# radosgw-admin user create --uid="testuser" --display-name="First User"
{
"user_id": "testuser",
"display_name": "First User",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "testuser",
"access_key": "S859J38AS6WW1CZSB90M",
"secret_key": "PCmfHoAsHw2GIEioSWvN887o02VXesOkX2gJ20fG"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
上面一大段主要有用的為access_key與secret_key，用於連接對象存儲網關
[root@client ~]# radosgw-admin user create --uid="testuser" --display-name="First User"|grep -E 'access_key|secret_key'
"access_key": "S859J38AS6WW1CZSB90M",
"secret_key": "PCmfHoAsHw2GIEioSWvN887o02VXesOkX2gJ20fG"
s3連接對象網關
第一步：客戶端安裝s3cmd工具，並編寫配置文件
yum install s3cmd
創建配置文件，內容如下
[root@client ~]# cat /root/.s3cfg [default] access_key = S859J38AS6WW1CZSB90M secret_key = PCmfHoAsHw2GIEioSWvN887o02VXesOkX2gJ20fG host_base = 192.168.1.101:7480 host_bucket = 192.168.1.101:7480/%(bucket) cloudfront_host = 192.168.1.101:7480 use_https = False
列出bucket
s3cmd ls
創建一個桶
s3cmd mb s3://test_bucket
上傳文件到桶
[root@client ~]# s3cmd put /etc/fstab s3://test_bucket
下載文件到當前目錄
[root@client ~]# s3cmd get s3://test_bucket/fstab