ceph手動安裝

本文轉載自查看原文 2020-05-02 19:53 1035 ceph

本文主要講述在無外網條件下安裝ceph存儲集群的過程。具體的安裝環境如下：

[root@ceph001-node1 /]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.1.1503 (Core) 
Release:        7.1.1503
Codename:       Core

[root@ceph001-node1 /]# uname -a
Linux ceph001-node1 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 
UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

這里采用了3台虛擬機：

主機IP	部署組件	主機名
10.133.134.211	node1	ceph001-node1
10.133.134.212	node2	ceph001-node2
10.133.134.213	node3	ceph001-node3

1. 下載軟件安裝包

因為我們是在無外網環境下安裝ceph，因此我們需要在另外一台能夠聯網的機器上下載到對應的軟件安裝包。

注意：這里我們的下載軟件包的主機環境最好與實際的安裝環境一致，以免后面有些軟件依賴出現問題

1.1 ADD KEYS

添加key到你的系統受信任keys列表來避免一些安全上的警告信息，對於major releases(例如hammer，jewel)請使用release.asc。

我們先從https://download.ceph.com/keys/release.asc 下載對應的release.asc文件，上傳到集群的每一個節點上，執行如下命令：

sudo rpm --import './release.asc'

1.2 DOWNLOAD PACKAGES

假如你是需要在無網絡訪問的防火牆后安裝ceph集群，在安裝之前你必須要獲得相應的安裝包。

注意，你的連接外網的下載ceph安裝包的機器環境與你實際安裝ceph集群的環境最好一致，否則可能出現安裝包版本不一致的情況而出現錯誤。。

RPM PACKAGES

先新建三個文件夾dependencies、ceph、ceph-deploy分別存放下面下載的安裝包。

1) Ceph需要一些額外的第三方庫。添加EPEL repository，執行如下命令：

sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Ceph需要如下一些依賴包：

snappy
leveldb
gdisk
python-argparse
gperftools-libs

現在一台可以連接外網的主機上下載這些依賴包，存放在dependencies文件夾：

sudo yumdownloader snappy
sudo yumdownloader leveldb
sudo yumdownloader gdisk
sudo yumdownloader python-argparse
sudo yumdownloader gperftools-libs

2) 安裝yum-plugin-priorities

sudo yum install yum-plugin-priorities

修改/etc/yum/pluginconf.d/priorities.conf文件：

[main]
enabled = 1

3) 通過如下的命令下載適當版本的ceph安裝包

su -c 'rpm -Uvh https://download.ceph.com/rpm-{release-name}/{distro}/noarch/ceph-{version}.{distro}.noarch.rpm'

也可以直接到官方對應的網站去下載：https://download.ceph.com/

這里我們在CentOS7.1上配置如下：

su -c 'rpm -Uvh https://download.ceph.com/rpm-jewel/el7/noarch/ceph-release-1-0.el7.noarch.rpm'

修改/etc/yum.repos.d/ceph.repo文件，添加priority項：

下載ceph安裝包：

yum clean packages #先清除本地可能緩存的一些包 yum clean yum repolist yum makecache sudo yum install --downloadonly --downloaddir=/ceph-cluster/packages/ceph ceph ceph-radosgw

NOTE1：這里我們目前就下載ceph,ceph-radosgw兩個包，其依賴的一些包會自動下載下來。如果在實際安裝中，仍缺少一些依賴包，我們可以通過yum search ${package-name} 查找到該包，然后再下載下來.

NOTE2: 上面這是下載最新版本的ceph安裝包，如果下載其他版本，請攜帶上版本號

4）下載ceph-deploy安裝包

這里我們是手動安裝，可以不用下載。

sudo yum install --downloadonly --downloaddir=/ceph-cluster/packages/ceph-deploy ceph-deploy

5) 將上述的依賴包分別打包壓縮稱dependencies.tar.gz、ceph.tar.gz、ceph-deploy.tar.gz，並上傳到集群的各個節點上來進行安裝

2. 安裝軟件包

1）建立相應的構建目錄

這里我們統一采用如下目錄來完成整個集群的安裝：

mkdir -p /ceph-cluster/build/script mkdir -p /ceph-cluster/packages mkdir -p /ceph-cluster/test chmod 777 /ceph-cluster -R

2）關閉iptables及SELinux

可以將如下shell命令寫成腳本來執行(disable-iptable-selinux.sh)：

systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

3）修改主機名

在上述所有節點上分別修改/etc/sysconfig/network文件，指定其主機名分別為ceph001-admin、ceph001-node1、ceph001-node2、ceph001-node3。例如：

[root@ceph001-node1 ~]# cat /etc/sysconfig/network
# Created by anaconda
NOZEROCONF=yes
HOSTNAME=ceph001-node1

再調用如下命令修改：

hostnamectl --static --transient --pretty set-hostname {host-name}

例如修改node1節點：

hostnamectl --static --transient --pretty set-hostname ceph001-node1

上述修改需要在系統下次重啟時才生效。

此外，我們需要分別在每一個節點上執行hostname命令來立即更改主機名稱。例如：

[root@ceph001-node1 ~]# sudo hostname ceph001-node1
[root@ceph001-node1 ~]# hostname -s
ceph001-node1

4) 修改/etc/hosts文件

分別修改3台宿主機節點上的/etc/hosts文件

# For Ceph Cluster
10.133.134.211	ceph001-node1
10.133.134.212	ceph001-node2
10.133.134.213	ceph001-node3

5) 通過主機名測試各節點之間是否聯通

分別測試各個主機節點是否通過主機名ping通。例如：

[root@ceph001-node1 ~]# ping ceph001-node2
PING ceph001-node2 (10.133.134.212) 56(84) bytes of data.
64 bytes from ceph001-node2 (10.133.134.212): icmp_seq=1 ttl=64 time=0.869 ms
64 bytes from ceph001-node2 (10.133.134.212): icmp_seq=2 ttl=64 time=0.524 ms
64 bytes from ceph001-node2 (10.133.134.212): icmp_seq=3 ttl=64 time=0.363 ms
64 bytes from ceph001-node2 (10.133.134.212): icmp_seq=4 ttl=64 time=0.416 ms

6) 檢查當前CentOS內核版本是否支持rbd，並裝載rbd模塊

modinfo rbd modprobe rbd #裝載rbd模塊 lsmod | grep rbd #查看模塊是否已經裝載

7) 安裝ntp，並配置ceph集群節點之間的時間同步

在各節點執行如下命令：

rpm –qa | grep ntp #查看當前是否已經安裝ntp ps –ef | grep ntp # 查看ntp服務器是否啟動 ntpstat #查看當前的同步狀態

在/etc/ntp.conf配置文件中配置時間同步服務器地址

參看：http://www.centoscn.com/CentosServer/test/2016/0129/6709.html

8) TTY

在CentOS及RHEL上，當你嘗試執行ceph-deploy時，你也許會收到一個錯誤。假如requiretty在你的Ceph節點上默認被設置了的話，可以執行sudo visudo然后定位到Defaults requiretty的設置部分，將其改變為Defaults:ceph !requiretty或者直接將其注釋掉

NOTE:假如直接修改/etc/sudoers，確保使用sudo visudo，而不要用其他的文本編輯器

9) 安裝ceph軟件包

安裝pre-requisite 包

在所有節點上安裝如下包:

snappy
leveldb
gdisk
python-argparse
gperftools-libs

執行如下命令進行安裝：

sudo yum localinstall *.rpm

安裝ceph包

在所有節點上執行如下命令安裝ceph包：

sudo yum localinstall *.rpm

3. 建立集群

3.1 建立monitor

我們會在ceph001-node1，ceph001-node2,ceph001-node3上分別部署monitor.請在/ceph-cluster/build目錄下完成構建。

在ceph001-node1上建立monitor

1) 生成monitor keyring

ceph-authtool --create-keyring ./ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'

查看生成的monitor keyring:

[root@ceph001-node1 build]# cat ./ceph.mon.keyring 
[mon.]
        key = AQCG0m5Z9AifFxAAreLxG7PXYPXRNyNlRzrGhQ==
        caps mon = "allow *"

2) 生成client.admin keyring

ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'

查看生成的client.admin keyring：

[root@ceph001-node1 build]# cat /etc/ceph/ceph.client.admin.keyring 
[client.admin]
        key = AQC61W5ZQlCOJRAAkXEE7xZtNiZwudgVqRtvuQ==
        auid = 0
        caps mds = "allow"
        caps mon = "allow *"
        caps osd = "allow *"

3) 生成用於集群初始化初始化的cluster.bootstrap keyring

cp ./ceph.mon.keyring ./cluster.bootstrap.keyring ceph-authtool ./cluster.bootstrap.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring

查看生成的集群初始化keyring:

[root@ceph001-node1 build]# cat ./cluster.bootstrap.keyring 
[mon.]
        key = AQCG0m5Z9AifFxAAreLxG7PXYPXRNyNlRzrGhQ==
        caps mon = "allow *"
[client.admin]
        key = AQC61W5ZQlCOJRAAkXEE7xZtNiZwudgVqRtvuQ==
        auid = 0
        caps mds = "allow"
        caps mon = "allow *"
        caps osd = "allow *"

4) 生成初始化monmap

這里我們為了方便，一開始就將ceph001-node1,ceph001-node2,ceph001-node3同時作為初始化monitor。這可以減少操作步驟，但是必須要等到3個monitor同時建立完成之后monitor集群才能正常工作。

UUID=`uuidgen` echo $UUID monmaptool --create --add ceph001-node1 10.133.134.211 --add ceph001-node2 10.133.134.212 --add ceph001-node3 10.133.134.213 --fsid $UUID ./bootstrap-monmap.bin

查看生成的bootstrap-monmap.bin文件：

[root@ceph001-node1 build]# monmaptool --print ./bootstrap-monmap.bin 
monmaptool: monmap file ./bootstrap-monmap.bin
epoch 0
fsid ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
last_changed 2017-07-19 12:27:17.374031
created 2017-07-19 12:27:17.374031
0: 10.133.134.211:6789/0 mon.ceph001-node1
1: 10.133.134.212:6789/0 mon.ceph001-node2
2: 10.133.134.213:6789/0 mon.ceph001-node3

5) 對ceph001-node1節點monitor初始化

sudo ceph-mon --mkfs -i ceph001-node1 --monmap ./bootstrap-monmap.bin --keyring ./cluster.bootstrap.keyring touch /var/lib/ceph/mon/ceph-ceph001-node1/{done,sysvinit} #在數據目錄建立done及sysvinit兩個文件

初始化完成后，查看數據目錄：

[root@ceph001-node1 build]# ls -al /var/lib/ceph/mon/ceph-ceph001-node1/
total 4
drwxr-xr-x 3 root root 61 Jul 19 13:54 .
drwxr-xr-x 3 root root 31 Jul 19 13:54 ..
-rw-r--r-- 1 root root  0 Jul 19 13:54 done
-rw------- 1 root root 77 Jul 19 13:54 keyring
drwxr-xr-x 2 root root 80 Jul 19 13:54 store.db
-rw-r--r-- 1 root root  0 Jul 19 13:54 sysvinit

6）創建monitor進程配置文件

創建/etc/ceph/ceph.conf文件：

[global] 
fsid = ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
mon_initial_members = ceph001-node1,ceph001-node2,ceph001-node3
mon_host = 10.133.134.211,10.133.134.212,10.133.134.213
auth_supported = cephx
auth_cluster_required = cephx
auth_client_required = cephx
auth_service_required = cephx
osd_pool_default_crush_rule = 2
osd_pool_default_size = 3
osd_pool_default_pg_num = 8
osd_pool_default_pgp_num = 8
osd_crush_chooseleaf_type = 0
mon_osd_full_ratio = 0.95
mon_osd_nearfull_ratio = 0.85

[mon.ceph001-node1] 
host = ceph001-node1
mon_data = /var/lib/ceph/mon/ceph-ceph001-node1
mon_addr = 10.133.134.211:6789

這里fsid為我們初始化monitor時所用的fsid，請參看第4步；mon_initial_members及mon_host為當前需要部署monitor的節點；[mon.ceph001-node1]節點為設置ceph001-node1節點的monitor的相關參數,使用時請注意對應.

7）啟動monitor

sudo /etc/init.d/ceph start mon.ceph001-node1

輸出信息如下：

[root@ceph001-node1 build]# sudo /etc/init.d/ceph start mon.ceph001-node1
=== mon.ceph001-node1 === 
Starting Ceph mon.ceph001-node1 on ceph001-node1...
Running as unit ceph-mon.ceph001-node1.1500444774.538123924.service.
Starting ceph-create-keys on ceph001-node1...

另外，如果當前Linux操作系統上ceph支持service的話，那么也可以用如下方式來啟動：

# chkconfig --list

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

ceph            0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off

# service ceph restart mon

8) 查看monitor狀態

此時，因為我們配置文件中同時指定了3個initial monitors，但是目前我們只啟動了1個，因此monitor會出現如下狀況：

[root@ceph001-node1 build]# ceph -s
2017-07-19 14:14:11.167910 7f7a10217700  0 -- :/1124433248 >> 10.133.134.213:6789/0 pipe(0x7f7a0c068550 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f7a0c05bb80).fault
2017-07-19 14:14:14.151613 7f7a10116700  0 -- :/1124433248 >> 10.133.134.212:6789/0 pipe(0x7f7a00000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f7a00004ef0).fault
2017-07-19 14:14:17.152012 7f7a10217700  0 -- :/1124433248 >> 10.133.134.213:6789/0 pipe(0x7f7a000081b0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f7a0000c450).fault

9）分發mon keyring,mon map及admin keyring

這里我們在后續建立其他monitor/osd節點時，都會用到上述生成的/ceph-cluster/build/bootstrap-monmap.bin , /ceph-cluster/build/ceph.mon.keyring 以及 /etc/ceph/ceph.client.admin.keyring 三個文件，因此這里先把這三個文件分別推送到ceph001-node2,ceph001-node3節點的對應目錄里。

采用如下命令拷貝文件：

scp /etc/ceph/ceph.client.admin.keyring root@10.133.134.212:/etc/ceph/ scp /ceph-cluster/build/bootstrap-monmap.bin root@10.133.134.212:/ceph-cluster/build/ scp /ceph-cluster/build/ceph.mon.keyring root@10.133.134.212:/ceph-cluster/build/ scp /etc/ceph/ceph.client.admin.keyring root@10.133.134.213:/etc/ceph/ scp /ceph-cluster/build/bootstrap-monmap.bin root@10.133.134.213:/ceph-cluster/build/ scp /ceph-cluster/build/ceph.mon.keyring root@10.133.134.213:/ceph-cluster/build/

在ceph001-node2上建立monitor

在上面我們已經完成了第一個monitor的初始化，建立第二個monitor就會簡單很多了。並且我們已經有了用於初始化的bootstrap-monmap.bin以及ceph.mon.keyring。在/ceph-cluster/build目錄下進行如下步驟：

1）初始化monitor

ceph-mon -i ceph001-node2 --mkfs --monmap ./bootstrap-monmap.bin --keyring ./ceph.mon.keyring

2) 初始化數據目錄

ceph-mon --inject-monmap ./bootstrap-monmap.bin --mon-data /var/lib/ceph/mon/ceph-ceph001-node2/ touch /var/lib/ceph/mon/ceph-ceph001-node2/{done,sysvinit}

初始化后，查看monitor默認的數據目錄:

[root@ceph001-node2 build]# ls -al /var/lib/ceph/mon/ceph-ceph001-node2/
total 4
drwxr-xr-x 3 root root  61 Jul 19 14:42 .
drwxr-xr-x 3 root root  31 Jul 19 14:41 ..
-rw-r--r-- 1 root root   0 Jul 19 14:42 done
-rw------- 1 root root  77 Jul 19 14:41 keyring
drwxr-xr-x 2 root root 111 Jul 19 14:42 store.db
-rw-r--r-- 1 root root   0 Jul 19 14:42 sysvinit

3) 修改ceph配置文件

可以從ceph001-node1節點將/etc/ceph/ceph.conf文件拷貝到ceph001-node2節點對應的目錄，然后對[mon] section進行修改：

[global] 
fsid = ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
mon_initial_members = ceph001-node1,ceph001-node2,ceph001-node3
mon_host = 10.133.134.211,10.133.134.212,10.133.134.213
auth_supported = cephx
auth_cluster_required = cephx
auth_client_required = cephx
auth_service_required = cephx
osd_pool_default_crush_rule = 2
osd_pool_default_size = 3
osd_pool_default_pg_num = 8
osd_pool_default_pgp_num = 8
osd_crush_chooseleaf_type = 0
mon_osd_full_ratio = 0.95
mon_osd_nearfull_ratio = 0.85

[mon.ceph001-node2] 
host = ceph001-node2
mon_data = /var/lib/ceph/mon/ceph-ceph001-node2
mon_addr = 10.133.134.212:6789

注意上面對[mon]段的修改。

3) 啟動monitor

sudo /etc/init.d/ceph start mon.ceph001-node2

輸出信息如下：

[root@ceph001-node2 build]# sudo /etc/init.d/ceph start mon.ceph001-node2
=== mon.ceph001-node2 === 
Starting Ceph mon.ceph001-node2 on ceph001-node2...
Running as unit ceph-mon.ceph001-node2.1500446828.301474392.service.
Starting ceph-create-keys on ceph001-node2...

另外，如果當前Linux操作系統上ceph支持service的話，那么也可以用如下方式來啟動：

# chkconfig --list

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

ceph            0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off

# service ceph restart mon

4) 查看monitor狀態

[root@ceph001-node2 build]# ceph -s
2017-07-19 14:47:11.705625 7f4f60124700  0 -- :/749124637 >> 10.133.134.213:6789/0 pipe(0x7f4f5c068550 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f4f5c05bb80).fault
    cluster ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
     health HEALTH_ERR
            no osds
            1 mons down, quorum 0,1 ceph001-node1,ceph001-node2
     monmap e1: 3 mons at {ceph001-node1=10.133.134.211:6789/0,ceph001-node2=10.133.134.212:6789/0,ceph001-node3=10.133.134.213:6789/0}
            election epoch 2, quorum 0,1 ceph001-node1,ceph001-node2
     osdmap e1: 0 osds: 0 up, 0 in
      pgmap v2: 0 pgs, 0 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail

上面我們可以看到，盡管我們的mon_initial_members中設置了3個monitor，但是只要啟動兩個就達到了PAXOS協議的法定人數，monitor集群這時已經可以工作了。但是因為我們還沒有建立OSD，因此當前集群狀態為HEALTH_ERR

在ceph001-node3上建立monitor

與ceph001-node2節點建立monitor類似，這里直接簡要列出：

1）初始化monitor及數據目錄

ceph-mon -i ceph001-node3 --mkfs --monmap ./bootstrap-monmap.bin --keyring ./ceph.mon.keyring ceph-mon --inject-monmap ./bootstrap-monmap.bin --mon-data /var/lib/ceph/mon/ceph-ceph001-node3/ touch /var/lib/ceph/mon/ceph-ceph001-node3/{done,sysvinit} ls -al /var/lib/ceph/mon/ceph-ceph001-node3/

2) 修改ceph配置文件

[global] 
fsid = ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
mon_initial_members = ceph001-node1,ceph001-node2,ceph001-node3
mon_host = 10.133.134.211,10.133.134.212,10.133.134.213
auth_supported = cephx
auth_cluster_required = cephx
auth_client_required = cephx
auth_service_required = cephx
osd_pool_default_crush_rule = 2
osd_pool_default_size = 3
osd_pool_default_pg_num = 8
osd_pool_default_pgp_num = 8
osd_crush_chooseleaf_type = 0
mon_osd_full_ratio = 0.95
mon_osd_nearfull_ratio = 0.85

[mon.ceph001-node3] 
host = ceph001-node3
mon_data = /var/lib/ceph/mon/ceph-ceph001-node3
mon_addr = 10.133.134.213:6789

3) 啟動及查看相應狀態

sudo /etc/init.d/ceph start mon.ceph001-node3 ceph -s

另外，如果當前Linux操作系統上ceph支持service的話，那么也可以用如下方式來啟動：

# chkconfig --list

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

ceph            0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off

# service ceph restart mon

4) 查看狀態信息

通過如下的命令查看當前ceph集群的狀態信息：

[root@ceph001-node3 build]# ceph -s
    cluster ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
     health HEALTH_ERR
            no osds
     monmap e1: 3 mons at {ceph001-node1=10.133.134.211:6789/0,ceph001-node2=10.133.134.212:6789/0,ceph001-node3=10.133.134.213:6789/0}
            election epoch 4, quorum 0,1,2 ceph001-node1,ceph001-node2,ceph001-node3
     osdmap e1: 0 osds: 0 up, 0 in
      pgmap v2: 0 pgs, 0 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail

如上所示，3個monitor已經正常啟動，只是因為我們還未添加任何OSD，導致當前集群處於HEALTH_ERR狀態。

3.2 建立OSD

我們目前會在每一個節點上部署3個OSD，總共3個節點則一共會部署9個OSD。首先查看我們當前的硬盤信息：

[root@ceph001-node1 build]# lsblk -a
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  436K  0 rom  
vda    253:0    0   20G  0 disk 
└─vda1 253:1    0   20G  0 part /
vdb    253:16   0  100G  0 disk 
vdc    253:32   0  100G  0 disk 
vdd    253:48   0   15G  0 disk

如上所示，我們有兩個100G的硬盤vdb、vdc；另外還有一個15G的硬盤vdd。我們做如下規划：將vdb、vdc各分成兩個50G分區，總共4個分區，其中前3個區用作當前OSD的數據目錄,第4個分區保留；另外將vdd作為該節點上所有OSD的日志目錄，分成4個分區。因此，這里我們首先對ceph001-node1,ceph001-node2,ceph001-node3三個節點進行分區：

parted -s /dev/vdb mklabel gpt parted -s /dev/vdb mkpart primary 0% 50% parted -s /dev/vdb mkpart primary 50% 100% parted -s /dev/vdc mklabel gpt parted -s /dev/vdc mkpart primary 0% 50% parted -s /dev/vdc mkpart primary 50% 100% parted -s /dev/vdd mklabel gpt parted -s /dev/vdd mkpart primary 0% 25% parted -s /dev/vdd mkpart primary 25% 50% parted -s /dev/vdd mkpart primary 50% 75% parted -s /dev/vdd mkpart primary 75% 100%

查看分區后的狀態：

[root@ceph001-node1 build]# lsblk -a 
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  436K  0 rom  
vda    253:0    0   20G  0 disk 
└─vda1 253:1    0   20G  0 part /
vdb    253:16   0  100G  0 disk 
├─vdb1 253:17   0   50G  0 part 
└─vdb2 253:18   0   50G  0 part 
vdc    253:32   0  100G  0 disk 
├─vdc1 253:33   0   50G  0 part 
└─vdc2 253:34   0   50G  0 part 
vdd    253:48   0   15G  0 disk 
├─vdd1 253:49   0  3.8G  0 part 
├─vdd2 253:50   0  3.8G  0 part 
├─vdd3 253:51   0  3.8G  0 part 
└─vdd4 253:52   0  3.8G  0 part

在ceph001-node1上建立3個OSD節點

1) 在ceph001-node1上建立3個OSD

使用腳本在ceph001-node1上建立OSD(請分步執行如下命令，減少可能的出錯機會）：

chmod 777 ./script/ -R ./script/init_osd.sh vdb1 vdd1 cat /tmp/init_osd-{OSD_ID}.log sudo /etc/init.d/ceph start osd.{OSD_ID} ceph -s ./script/init_osd.sh vdb2 vdd2 cat /etc/init_osd-{OSD_ID}.log sudo /etc/init.d/ceph start osd.{OSD_ID} ceph -s ./script/init_osd.sh vdc1 vdd3 cat /etc/init_osd-{OSD_ID}.log sudo /etc/init.d/ceph start osd.{OSD_ID} ceph -s

上面請用對應的具體的OSD_ID值代替。輸出信息類似於：

[root@ceph001-node1 build]# cat /tmp/init_osd-0.log
meta-data=/dev/vdb1              isize=2048   agcount=4, agsize=3276736 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=13106944, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=6399, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
mkdir: created directory ‘/var/lib/ceph/osd/ceph-0’
########WWN#######
########WWN#######
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
2017-07-19 15:41:43.124127 7ff5ec9d3880 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected 26383e0b-ea75-4ac2-b2e3-227560101a62, invalid (someone else's?) journal
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
2017-07-19 15:41:43.245550 7ff5ec9d3880 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2017-07-19 15:41:43.358726 7ff5ec9d3880 -1 created object store /var/lib/ceph/osd/ceph-0 journal /dev/disk/by-id/virtio-06ef4ac3-ecec-45b5-8-part1 for osd.0 fsid ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
2017-07-19 15:41:43.358784 7ff5ec9d3880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory
2017-07-19 15:41:43.359121 7ff5ec9d3880 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring
added key for osd.0

另外，如果當前Linux操作系統的ceph支持service的話，也可以通過如下方式啟動：

# chkconfig --list

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

ceph            0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off

# service ceph restart osd.{OSD_ID}

2）查看OSD的啟動狀況

ps -ef | grep osd ceph -s

查看信息如下：

[root@ceph001-node1 build]# ceph -s
    cluster ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
     health HEALTH_WARN
            too few PGs per OSD (0 < min 30)
     monmap e1: 3 mons at {ceph001-node1=10.133.134.211:6789/0,ceph001-node2=10.133.134.212:6789/0,ceph001-node3=10.133.134.213:6789/0}
            election epoch 4, quorum 0,1,2 ceph001-node1,ceph001-node2,ceph001-node3
     osdmap e10: 3 osds: 3 up, 3 in
      pgmap v14: 0 pgs, 0 pools, 0 bytes data, 0 objects
            101136 kB used, 149 GB / 149 GB avail
[root@ceph001-node1 build]# ps -ef | grep osd
root     17392     1  0 15:52 ?        00:00:00 /bin/bash -c ulimit -n 32768;  /usr/bin/ceph-osd -i 0 --pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root     17393 17392  0 15:52 ?        00:00:00 /usr/bin/ceph-osd -i 0 --pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root     17809     1  0 15:52 ?        00:00:00 /bin/bash -c ulimit -n 32768;  /usr/bin/ceph-osd -i 1 --pid-file /var/run/ceph/osd.1.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root     17810 17809  0 15:52 ?        00:00:00 /usr/bin/ceph-osd -i 1 --pid-file /var/run/ceph/osd.1.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root     18213     1  0 15:53 ?        00:00:00 /bin/bash -c ulimit -n 32768;  /usr/bin/ceph-osd -i 2 --pid-file /var/run/ceph/osd.2.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root     18215 18213  0 15:53 ?        00:00:00 /usr/bin/ceph-osd -i 2 --pid-file /var/run/ceph/osd.2.pid -c /etc/ceph/ceph.conf --cluster ceph -f
root     18370 16930  0 15:53 pts/0    00:00:00 grep --color=auto osd
[root@ceph001-node1 build]# ceph -s
    cluster ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
     health HEALTH_WARN
            too few PGs per OSD (0 < min 30)
     monmap e1: 3 mons at {ceph001-node1=10.133.134.211:6789/0,ceph001-node2=10.133.134.212:6789/0,ceph001-node3=10.133.134.213:6789/0}
            election epoch 4, quorum 0,1,2 ceph001-node1,ceph001-node2,ceph001-node3
     osdmap e10: 3 osds: 3 up, 3 in
      pgmap v14: 0 pgs, 0 pools, 0 bytes data, 0 objects
            101136 kB used, 149 GB / 149 GB avail

在ceph001-node2上建立3個OSD節點

與ceph001-node1類似，只需要注意少許參數的修改，這里不在贅述。

在ceph001-node3上建立3個OSD節點

與ceph001-node1類似，只需要注意少許參數的修改，這里不再贅述。

3.3 構建crush map

在上面3.1節建立好OSD之后，默認的crush map如下圖所示：

[root@ceph001-node1 build]# ceph osd tree
ID WEIGHT  TYPE NAME              UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 0.44989 root default                                             
-2 0.14996     host ceph001-node1                                   
 0 0.04999         osd.0               up  1.00000          1.00000 
 1 0.04999         osd.1               up  1.00000          1.00000 
 2 0.04999         osd.2               up  1.00000          1.00000 
-3 0.14996     host ceph001-node2                                   
 3 0.04999         osd.3               up  1.00000          1.00000 
 4 0.04999         osd.4               up  1.00000          1.00000 
 5 0.04999         osd.5               up  1.00000          1.00000 
-4 0.14996     host ceph001-node3                                   
 6 0.04999         osd.6               up  1.00000          1.00000 
 7 0.04999         osd.7               up  1.00000          1.00000 
 8 0.04999         osd.8               up  1.00000          1.00000

這不適用與我們的生產環境。下面我們就來構建我們自己的crush map.

1) 刪除默認的crush map結構

for i in {0..8}; do ceph osd crush rm osd.$i; done for i in {1..3}; do ceph osd crush rm ceph001-node$i; done ceph osd tree

執行完后，當前的ceph集群視圖如下：

[root@ceph001-node1 build]# ceph osd tree
ID WEIGHT TYPE NAME    UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1      0 root default                                   
 0      0 osd.0             up  1.00000          1.00000 
 1      0 osd.1             up  1.00000          1.00000 
 2      0 osd.2             up  1.00000          1.00000 
 3      0 osd.3             up  1.00000          1.00000 
 4      0 osd.4             up  1.00000          1.00000 
 5      0 osd.5             up  1.00000          1.00000 
 6      0 osd.6             up  1.00000          1.00000 
 7      0 osd.7             up  1.00000          1.00000 
 8      0 osd.8             up  1.00000          1.00000

2) 修改crush rule規則內容

ceph osd getcrushmap -o ./old_crushmap.bin crushtool -d ./old_crushmap.bin -o ./old_crushmap.txt cp old_crushmap.txt new_crushmap.txt

下面修改new_crushmap.txt:

# 在type 10 root下面添加邏輯拓撲中的bucket類型, 其中數值越大, 表示在crush map中的層級越大
type 11 osd-domain
type 12 host-domain
type 13 replica-domain
type 14 failure-domain
# 將crush map中所有的 alg straw2 修改為 alg starw

修改后重新設置到ceph集群中：

crushtool -c new_crushmap.txt -o new_crushmap.bin ceph osd setcrushmap -i new_crushmap.bin ceph osd crush dump

3) 重新構建crush map中的物理拓撲

請分步執行如下命令：

for i in {0..2}; do ceph osd crush create-or-move osd.$i 0.15 host=ceph001-node1 rack=rack-01 root=default; done for i in {3..5}; do ceph osd crush create-or-move osd.$i 0.15 host=ceph001-node2 rack=rack-02 root=default; done for i in {6..8}; do ceph osd crush create-or-move osd.$i 0.15 host=ceph001-node3 rack=rack-03 root=default; done ceph osd tree

構建完成后，查看對應的物理拓撲結構：

[root@ceph001-node1 build]# ceph osd tree
ID WEIGHT  TYPE NAME                  UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 1.34995 root default                                                 
-3 0.44998     rack rack-02                                             
-2 0.44998         host ceph001-node2                                   
 3 0.14999             osd.3               up  1.00000          1.00000 
 4 0.14999             osd.4               up  1.00000          1.00000 
 5 0.14999             osd.5               up  1.00000          1.00000 
-5 0.44998     rack rack-03                                             
-4 0.44998         host ceph001-node3                                   
 6 0.14999             osd.6               up  1.00000          1.00000 
 7 0.14999             osd.7               up  1.00000          1.00000 
 8 0.14999             osd.8               up  1.00000          1.00000 
-7 0.44998     rack rack-01                                             
-6 0.44998         host ceph001-node1                                   
 0 0.14999             osd.0               up  1.00000          1.00000 
 1 0.14999             osd.1               up  1.00000          1.00000 
 2 0.14999             osd.2               up  1.00000          1.00000

4）重新構建crush map中的邏輯拓撲

請分步執行如下命令：

ceph osd crush link ceph001-node1 host-domain=host-group-0-rack-01 replica-domain=replica-0 failure-domain=sata-00 ceph osd crush link ceph001-node2 host-domain=host-group-0-rack-02 replica-domain=replica-0 failure-domain=sata-00 ceph osd crush link ceph001-node3 host-domain=host-group-0-rack-03 replica-domain=replica-0 failure-domain=sata-00 ceph osd tree

構建完成后，查看對應的邏輯拓撲結構：

[root@ceph001-node1 build]# ceph osd tree
ID  WEIGHT  TYPE NAME                                UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-10 1.34995 failure-domain sata-00                                                     
 -9 1.34995     replica-domain replica-0                                               
 -8 0.44998         host-domain host-group-0-rack-01                                   
 -6 0.44998             host ceph001-node1                                             
  0 0.14999                 osd.0                         up  1.00000          1.00000 
  1 0.14999                 osd.1                         up  1.00000          1.00000 
  2 0.14999                 osd.2                         up  1.00000          1.00000 
-11 0.44998         host-domain host-group-0-rack-02                                   
 -2 0.44998             host ceph001-node2                                             
  3 0.14999                 osd.3                         up  1.00000          1.00000 
  4 0.14999                 osd.4                         up  1.00000          1.00000 
  5 0.14999                 osd.5                         up  1.00000          1.00000 
-12 0.44998         host-domain host-group-0-rack-03                                   
 -4 0.44998             host ceph001-node3                                             
  6 0.14999                 osd.6                         up  1.00000          1.00000 
  7 0.14999                 osd.7                         up  1.00000          1.00000 
  8 0.14999                 osd.8                         up  1.00000          1.00000 
 -1 1.34995 root default                                                               
 -3 0.44998     rack rack-02                                                           
 -2 0.44998         host ceph001-node2                                                 
  3 0.14999             osd.3                             up  1.00000          1.00000 
  4 0.14999             osd.4                             up  1.00000          1.00000 
  5 0.14999             osd.5                             up  1.00000          1.00000 
 -5 0.44998     rack rack-03                                                           
 -4 0.44998         host ceph001-node3                                                 
  6 0.14999             osd.6                             up  1.00000          1.00000 
  7 0.14999             osd.7                             up  1.00000          1.00000 
  8 0.14999             osd.8                             up  1.00000          1.00000 
 -7 0.44998     rack rack-01                                                           
 -6 0.44998         host ceph001-node1                                                 
  0 0.14999             osd.0                             up  1.00000          1.00000 
  1 0.14999             osd.1                             up  1.00000          1.00000 
  2 0.14999             osd.2                             up  1.00000          1.00000

5）構建crush rule規則

ceph osd getcrushmap -o origin_crushmap.bin crushtool -d origin_crushmap.bin -o origin_crushmap.txt cp origin_crushmap.txt tobuild_crushmap.txt

修改tobuild_crushmap.txt文件,手動添加如下內容：

rule replicated_rule-5 {
     ruleset 5
     type replicated
     min_size 1
     max_size 10
     step take sata-00
     step choose firstn 1 type replica-domain
     step chooseleaf firstn 0 type host-domain 
     step emit
}

修改完成后，重新設置到ceph集群中。

crushtool -c tobuild_crushmap.txt -o tobuild_crushmap.bin ceph osd setcrushmap -i tobuild_crushmap.bin ceph osd crush dump

6) 創建pool, 並將pool綁定之指定crush_ruleset

ceph osd pool delete rbd rbd --yes-i-really-really-mean-it ceph osd pool create rbd-01 128 128 ceph osd pool set rbd-01 size 3 ceph osd pool set rbd-01 crush_ruleset 5

7) 使用rbd命令簡單測試創建的pool是否能夠正常使用

rbd create rbd-01/test-image --size 4096 rbd info rbd-01/test-image rbd rm rbd-01/test-image

到此為止，crush map就已經構建完畢。

4. 構建RGW

4.1 在ceph001-node1上部署RGW

1) 創建存儲池

這里直接執行創建存儲池的腳本：

./script/create_pool.sh ceph osd dump

創建存儲池后，執行結果如下(部分打印信息)：

[root@ceph001-node1 build]# ceph osd dump
epoch 91
fsid ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
created 2017-07-19 14:47:13.802418
modified 2017-07-19 17:11:50.806402
flags 
pool 1 'rbd-01' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 128 pgp_num 128 last_change 61 flags hashpspool stripe_width 0
        removed_snaps [1~3]
pool 2 '.rgw' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 76 flags hashpspool stripe_width 0
pool 3 '.rgw.root' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 77 flags hashpspool stripe_width 0
pool 4 '.rgw.control' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 78 flags hashpspool stripe_width 0
pool 5 '.rgw.gc' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 79 flags hashpspool stripe_width 0
pool 6 '.log' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 80 flags hashpspool stripe_width 0
pool 7 '.intent-log' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 81 flags hashpspool stripe_width 0
pool 8 '.usage' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 82 flags hashpspool stripe_width 0
pool 9 '.users' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 83 flags hashpspool stripe_width 0
pool 10 '.users.email' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 84 flags hashpspool stripe_width 0
pool 11 '.users.swift' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 85 flags hashpspool stripe_width 0
pool 12 '.users.uid' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 86 flags hashpspool stripe_width 0
pool 13 '.rgw.buckets' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 256 pgp_num 256 last_change 87 flags hashpspool stripe_width 0
pool 14 '.rgw.buckets.index' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 88 flags hashpspool stripe_width 0
pool 15 '.rgw.buckets.extra' replicated size 3 min_size 2 crush_ruleset 5 object_hash rjenkins pg_num 8 pgp_num 8 last_change 89 flags hashpspool stripe_width 0
max_osd 9

2）創建RGW秘鑰並加入集群

# ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring --gen-key -n client.radosgw.ceph001-node1 --cap mon 'allow rwx' --cap osd 'allow rwx' # ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.ceph001-node1 -i /etc/ceph/ceph.client.radosgw.keyring # ceph auth list

3) 修改ceph配置文件

修改/etc/ceph/ceph.conf配置文件，在其中添加：

[client.radosgw.ceph001-node1]
host = ceph001-node1
log_file = /var/log/ceph/radosgw-ceph001-node1.log
rgw_s3_auth_use_keystone = False
rgw_frontends = civetweb port=7480
rgw_socket_path = /var/run/ceph/ceph.radosgw.ceph001-node1.sock
user = root
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw_override_bucket_index_max_shards = 0
rgw_swift_token_expiration = 86400
rgw_enable_usage_log = True
rgw_cache_lru_size = 10000
rgw_print_continue = False
rgw_cache_enabled = True
admin_socket = /var/run/ceph/radosgw-ceph001-node1.asok
rgw_thread_pool_size=512
rgw_num_rados_handles=512

4) 啟動RGW

radosgw -c /etc/ceph/ceph.conf -n client.radosgw.ceph001-node1

查看是否啟動成功：

[root@ceph001-node1 build]# netstat -lpn | grep radosgw
tcp        0      0 0.0.0.0:7480            0.0.0.0:*               LISTEN      21229/radosgw       
unix  2      [ ACC ]     STREAM     LISTENING     403819   21229/radosgw        /var/run/ceph/radosgw-ceph001-node1.asok

5）創建RADOS RGW admin用戶並為其分配權限

radosgw-admin user create --uid=admin --display-name="admin" radosgw-admin caps add --uid=admin --caps="buckets=*" radosgw-admin caps add --uid=admin --caps="data=*" radosgw-admin caps add --uid=admin --caps="metadata=*" radosgw-admin caps add --uid=admin --caps="usage=*" radosgw-admin caps add --uid=admin --caps="users=*" radosgw-admin caps add --uid=admin --caps="zone=*"

至此，ceph001-node1節點的radosgw已經部署完成

4.2 在ceph001-node2上部署RGW

此處只需要創建相應的rgw用戶並加入集群，然后配置ceph.conf文件，再啟動rgw即可。下面是詳細步驟：

1）創建RGW秘鑰並加入集群

ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring --gen-key -n client.radosgw.ceph001-node2 --cap mon 'allow rwx' --cap osd 'allow rwx' ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.ceph001-node2 -i /etc/ceph/ceph.client.radosgw.keyring ceph auth list

2) 修改ceph配置文件

修改/etc/ceph/ceph.conf配置文件，在其中添加：

[client.radosgw.ceph001-node2]
host = ceph001-node2
log_file = /var/log/ceph/radosgw-ceph001-node2.log
rgw_s3_auth_use_keystone = False
rgw_frontends = civetweb port=7480
rgw_socket_path = /var/run/ceph/ceph.radosgw.ceph001-node2.sock
user = root
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw_override_bucket_index_max_shards = 0
rgw_swift_token_expiration = 86400
rgw_enable_usage_log = True
rgw_cache_lru_size = 10000
rgw_print_continue = False
rgw_cache_enabled = True
admin_socket = /var/run/ceph/radosgw-ceph001-node2.asok
rgw_thread_pool_size=512
rgw_num_rados_handles=512

3) 啟動RGW

radosgw -c /etc/ceph/ceph.conf -n client.radosgw.ceph001-node2

查看是否啟動成功：

[root@ceph001-node2 build]# netstat -lpn | grep radosgw
tcp        0      0 0.0.0.0:7480            0.0.0.0:*               LISTEN      9756/radosgw        
unix  2      [ ACC ]     STREAM     LISTENING     312548   9756/radosgw         /var/run/ceph/radosgw-ceph001-node2.asok

因為節點ceph001-node1已經創建好了admin賬號以及初始化權限，所以之后的節點都不需要再進行創建了。

至此，節點ceph001-node2部署rgw完畢。

4.3 在ceph001-node3上部署RGW

此處只需要創建相應的rgw用戶並加入集群，然后配置ceph.conf文件，再啟動rgw即可。下面是詳細步驟：

1）創建RGW秘鑰並加入集群

ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring --gen-key -n client.radosgw.ceph001-node3 --cap mon 'allow rwx' --cap osd 'allow rwx' ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.ceph001-node3 -i /etc/ceph/ceph.client.radosgw.keyring ceph auth list

2) 修改ceph配置文件

修改/etc/ceph/ceph.conf配置文件，在其中添加：

[client.radosgw.ceph001-node3]
host = ceph001-node3
log_file = /var/log/ceph/radosgw-ceph001-node3.log
rgw_s3_auth_use_keystone = False
rgw_frontends = civetweb port=7480
rgw_socket_path = /var/run/ceph/ceph.radosgw.ceph001-node3.sock
user = root
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw_override_bucket_index_max_shards = 0
rgw_swift_token_expiration = 86400
rgw_enable_usage_log = True
rgw_cache_lru_size = 10000
rgw_print_continue = False
rgw_cache_enabled = True
admin_socket = /var/run/ceph/radosgw-ceph001-node3.asok
rgw_thread_pool_size=512
rgw_num_rados_handles=512

3) 啟動RGW

radosgw -c /etc/ceph/ceph.conf -n client.radosgw.ceph001-node3

查看是否啟動成功：

[root@ceph001-node3 build]# netstat -lpn | grep radosgw
tcp        0      0 0.0.0.0:7480            0.0.0.0:*               LISTEN      15626/radosgw       
unix  2      [ ACC ]     STREAM     LISTENING     326358   15626/radosgw        /var/run/ceph/radosgw-ceph001-node3.asok

因為節點ceph001-node1已經創建好了admin賬號以及初始化權限，所以之后的節點都不需要再進行創建了。

至此，節點ceph001-node3部署rgw完畢。

最后，到此為止整個集群已經部署完畢.

5. 添加新的mon(附錄）

有時候，我們在ceph運行了一段時間之后，需要添加新的mon。假設我們這里要在10.133.134.214這台主機上添加新的mon節點，此時我們可以參照如下步驟：

5.1 初始化主機

這里參照上面，將10.133.134.214這台主機的主機名設置為ceph001-node4，同時關閉相應的防火牆。請參考上文。

5.2 准備文件

我們需要從原來mon集群中獲得monmap、mon.keyring以及admin.keyring文件，然后分發到ceph001-node4節點上。

1) 導出monmap

我們可以從原來老的mon集群中導出monmap:

# ceph mon getmap -o ./new_monmap.bin
got monmap epoch 43286

# monmaptool --print ./new_monmap.bin

2) 導出mon.keying

我們可以從原來老的mon集群中導出keying:

# ceph auth get mon. -o ./new_keyfile
# cat ./new_keyfile

3) 導出admin.keyring

admin.keyring即為原來集群的/etc/ceph/ceph.client.admin.keyring，此處直接獲取即可。

獲取完上面三個文件之后，直接分發到ceph001-node4節點即可。

5.3 初始化mon節點

1) 創建新的monmap

首先在原來new_monmap.bin的基礎上創建新的monmap:

# monmaptool --create --add ceph001-node4 10.133.134.214 --fsid ba47fcbc-b2f7-4071-9c37-be859d8c7e6e --clobber ./new_monmap.bin

# monmaptool --print ./new_monmap.bin

2) 修改ceph.conf文件

修改配置文件/etc/ceph/ceph.conf:

[global] fsid = ba47fcbc-b2f7-4071-9c37-be859d8c7e6e mon_initial_members = ceph001-node1,ceph001-node2,ceph001-node3,ceph001-node4 mon_host = 10.133.134.211,10.133.134.212,10.133.134.213,10.133.134.214 auth_supported = cephx auth_cluster_required = cephx auth_client_required = cephx auth_service_required = cephx osd_pool_default_crush_rule = 2 osd_pool_default_size = 2 osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8 osd_crush_chooseleaf_type = 0 [mon.ceph001-node4] host = ceph001-node4 mon_data = /var/lib/ceph/mon/ceph-ceph001-node4 mon_addr = 10.133.134.214:6789

3) 創建新的monitor fs

# ceph-mon -i ceph001-node4 --mkfs --monmap ./new_monmap.bin --keyring  ./new_keyfile
ceph-mon: set fsid to ba47fcbc-b2f7-4071-9c37-be859d8c7e6e
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph001-node4 for mon.ceph001-node4

4) 注入monmap

這里我們將上面生成的new_monmap.bin注入到monitor fs中：

# ceph-mon --inject-monmap ./new_monmap.bin --mon-data  /var/lib/ceph/mon/ceph-ceph001-node4/

5) 啟動monitor

# sudo /etc/init.d/ceph start mon.ceph001-node4

這里注意需要將上面的admin.keyring也拷貝到/etc/ceph/ceph.client.admin.keyring。

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 ceph nautilus版本手動安裝手動搭建ceph集群 007 Ceph手動部署單節點 Ceph N版手動部署 Ceph——手動部署（單節點）【Ceph運維】Ceph手動修復mon 集群 CEPH 14.2.9(nautilus)手動安裝遇到的問題及解決方法如何手動安裝MySql 手動安裝gradle 如何手動安裝yapi