ceph 安裝ceph問題匯總

1、在不同節點安裝ceph時，出現以下異常：
參考這里

ceph deploy RuntimeError: NoSectionError: No section: 'ceph' 

解決方法：

在報錯的機器上嘗試：
yum remove ceph-release
rm  /etc/yum.repos.d/ceph.repo.rpmsave

執行以后再執行

ceph-deploy install admin  node1 node2 node3

2、ceph用戶沒有權限
**執行激活osd盤時出現 **

ceph-deploy osd activate node2:/var/local/osd0 node3:/var/local/osd1 

[node3][WARNIN] ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '1', '--monmap', '/var/local/osd1/activate.monmap', '--osd-data', '/var/local/osd1', '--osd-journal', '/var/local/osd1/journal', '--osd-uuid', 'ee9c4a6c-e2b5-46cd-9e91-fd5d6f36fd57', '--keyring', '/var/local/osd1/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2016-08-31 16:38:58.909449 7f48adbb7800 -1 filestore(/var/local/osd1) mkfs: write_version_stamp() failed: (13) Permission denied [node3][WARNIN] 2016-08-31 16:38:58.909475 7f48adbb7800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13 [node3][WARNIN] 2016-08-31 16:38:58.909529 7f48adbb7800 -1 ** ERROR: error creating empty object store in /var/local/osd1: (13) Permission denied [node3][WARNIN] [node3][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /var/local/osd1 

解決方法

chown ceph:ceph  /var/local/osd0 chown ceph:ceph /var/local/osd1 

---

 

創建成功ceph健康狀況良好

3、RuntimeError: bootstrap-rgw keyring not found; run 'gatherkeys'

root@admin ceph]# ceph-deploy rgw create node2 [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.34): /usr/bin/ceph-deploy rgw create node2 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] rgw : [('node2', 'rgw.node2')] [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] subcommand : create [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x24aed88> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] func : <function rgw at 0x241da28> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts node2:rgw.node2 [ceph_deploy][ERROR ] RuntimeError: bootstrap-rgw keyring not found; run 'gatherkeys' 

執行如下命令解決

ceph-deploy gatherkeys node2

4 解決找不到/bootstrap-osd/ceph.keyring

ceph-deploy mon create-initial 命令執行后， 如果提示：

[ceph_deploy.mon][INFO ] Running gatherkeys... [ceph_deploy.gatherkeys][DEBUG ] Checking manila59 for /etc/ceph/ceph.client.admin.keyring [manila59][DEBUG ] connected to host: manila59 [manila59][DEBUG ] detect platform information from remote host [manila59][DEBUG ] detect machine type [manila59][DEBUG ] fetch remote file [ceph_deploy.gatherkeys][WARNIN] Unable to find /etc/ceph/ceph.client.admin.keyring on ['manila59'] [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring [ceph_deploy.gatherkeys][DEBUG ] Checking manila59 for /var/lib/ceph/bootstrap-osd/ceph.keyring [manila59][DEBUG ] connected to host: manila59 [manila59][DEBUG ] detect platform information from remote host [manila59][DEBUG ] detect machine type [manila59][DEBUG ] fetch remote file [ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-osd/ceph.keyring on ['manila59'] [ceph_deploy.gatherkeys][DEBUG ] Checking manila59 for /var/lib/ceph/bootstrap-mds/ceph.keyring [manila59][DEBUG ] connected to host: manila59 [manila59][DEBUG ] detect platform information from remote host [manila59][DEBUG ] detect machine type [manila59][DEBUG ] fetch remote file [ceph_deploy.gatherkeys][WARNIN] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on ['manila59'] 

可以執行： [root@manila59 deploy]# ceph-create-keys --id manila59

分析發現此處可能連接了外網，要確保DNS正常。

ceph-deploy osd prepare manila59:/dev/sdb命令后，如果提示：

[ceph_deploy][ERROR ] RuntimeError: bootstrap-osd keyring not found; run 'gatherkeys'

執行ceph-deploy gatherkeys manila59：

那么就會先找：/etc/ceph/ceph.client.admin.keyring，然后再找/var/lib/ceph/bootstrap-osd/ceph.keyring和/var/lib/ceph/bootstrap-mds/ceph.keyring ，最后發現ceph.bootstrap-mds.keyring key from manila59。

注意：disk zap 命令是不需要權限的，而osd prepare 需要bootstrap-mds.keyring。

 

在上面的 map 映射操作時，可能出現如下錯誤

RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable"

查看系統日志可以看到如下輸出

~ dmesg | tail[-1127592253.530346] rbd: image data: image uses unsupported features: 0x38[-1127590337.563180] libceph: mon0 192.168.1.11:6789 session established[-1127590337.563741] libceph: client4200 fsid dd9fdfee-438a-47aa-be21-114372bc1f44

問題原因: 在 Ceph 高版本進行 map image 時，默認 Ceph 在創建 image(上文 data)時會增加許多 features，這些 features 需要內核支持，在 Centos7 的內核上支持有限，所以需要手動關掉一些 features

首先使用 rbd info data 命令列出創建的 image 的 features

rbd image 'data':
        size 10240 MB in 2560 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.37c6238e1f29
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        flags:

在 features 中我們可以看到默認開啟了很多:

• layering: 支持分層

• striping: 支持條帶化 v2

• exclusive-lock: 支持獨占鎖

• object-map: 支持對象映射(依賴 exclusive-lock)

• fast-diff: 快速計算差異(依賴 object-map)

• deep-flatten: 支持快照扁平化操作

• journaling: 支持記錄 IO 操作(依賴獨占鎖）

而實際上 Centos 7 的 3.10 內核只支持 layering… 所以我們要手動關閉一些 features，然后重新 map；如果想要一勞永逸，可以在 ceph.conf 中加入 rbd_default_features = 1 來設置默認 features(數值僅是 layering 對應的 bit 碼所對應的整數值)。

關閉不支持特性
rbd feature disable data exclusive-lock, object-map, fast-diff, deep-flatten
重新映射
rbd map data --name client.admin
成功后返回設備位置
/dev/rbd0