情況：
ceph 在一次掉盤恢復后, 有 pg 出現 state unknown 的狀況. 運行 ceph health detail, 顯示:
[root@controller ~]# ceph health detail HEALTH_WARN Reduced data availability: 3 pgs inactive PG_AVAILABILITY Reduced data availability: 3 pgs inactive pg 3.4 is stuck inactive for 97176.554825, current state unknown, last acting [] pg 3.b is stuck inactive for 97176.554825, current state unknown, last acting [] pg 3.1e is stuck inactive for 97176.554825, current state unknown, last acting []
顯示這3個 pg 卡住了
運行 pg query, 查看該 pg 的具體信息, 顯示:
[root@controller ~]# ceph pg 3.1e query Error ENOENT: i don't have pgid 3.1e
無法找到該 pg id.
運行 pg dump_stuck unclean, 顯示:
分析
看來是這幾個 pgid 徹底找不到了. 我的 osd pool 有三個, 分別叫 l1 (1副本), l2 (2副本), l3 (3副本).
估計可能是之前寫入 1 副本的數據由於硬盤掛掉導致的數據丟失.
既然是1副本, 也不要求數據可靠性了. 並且本身存儲的也是一些下載到一半的數據, 也沒什么關系
修正
通過閱讀 CEPH 官方 PG troubleshooting 文檔 , 發現了解決方案:
POOL SIZE = 1
If you have the osd pool default size set to 1, you will only have one copy of the object. OSDs rely on other OSDs to tell them which objects they should have. If a first OSD has a copy of an object and there is no second copy, then no second OSD can tell the first OSD that it should have that copy. For each placement group mapped to the first OSD (see ceph pg dump), you can force the first OSD to notice the placement groups it needs by running:
ceph osd force-create-pg <pgid>
即, 多 osd 副本可以互相通知 pg 信息, 但是單副本就會丟, 為了恢復這個pg, 我們可以強行創建它.
[root@controller ~]# ceph osd force-create-pg 3.1e Error EPERM: This command will recreate a lost (as in data lost) PG with data in it, such that the cluster will give up ever trying to recover the lost data. Do this only if you are certain that all copies of the PG are in fact lost and you are willing to accept that the data is permanently destroyed. Pass --yes-i-really-mean-it to proceed.
運行創建命令, 提示, 運行會永久的丟失該 pg 的數據, 需要加上 --yes-i-really-mean-it.
[root@controller ~]# ceph osd force-create-pg 3.1e --yes-i-really-mean-it pg 3.1e now creating, ok [root@controller ~]# ceph osd force-create-pg 3.b --yes-i-really-mean-it pg 3.b now creating, ok [root@controller ~]# ceph osd force-create-pg 3.4 --yes-i-really-mean-it pg 3.4 now creating, ok
執行成功.
查看新創建的 pg.
結論
至此, 修復完畢.
后續檢查, 1 副本的下載文件夾丟了幾個
建議在直到具體是什么問題的情況下才進行這樣的操作. 如果是重要的數據, 請不要使用 1 副本, 並做好備份后再進行操作.
如果是2,3 副本情況下的 pg stat unknown, 建議做好心理准備... 很可能就是沒了.
如果是其他 pg stuck 的情況, 建議仔細分析再進行操作.
以上
轉載於：https://zhuanlan.zhihu.com/p/74323736?from_voters_page=true