RDB 回收站
官網：https://docs.ceph.com/en/latest/rbd/rados-rbd-cmds/
數據可以先保存在回收站里面，配置一個策略周期，當你不需要的時候，再去刪除，當你需要時，可從回收站里面恢復出來，這就是簡單的回收站實現機制；
創建鏡像
[root@ceph-node01 ~]# rbd create ceph-demo/ceph-trash.img --size 10G
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]# rbd info ceph-demo/ceph-trash.img
rbd image 'ceph-trash.img':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 392c7a4143a46
block_name_prefix: rbd_data.392c7a4143a46
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Fri Oct 16 23:11:09 2020
access_timestamp: Fri Oct 16 23:11:09 2020
modify_timestamp: Fri Oct 16 23:11:09 2020
[root@ceph-node01 ~]#
未設置回機制前刪除
[root@ceph-node01 ~]# rbd remove -p ceph-demo ceph-trash.img
Removing image: 100% complete...done.
[root@ceph-node01 ~]#
驗證直接刪除
[root@ceph-node01 ~]# rbd -p ceph-demo ls
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]# rbd -p ceph-demo trash ls
[root@ceph-node01 ~]#
直接刪除的話，直接就沒有了，在對應的pool池和回收站里面都沒有；
驗證回收站刪除
[root@ceph-node01 ~]# rbd create ceph-demo/ceph-trash.img --size 10G
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]# rbd trash move ceph-demo/ceph-trash.img --expires-at 20201020
[root@ceph-node01 ~]# rbd -p ceph-demo ls
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]# rbd -p ceph-demo trash ls
3931bebcf0579 ceph-trash.img
[root@ceph-node01 ~]#
注意在回收站的時間，是根據你上面設置的回收周期而定的，到期后，會自動清理；
找回刪除塊設備
[root@ceph-node01 ~]# rbd trash restore -p ceph-demo 3931bebcf0579
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
[root@ceph-node01 ~]#
這樣就可以防止誤刪除；
RBD 快照
官網：https://docs.ceph.com/en/latest/rbd/rbd-snapshot/
A snapshot is a read-only logical copy of an image at a particular point in time: a checkpoint. One of the advanced features of Ceph block devices is that you can create snapshots of images to retain（保留、持有、保有） point-in-time state history. Ceph also supports snapshot layering（快照分層）, which allows you to clone images (e.g., a VM image) quickly and easily. Ceph block device snapshots are managed using the rbd command and multiple higher level interfaces, including QEMU, libvirt, OpenStack and CloudStack.
Important
To use RBD snapshots, you must have a running Ceph cluster.
Note
Because RBD does not know about any filesystem within an image (volume), snapshots are not crash-consistent unless they are coordinated within the mounting (attaching) operating system. We therefore recommend that you pause or stop I/O before taking a snapshot. If the volume contains a filesystem, it must be in an internally consistent state before taking a snapshot. Snapshots taken at inconsistent points may need a fsck pass before subsequent mounting. To stop I/O you can use fsfreeze command. See fsfreeze(8) man page for more details. For virtual machines, qemu-guest-agent can be used to automatically freeze file systems when creating a snapshot.
如果在做快照時映像仍在進行 I/O 操作，快照可能就獲取不到該映像准確的或最新的數據，並且該快照可能不得不被克隆到一個新的可掛載的映像中。所以，我們建議在做快照前先停止 I/O 操作。如果映像內包含文件系統，在做快照前請確保文件系統處於一致的狀態。要停止 I/O 操作可以使用 fsfreeze 命令。詳情可參考 fsfreeze(8) 手冊頁。對於虛擬機，qemu-guest-agent 被用來在做快照時自動凍結文件系統。
其實快照就是一種備份機制；
創建一塊設備
[root@ceph-node01 ~]# rbd create ceph-demo/rbd-test.img --image-feature layering --size 10G
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
rbd-test.img
[root@ceph-node01 ~]#
查看塊設置信息
[root@ceph-node01 ~]# rbd info -p ceph-demo rbd-test.img
rbd image 'rbd-test.img':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 3936fc8add13c
block_name_prefix: rbd_data.3936fc8add13c
format: 2
features: layering
op_features:
flags:
create_timestamp: Fri Oct 16 23:52:31 2020
access_timestamp: Fri Oct 16 23:52:31 2020
modify_timestamp: Fri Oct 16 23:52:31 2020
[root@ceph-node01 ~]#
掛載
[root@ceph-node01 ~]# rbd device map ceph-demo/rbd-test.img
/dev/rbd1
[root@ceph-node01 ~]# mkfs.ext4 /dev/rbd1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: 完成
文件系統標簽=
OS type: Linux
塊大小=4096 (log=2)
分塊大小=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
655360 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
第一個數據塊=0
Maximum filesystem blocks=2151677952
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: 完成
正在寫入inode表: 完成
Creating journal (32768 blocks): 完成
Writing superblocks and filesystem accounting information: 完成
[root@ceph-node01 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 50G 0 disk
├─vda1 252:1 0 500M 0 part /boot
└─vda2 252:2 0 49.5G 0 part
├─centos-root 253:0 0 44.5G 0 lvm /
└─centos-swap 253:1 0 5G 0 lvm [SWAP]
vdb 252:16 0 100G 0 disk
└─ceph--48dc8bff--6f69--41e1--9bfa--b188edc8f419-osd--block--909ff4ad--d05b--476f--b7e3--56d9e77004b4
253:2 0 100G 0 lvm
vdc 252:32 0 100G 0 disk
└─ceph--6f6493a0--ecaa--45de--8dcb--04f5b6f8e957-osd--block--ded9a04f--71b5--4088--87c3--0e71604c7d75
253:3 0 100G 0 lvm
rbd0 251:0 0 20G 0 disk /mnt/rbd-demo
rbd1 251:16 0 10G 0 disk
[root@ceph-node01 ~]#
[root@ceph-node01 ~]# mount /dev/rbd1 /media/
[root@ceph-node01 ~]# cd /media/
[root@ceph-node01 media]# ls
lost+found
[root@ceph-node01 media]# echo `date` > file.log
[root@ceph-node01 media]# ls
file.log lost+found
[root@ceph-node01 media]# sync
[root@ceph-node01 media]#
使用sync進行刷盤，把當前的狀態保存下來，下面制作快照使用。
制作快照
[root@ceph-node01 media]# rbd snap create ceph-demo/rbd-test.img@snap_20201011
[root@ceph-node01 media]# rbd snap ls ceph-demo/rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 snap_20201011 10 GiB Sat Oct 17 00:02:00 2020
[root@ceph-node01 media]#
這里可以看到快照的名稱及制作快照的時間；
快照恢復
# 1. 模擬文件刪除
[root@ceph-node01 media]# rm -rf file.log
[root@ceph-node01 media]#
# 2. 查看要恢復鏡像的快照名稱
[root@ceph-node01 ~]# rbd snap ls ceph-demo/rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 snap_20201011 10 GiB Sat Oct 17 00:02:00 2020
# 3. 快照恢復
[root@ceph-node01 ~]# rbd snap rollback ceph-demo/rbd-test.img@snap_20201011
Rolling back to snapshot: 100% complete...done.
# 4. 注意這里需要重新掛載
[root@ceph-node01 ~]# umount /media/
[root@ceph-node01 ~]# mount /dev/rbd1 /media/
[root@ceph-node01 ~]# cd /media/
# 5. 查看恢復文件信息
[root@ceph-node01 media]# ls
file.log lost+found
[root@ceph-node01 media]# cat file.log
2020年 10月 16日 星期五 23:58:43 EDT
[root@ceph-node01 media]#
快照的一個目的，就是對鏡像做備份，比如，當你做一些高風險的操作時，把鏡像當時的狀態先保留下來，當數據異常時，可以把數據進行一次回滾；
快照刪除
# 1. 刪除單個快照
[root@ceph-node01 media]# rbd snap remove ceph-demo/rbd-test.img@snap_20201011
Removing snap: 100% complete...done.
[root@ceph-node01 media]#
# 2. 驗證刪除所有快照
[root@ceph-node01 media]# rbd snap create ceph-demo/rbd-test.img@snap_20201011
[root@ceph-node01 media]# rbd snap create ceph-demo/rbd-test.img@snap_20201012
[root@ceph-node01 media]# rbd snap create ceph-demo/rbd-test.img@snap_20201013
[root@ceph-node01 media]# rbd snap ls ceph-demo/rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
6 snap_20201011 10 GiB Sat Oct 17 00:17:58 2020
7 snap_20201012 10 GiB Sat Oct 17 00:18:00 2020
8 snap_20201013 10 GiB Sat Oct 17 00:18:04 2020
[root@ceph-node01 media]#
[root@ceph-node01 media]# rbd snap purge ceph-demo/rbd-test.img
Removing all snapshots: 100% complete...done.
[root@ceph-node01 media]# rbd snap ls ceph-demo/rbd-test.img
[root@ceph-node01 media]#
鏡像克隆機制
Ceph supports the ability to create many copy-on-write (COW) clones of a block device snapshot. Snapshot layering enables Ceph block device clients to create images very quickly. For example, you might create a block device image with a Linux VM written to it; then, snapshot the image, protect the snapshot, and create as many copy-on-write clones as you like. A snapshot is read-only, so cloning a snapshot simplifies semantics–making（克隆快照的語義） it possible to create clones rapidly.
The terms “parent” and “child” refer to a Ceph block device snapshot (parent), and the corresponding image cloned from the snapshot (child). These terms are important for the command line usage below.
這里的術語“父”和“子”指的是一個 Ceph 塊設備快照（父），和從此快照克隆出來的對應映像（子）。這些術語對下列的命令行用法來說很重要。
Each cloned image (child) stores a reference to its parent image, which enables the cloned image to open the parent snapshot and read it.
每一個克隆出來的映像（子）都存儲着對父映像的引用，這使得克隆出來的映像可以打開父映像並讀取它。
A COW clone of a snapshot behaves exactly like any other Ceph block device image. You can read to, write from, clone, and resize cloned images. There are no special restrictions with cloned images. However, the copy-on-write clone of a snapshot depends on the snapshot, so you MUST protect the snapshot before you clone it. The following diagram depicts the process.
一個快照的 COW 克隆的行為更像其它任何 Ceph 塊設備映像的一樣。克隆出的映像沒有特別的限制，你可以讀出、寫入、克隆、調整克隆映像的大小。然而快照的寫時復制克隆引用了快照，所以你克隆快照前必須保護它。下圖描述了此過程。
Ceph only supports cloning of RBD format 2 images (i.e., created with rbd create --image-format 2). The kernel client supports cloned images beginning with the 3.10 release.
Ceph 僅支持克隆 format 2 的映像（即用 rbd create --image-format 2 創建的）。內核客戶端從 3.10 版開始支持克隆的映像。
GETTING STARTED WITH LAYERING
Ceph block device layering is a simple process. You must have an image. You must create a snapshot of the image. You must protect the snapshot. Once you have performed these steps, you can begin cloning the snapshot.
Ceph 塊設備的分層是個簡單的過程。你必須有個映像、必須為它創建快照、並且必須保護快照，執行過這些步驟后，你才能克隆快照。
The cloned image has a reference （引用）to the parent snapshot, and includes the pool ID, image ID and snapshot ID. The inclusion （包含）of the pool ID means that you may clone snapshots from one pool to images in another pool.
1.Image Template（鏡像模板）: A common use case for block device layering is to create a master image and a snapshot that serves as a template for clones. For example, a user may create an image for a Linux distribution（Linux 發行版的映像） (e.g., Ubuntu 12.04), and create a snapshot for it. Periodically（定期的）, the user may update the image and create a new snapshot (e.g., sudo apt-get update, sudo apt-get upgrade, sudo apt-get dist-upgrade followed by rbd snap create). As the image matures（隨着映像的成熟）, the user can clone any one of the snapshots.
2.Extended Template（擴展模板）: A more advanced use case includes extending a template image that provides more information than a base image. For example, a user may clone an image (e.g., a VM template) and install other software (e.g., a database, a content management system, an analytics system, etc.) and then snapshot the extended image, which itself may be updated just like the base image.
3.Template Pool（模板池）: One way to use block device layering is to create a pool that contains master images that act as templates, and snapshots of those templates. You may then extend read-only privileges to users so that they may clone the snapshots without the ability to write or execute within the pool.
4.Image Migration/Recovery（鏡像遷移和恢復）: One way to use block device layering is to migrate or recover data from one pool into another pool.
鏡像克隆用例
# 1. 查看 rbd 塊設備
[root@ceph-node01 ~]# rbd ls -p ceph-demo rbd-test.img
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
rbd-test.img
# 2. 查看快照
[root@ceph-node01 ~]# rbd snap ls -p ceph-demo rbd-test.img
# 3. 創建快照
[root@ceph-node01 ~]# rbd snap create ceph-demo/rbd-test.img@template
[root@ceph-node01 ~]# rbd snap ls -p ceph-demo rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
12 template 10 GiB Sat Oct 17 03:23:22 2020
# 4. 把快照保護起來，以免誤刪
[root@ceph-node01 ~]# rbd snap protect ceph-demo/rbd-test.img@template
# 5. 刪除測試，發現根據刪除不了
[root@ceph-node01 ~]# rbd snap rm ceph-demo/rbd-test.img@template
Removing snap: 2020-10-17 03:23:59.647 7fb6e3844c80 -1 librbd::Operations: snapshot is protected
0% complete...failed.
rbd: snapshot 'template' is protected from removal.
# 5. 基於模板進行 clone 鏡像
[root@ceph-node01 ~]# rbd clone ceph-demo/rbd-test.img@template ceph-demo/vm1-clone.img
[root@ceph-node01 ~]# rbd clone ceph-demo/rbd-test.img@template ceph-demo/vm2-clone.img
[root@ceph-node01 ~]# rbd clone ceph-demo/rbd-test.img@template ceph-demo/vm3-clone.img
[root@ceph-node01 ~]# rbd -p ceph-demo ls
ceph-trash.img
demo.img
rbd-demo.img
rbd-demo2.img
rbd-test.img
vm1-clone.img
vm2-clone.img
vm3-clone.img
# 6. 查看鏡像信息，可以看到繼承信息；
[root@ceph-node01 ~]# rbd -p ceph-demo info vm3-clone.img
rbd image 'vm3-clone.img':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 394a48fcc1688
block_name_prefix: rbd_data.394a48fcc1688
format: 2
features: layering
op_features:
flags:
create_timestamp: Sat Oct 17 03:24:59 2020
access_timestamp: Sat Oct 17 03:24:59 2020
modify_timestamp: Sat Oct 17 03:24:59 2020
parent: ceph-demo/rbd-test.img@template
overlap: 10 GiB
[root@ceph-node01 ~]#
掛載使用
[root@ceph-node01 ~]# rbd device map ceph-demo/vm1-clone.img
/dev/rbd2
[root@ceph-node01 ~]# rbd device map -p ceph-demo vm2-clone.img
/dev/rbd4
[root@ceph-node01 ~]# mkdir abc
[root@ceph-node01 ~]# mount /dev/rbd4 /root/abc
[root@ceph-node01 ~]# cd /root/abc/
[root@ceph-node01 abc]# ls
file.log lost+found
[root@ceph-node01 abc]#
父子鏡像剝離
Cloned images retain（保留） a reference to the parent snapshot. When you remove the reference from the child clone to the parent snapshot, you effectively “flatten” the image by copying the information from the snapshot to the clone. The time it takes to flatten a clone increases with the size of the snapshot. To delete a snapshot, you must flatten the child images first.
克隆出來的映像仍保留了對父快照的引用，要從子克隆刪除這些到父快照的引用，你可以把快照的信息復制給子克隆，也就是“拍平”它，拍平克隆映像的時間隨快照尺寸增大而增加，要刪除快照，必須先拍平子映像。
Since a flattened image contains all the information from the snapshot, a flattened image will take up more storage space than a layered clone。
因為拍平的映像包含了快照的所有信息，所以拍平的映像占用的存儲空間會比分層克隆要大。
查看父鏡像 clone 了多少個子鏡像
[root@ceph-node01 abc]# rbd children ceph-demo/rbd-test.img@template
ceph-demo/vm1-clone.img
ceph-demo/vm2-clone.img
ceph-demo/vm3-clone.img
[root@ceph-node01 abc]#
解除父子關系
[root@ceph-node01 abc]# rbd flatten ceph-demo/vm1-clone.img
Image flatten: 100% complete...done.
[root@ceph-node01 abc]# rbd flatten ceph-demo/vm2-clone.img
Image flatten: 100% complete...done.
[root@ceph-node01 abc]# rbd info ceph-demo/vm1-clone.img
rbd image 'vm1-clone.img':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 3948c22f5e5f
block_name_prefix: rbd_data.3948c22f5e5f
format: 2
features: layering
op_features:
flags:
create_timestamp: Sat Oct 17 03:24:52 2020
access_timestamp: Sat Oct 17 03:24:52 2020
modify_timestamp: Sat Oct 17 03:24:52 2020
[root@ceph-node01 abc]#
刪除父鏡像
# 1. 刪除鏡像模板，無法刪除，原因是還有一個clone對象
[root@ceph-node01 abc]# rbd snap rm ceph-demo/rbd-test.img@template
Removing snap: 0% complete...failed.
rbd: snapshot 'template'2020-10-17 03:51:31.135 7f618c8d8c80 -1 librbd::Operations: snapshot is protected
is protected from removal.
# 2. 解除父子關系
[root@ceph-node01 abc]# rbd flatten ceph-demo/vm3-clone.img
Image flatten: 100% complete...done.
# 3. 查看父子鏡像
[root@ceph-node01 abc]# rbd children ceph-demo/rbd-test.img@template
# 4. 刪除父鏡像
[root@ceph-node01 abc]# rbd snap rm ceph-demo/rbd-test.img@template
Removing snap: 2020-10-17 03:52:21.409 7f3141b96c80 -1 librbd::Operations: snapshot is protected0% complete...failed.
rbd: snapshot 'template' is protected from removal.
# 5. 取消保護機制
[root@ceph-node01 abc]# rbd snap unprotect ceph-demo/rbd-test.img@template
# 6. 刪除原來的鏡像模板（父鏡像）
[root@ceph-node01 abc]# rbd snap rm ceph-demo/rbd-test.img@template
Removing snap: 100% complete...done.
[root@ceph-node01 abc]#
原來的父子鏡像關系解除后，原來的子鏡像與父鏡像就沒有任何關系了，鏡像依然可以掛載映射使用。
[root@ceph-node01 abc]# rbd device ls
id pool namespace image snap device
0 ceph-demo rbd-demo.img - /dev/rbd0
1 ceph-demo rbd-test.img - /dev/rbd1
2 ceph-demo vm1-clone.img - /dev/rbd2
3 ceph-demo vm1-clone.img - /dev/rbd3
4 ceph-demo vm2-clone.img - /dev/rbd4
[root@ceph-node01 abc]#
RBD 備份與恢復
當集群功能不可用時，哪么塊設備還能正常使用嗎，很顯然是不可以的，但在不可抗因素下，很有可能造成整個集群掛掉，哪有什么辦法呢，我們可以使用離線的備份與恢復機制來實現，可以把鏡像導入到磁帶庫或者別外的Ceph 集群；
# 1. 創建一個快照
[root@ceph-node01 abc]# rbd snap create ceph-demo/rbd-test.img@snap-demo
# 2. 查看快照
[root@ceph-node01 abc]# rbd snap ls ceph-demo/rbd-test.img
SNAPID NAME SIZE PROTECTED TIMESTAMP
14 snap-demo 10 GiB Sat Oct 17 04:09:23 2020
[root@ceph-node01 abc]#
備份 （導出）
[root@ceph-node01 abc]# rbd export ceph-demo/rbd-test.img@snap-demo /root/rbd-test.img
Exporting image: 100% complete...done.
[root@ceph-node01 abc]# ls /root/rbd-test.img -lrth
-rw-r--r-- 1 root root 10G 10月 17 04:13 /root/rbd-test.img
[root@ceph-node01 abc]#
恢復（導入）
[root@ceph-node01 abc]# rbd import /root/rbd-test.img ceph-demo/rbd-test-new.img
Importing image: 100% complete...done.
[root@ceph-node01 abc]#
使用
# 1. 導入后，發現有些feature 功能加了進來，我們需要關掉
[root@ceph-node01 abc]# rbd device map ceph-demo/rbd-test-new.img
rbd: sysfs write failed
RBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
# 2. 查看info
[root@ceph-node01 abc]# rbd info ceph-demo/rbd-test-new.img
rbd image 'rbd-test-new.img':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 39594d499ff4f
block_name_prefix: rbd_data.39594d499ff4f
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Sat Oct 17 04:18:02 2020
access_timestamp: Sat Oct 17 04:18:02 2020
modify_timestamp: Sat Oct 17 04:18:02 2020
# 3. 關掉 feature
[root@ceph-node01 abc]# rbd feature disable ceph-demo/rbd-test-new.img exclusive-lock object-map fast-diff deep-flatten
[root@ceph-node01 abc]# rbd device map ceph-demo/rbd-test-new.img
/dev/rbd5
[root@ceph-node01 abc]# rbd device ls
id pool namespace image snap device
0 ceph-demo rbd-demo.img - /dev/rbd0
1 ceph-demo rbd-test.img - /dev/rbd1
2 ceph-demo vm1-clone.img - /dev/rbd2
3 ceph-demo vm1-clone.img - /dev/rbd3
4 ceph-demo vm2-clone.img - /dev/rbd4
5 ceph-demo rbd-test-new.img - /dev/rbd5
# 4. 掛載查看
[root@ceph-node01 abc]# mount /dev/rbd5 /data/abc
[root@ceph-node01 abc]# cd /data/abc/
[root@ceph-node01 abc]# cat file.log
2020年 10月 16日 星期五 23:58:43 EDT
[root@ceph-node01 abc]#
增量備份演示
# 1. 創建塊設備
[root@ceph-node01 ~]# rbd create ceph-demo/rbd-test-k8s.img --image-feature layering --size 10G
# 2. 映射塊設備使用
[root@ceph-node01 ~]# rbd device map ceph-demo/rbd-test-k8s.img
/dev/rbd5
# 3. 格式化
[root@ceph-node01 ~]# mkfs.ext4 /dev/rbd5
mke2fs 1.42.9 (28-Dec-2013)
。。。
Creating journal (32768 blocks): 完成
Writing superblocks and filesystem accounting information: 完成
# 4. 掛載
[root@ceph-node01 ~]# mount /dev/rbd5 /data/abe/
[root@ceph-node01 ~]# cd /data/abe/
[root@ceph-node01 abe]# ls
lost+found
[root@ceph-node01 abe]# echo `date` >> a
[root@ceph-node01 abe]# echo `date` >> aa
[root@ceph-node01 abe]# sync
[root@ceph-node01 abe]#
# 5. 創建快照
[root@ceph-node01 abe]# rbd snap create ceph-demo/rbd-test-k8s.img@v1
# 6. 備份
[root@ceph-node01 abe]# rbd export ceph-demo/rbd-test-k8s.img@v1 /root/rbd-test-k8s.img
Exporting image: 100% complete...done.
# 7. 增量寫入
[root@ceph-node01 abe]# ls
a aa lost+found
[root@ceph-node01 abe]# echo `date` >>a
[root@ceph-node01 abe]# echo `date` >>a
[root@ceph-node01 abe]# echo `date` >>a
[root@ceph-node01 abe]# echo `date` >>a
[root@ceph-node01 abe]# echo `date` >>aa
[root@ceph-node01 abe]# echo `date` >>aa
[root@ceph-node01 abe]# echo `date` >>bb
[root@ceph-node01 abe]# sync
# 8. 再次創建快照
[root@ceph-node01 abe]# rbd snap create ceph-demo/rbd-test-k8s.img@v2
# 9. 導出備份
[root@ceph-node01 abe]# rbd export-diff ceph-demo/rbd-test-k8s.img@v2 /root/rbd-test-k8s_v2.img
Exporting image: 100% complete...done.
[root@ceph-node01 abe]#
# 10. 刪除所有快照
[root@ceph-node01 ~]# rbd snap purge ceph-demo/rbd-test-k8s.img
Removing all snapshots: 100% complete...done.
[root@ceph-node01 ~]#
# 11. 取消設備映射
[root@ceph-node01 ~]# rbd device unmap ceph-demo/rbd-test-k8s.img
# 12. 刪除鏡像
[root@ceph-node01 ~]# rbd rm ceph-demo/rbd-test-k8s.img
Removing image: 100% complete...done.
[root@ceph-node01 ~]#
恢復
# 1. 備份恢復原始鏡像
[root@ceph-node01 ~]# rbd import /root/rbd-test-k8s.img ceph-demo/rbd-test-k8s.img
Importing image: 100% complete...done.
[root@ceph-node01 ~]#
# 2. 映射原始鏡像
[root@ceph-node01 ~]# rbd device map ceph-demo/rbd-test-k8s.img
rbd: sysfs write failed
RBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[root@ceph-node01 ~]# rbd info ceph-demo/rbd-test-k8s.img
rbd image 'rbd-test-k8s.img':
size 10 GiB in 2560 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 3988541adbc3b
block_name_prefix: rbd_data.3988541adbc3b
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Sat Oct 17 05:43:24 2020
access_timestamp: Sat Oct 17 05:43:24 2020
modify_timestamp: Sat Oct 17 05:43:24 2020
[root@ceph-node01 ~]#
[root@ceph-node01 ~]# rbd feature disable ceph-demo/rbd-test-k8s.img exclusive-lock object-map fast-diff deep-flatten
[root@ceph-node01 ~]#
[root@ceph-node01 ~]# rbd device map ceph-demo/rbd-test-k8s.img
/dev/rbd5
# 3. 掛載查看
[root@ceph-node01 ~]# mount /dev/rbd5 /data/abe/
[root@ceph-node01 ~]# cd /data/abe/
[root@ceph-node01 abe]# ls
a aa lost+found
[root@ceph-node01 abe]# cat a
2020年 10月 17日 星期六 05:24:57 EDT
[root@ceph-node01 abe]# cat aa
2020年 10月 17日 星期六 05:25:00 EDT
# 4. 導入增量備份
[root@ceph-node01 abe]# rbd import-diff /root/rbd-test-k8s_v2.img ceph-demo/rbd-test-k8s.img
Importing image diff: 100% complete...done.
[root@ceph-node01 abe]#
# 5. 重新掛載並驗證
[root@ceph-node01 ~]# umount /data/abe/
[root@ceph-node01 ~]# mount /dev/rbd5 /data/abe/
[root@ceph-node01 ~]# cd /data/abe/
[root@ceph-node01 abe]# ls
a aa bb lost+found
[root@ceph-node01 abe]# cat a
2020年 10月 17日 星期六 05:24:57 EDT
2020年 10月 17日 星期六 05:27:04 EDT
2020年 10月 17日 星期六 05:27:05 EDT
2020年 10月 17日 星期六 05:27:05 EDT
2020年 10月 17日 星期六 05:27:06 EDT
[root@ceph-node01 abe]# cat aa
2020年 10月 17日 星期六 05:25:00 EDT
2020年 10月 17日 星期六 05:27:12 EDT
2020年 10月 17日 星期六 05:27:13 EDT
[root@ceph-node01 abe]# cat bb
2020年 10月 17日 星期六 05:27:17 EDT
[root@ceph-node01 abe]#
清理主機映射
[root@ceph-node01 ~]# rbd device ls
id pool namespace image snap device
0 ceph-demo rbd-demo.img - /dev/rbd0
1 ceph-demo rbd-test.img - /dev/rbd1
2 ceph-demo vm1-clone.img - /dev/rbd2
3 ceph-demo vm1-clone.img - /dev/rbd3
4 ceph-demo vm2-clone.img - /dev/rbd4
5 ceph-demo rbd-test-new.img - /dev/rbd5
6 ceph-demo rbd-test-new2.img - /dev/rbd6
[root@ceph-node01 ~]# rbd device unmap ceph-demo/rbd-test-new2.img
[root@ceph-node01 ~]# rbd device unmap ceph-demo/rbd-test-new.img
總結
創建存儲池
# 1. 創建存儲池
[root@ceph-node01 ~]# ceph osd pool create kube 64 64
pool 'kube' created
# 2. 查看
[root@ceph-node01 ~]# ceph osd pool ls
。。。。
kube
# 3. 設置存儲池類型為rbd
[root@ceph-node01 ~]# ceph osd pool application enable kube rbd
enabled application 'rbd' on pool 'kube'
# 4. 初始化
[root@ceph-node01 ~]# rbd pool init kube
[root@ceph-node01 ~]#
創建rbd塊設備
[root@ceph-node01 ~]# rbd create -p kube k8s01 --size 2G
[root@ceph-node01 ~]# rbd create --pool kube --image k8s02 --size 2G
[root@ceph-node01 ~]# rbd create kube/k8s03 --size 2G
查看rbd信息
[root@ceph-node01 ~]# rbd -p kube ls
[root@ceph-node01 ~]# rbd -p kube ls -l
[root@ceph-node01 ~]# rbd info kube/k8s03
[root@ceph-node01 ~]# rbd -p kube ls -l --format json --pretty-format
rbd 特性
layering：磁盤分層技術；exclusive-lock：排它鎖，單獨讀寫操作；object-map：對象位圖功能；fast-diff：快照對比使用的；deep-flatten：多層快照之間分離時使用的一種特性；
[root@ceph-node01 ~]# rbd feature disable kube/k8s03 object-map fast-diff deep-flatten
客戶端配置(注意yum源)
[root@ceph-node01 ~]# yum -y install ceph-common
客戶端需要/etc/ceph/ceph.conf和密鑰環文件文件；
[root@ceph-node01 ceph-deploy]# ceph --user kube -s
客戶端使用rbd map，然后可以格式化並且mount掛載即可；
[root@ceph-node01 ceph-deploy]# rbd map kube/k8s01
/dev/rbd6
[root@ceph-node01 ceph-deploy]# rbd --user kube map kube/k8s02
/dev/rbd7
[root@ceph-node01 ceph-deploy]# lsblk
...
rbd6 251:96 0 2G 0 disk
rbd7 251:112 0 2G 0 disk
[root@ceph-node01 ceph-deploy]#
查看映射、刪除映射信息
# 1. 查看映射
[root@ceph-node01 ceph-deploy]# rbd showmapped
id pool namespace image snap device
....
6 kube k8s01 - /dev/rbd6
7 kube k8s02 - /dev/rbd7
# 2. 在集群上面查看哪些被掛載了，注意LOCK
[root@ceph-node01 ceph-deploy]# rbd ls -p kube -l
NAME SIZE PARENT FMT PROT LOCK
k8s01 2 GiB 2
k8s02 2 GiB 2
k8s03 2 GiB 2
[root@ceph-node01 ceph-deploy]#
# 3. 刪除映射關系
[root@ceph-node01 ceph-deploy]# rbd --user kube unmap kube/k8s02
[root@ceph-node01 ceph-deploy]# rbd showmapped
id pool namespace image snap device
。。。
6 kube k8s01 - /dev/rbd6
[root@ceph-node01 ceph-deploy]#
調整鏡像空間容量
[root@ceph-node01 ceph-deploy]# rbd resize -s 5G kube/k8s02
Resizing image: 100% complete...done.
[root@ceph-node01 ceph-deploy]# rbd ls -p kube -l
NAME SIZE PARENT FMT PROT LOCK
k8s01 2 GiB 2
k8s02 5 GiB 2
k8s03 2 GiB 2
[root@ceph-node01 ceph-deploy]#
刪除鏡像
[root@ceph-node01 ceph-deploy]# rbd rm kube/k8s03
Removing image: 100% complete...done.
[root@ceph-node01 ceph-deploy]# rbd ls -p kube -l
NAME SIZE PARENT FMT PROT LOCK
k8s01 2 GiB 2
k8s02 5 GiB 2
[root@ceph-node01 ceph-deploy]#
刪除到回收站並恢復
[root@ceph-node01 ceph-deploy]# rbd trash move kube/k8s01
[root@ceph-node01 ceph-deploy]# rbd trash list kube
3edbd828dee7c k8s01
[root@ceph-node01 ceph-deploy]# rbd -p kube ls -l
NAME SIZE PARENT FMT PROT LOCK
k8s02 5 GiB 2
[root@ceph-node01 ceph-deploy]# rbd trash restore -p kube 3edbd828dee7c
[root@ceph-node01 ceph-deploy]# rbd -p kube ls -l
NAME SIZE PARENT FMT PROT LOCK
k8s01 2 GiB 2
k8s02 5 GiB 2
[root@ceph-node01 ceph-deploy]#
快照總結
Ceph 服務端創建快照：rbd snap create kube/k8s01@k8s01snap01
Ceph 服務端查看快照：rbd snap list kube/k8s01
要想恢復快照，需要先把磁盤掛載刪除掉，綁定着的鏡像是不允許進行回滾的，並且需要把映射也需要刪除掉；
Ceph 客戶端：先 umount， 然后rbd unmap /dev/rbd0；
Ceph 服務端回滾快照：rbd snap rollback kube/k8s01@k8s01snap01
Ceph 客戶端再mount，查看數據，可以從快照中恢復；
Ceph 服務端刪除快照：rbd snap rm kube/k8s01@k8s01snap01
Ceph 服務端限制快照數量：rbd snap limit set kube/k8s01 --limit 10
Ceph 服務端清除快照限制：rbd snap limit clear kube/k8s01
克隆（多重快照）總結
對原始的磁盤鏡像數據做第一次快照，然后對快照置於保護模式下，此時即使你修改了原始磁盤鏡像中的數據，你也無法改變第一次快照的磁盤數據，因為他被設置為保護模式下了，接下來，我們基於第一次的快照，再進行一次快照，二次快照，這也就是克隆技術；
鏡像克隆的話，可以跨存儲池進行；
Ceph 服務端創建一個鏡像：略
Ceph 服務端創建一個快照：rbd snap create kube/k8s01@clone01
Ceph 服務端查看快照：rbd snap ls kube/k8s01
Ceph 服務端設置快照為保護模式： rbd snap protect kube/k8s01@clone01
Ceph 服務端基於保持的快照進行克隆鏡像：rbd clone kube/k8s01@clone01 kube/cloneimg01（注意可以跨存儲池）
Ceph 服務端查看鏡像：rbd -p kube ls -l
Ceph 客戶端直接映射：rbd --user kube map kube/cloneimg01
Ceph 客戶端直接掛載：略；
Ceph 服務端查看快照克隆了多少鏡像：rbd children kube/k8s01@clone01
Ceph 服務端刪除克隆的鏡像：rbd rm kube/cloneimg02
如果此時，我們想把原數據kube/k8s01刪除，此時應該如何做？刪除后，剛才克隆的兩個鏡像就無根了，因為兩個克隆鏡像的快照無根了，這個時候，就需要做展平；此時只需要指定把誰flatten（展平）操作即可，它的意思就是把引用別人的，都復制過來，不再依賴他人；
Ceph 服務端進行flatten操作：rbd flatten kube/cloneimg01，此時它已經不需要依賴任何人了；
Ceph 服務端可以把快照取消保護：rbd snap unprotect kube/k8s01@clone01，此時快照刪除也沒有關系，因為其它人也不依賴你了；
Ceph 服務端刪除快照：rbd snap rm kube/k8s01@clone01
Ceph 服務端可以導入外部image：rbd import cenots.img kube/centos7