使用mdadm工具實現軟RAID 5實戰案例
作者：尹正傑
版權聲明：原創作品，謝絕轉載！否則將追究法律責任。
一.准備工作
1>.實驗環境說明
本次試驗我打算采用4個分區設備來做RAID 5，如下圖所示。
一般情況下使用3塊設備就可以做RAID 5,但我之所以使用4塊設備，是想用第4塊設備充當空閑硬盤，當RAID 5中某一塊設備損壞時，它會立即頂上去。 溫馨提示： 由於我們使用的是分區來模擬RAID實戰(即軟RAID)，因此在分區時務必指定Id類型為"fd"(即"Linux raid auto") 生產環境中我們很少用磁盤分區來做RAID，而是直接使用整塊磁盤來做RAID陣列，而且會有專門的磁盤陣列卡。
2>.為/dev/sdb磁盤新增一個200G的分區
[root@yinzhengjie.com ~]# fdisk /dev/sdb Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): p Disk /dev/sdb: 644.2 GB, 644245094400 bytes, 1258291200 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x8c3b1f6e Device Boot Start End Blocks Id System /dev/sdb1 2048 209717247 104857600 fd Linux raid autodetect Command (m for help): n Partition type: p primary (1 primary, 0 extended, 3 free) e extended Select (default p): p Partition number (2-4, default 2): First sector (209717248-1258291199, default 209717248): Using default value 209717248 Last sector, +sectors or +size{K,M,G} (209717248-1258291199, default 1258291199): +200G Partition 2 of type Linux and of size 200 GiB is set Command (m for help): p Disk /dev/sdb: 644.2 GB, 644245094400 bytes, 1258291200 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x8c3b1f6e Device Boot Start End Blocks Id System /dev/sdb1 2048 209717247 104857600 fd Linux raid autodetect /dev/sdb2 209717248 629147647 209715200 83 Linux Command (m for help): t Partition number (1,2, default 2): 2 Hex code (type L to list all codes): L 0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris 1 FAT12 27 Hidden NTFS Win 82 Linux swap / So c1 DRDOS/sec (FAT- 2 XENIX root 39 Plan 9 83 Linux c4 DRDOS/sec (FAT- 3 XENIX usr 3c PartitionMagic 84 OS/2 hidden C: c6 DRDOS/sec (FAT- 4 FAT16 <32M 40 Venix 80286 85 Linux extended c7 Syrinx 5 Extended 41 PPC PReP Boot 86 NTFS volume set da Non-FS data 6 FAT16 42 SFS 87 NTFS volume set db CP/M / CTOS / . 7 HPFS/NTFS/exFAT 4d QNX4.x 88 Linux plaintext de Dell Utility 8 AIX 4e QNX4.x 2nd part 8e Linux LVM df BootIt 9 AIX bootable 4f QNX4.x 3rd part 93 Amoeba e1 DOS access a OS/2 Boot Manag 50 OnTrack DM 94 Amoeba BBT e3 DOS R/O b W95 FAT32 51 OnTrack DM6 Aux 9f BSD/OS e4 SpeedStor c W95 FAT32 (LBA) 52 CP/M a0 IBM Thinkpad hi eb BeOS fs e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a5 FreeBSD ee GPT f W95 Ext'd (LBA) 54 OnTrackDM6 a6 OpenBSD ef EFI (FAT-12/16/ 10 OPUS 55 EZ-Drive a7 NeXTSTEP f0 Linux/PA-RISC b 11 Hidden FAT12 56 Golden Bow a8 Darwin UFS f1 SpeedStor 12 Compaq diagnost 5c Priam Edisk a9 NetBSD f4 SpeedStor 14 Hidden FAT16 <3 61 SpeedStor ab Darwin boot f2 DOS secondary 16 Hidden FAT16 63 GNU HURD or Sys af HFS / HFS+ fb VMware VMFS 17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fc VMware VMKCORE 18 AST SmartSleep 65 Novell Netware b8 BSDI swap fd Linux raid auto 1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid fe LANstep 1c Hidden W95 FAT3 75 PC/IX be Solaris boot ff BBT 1e Hidden W95 FAT1 80 Old Minix Hex code (type L to list all codes): fd Changed type of partition 'Linux' to 'Linux raid autodetect' Command (m for help): p Disk /dev/sdb: 644.2 GB, 644245094400 bytes, 1258291200 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x8c3b1f6e Device Boot Start End Blocks Id System /dev/sdb1 2048 209717247 104857600 fd Linux raid autodetect /dev/sdb2 209717248 629147647 209715200 fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) Syncing disks. [root@yinzhengjie.com ~]#
3>.為/dev/sdc磁盤新增一個200G的分區
[root@yinzhengjie.com ~]# fdisk /dev/sdc Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): p Disk /dev/sdc: 429.5 GB, 429496729600 bytes, 838860800 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x8c3b1f6e Device Boot Start End Blocks Id System /dev/sdc1 2048 209717247 104857600 fd Linux raid autodetect Command (m for help): n Partition type: p primary (1 primary, 0 extended, 3 free) e extended Select (default p): Using default response p Partition number (2-4, default 2): First sector (209717248-838860799, default 209717248): Using default value 209717248 Last sector, +sectors or +size{K,M,G} (209717248-838860799, default 838860799): +200G Partition 2 of type Linux and of size 200 GiB is set Command (m for help): t Partition number (1,2, default 2): Hex code (type L to list all codes): fd Changed type of partition 'Linux' to 'Linux raid autodetect' Command (m for help): p Disk /dev/sdc: 429.5 GB, 429496729600 bytes, 838860800 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x8c3b1f6e Device Boot Start End Blocks Id System /dev/sdc1 2048 209717247 104857600 fd Linux raid autodetect /dev/sdc2 209717248 629147647 209715200 fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) Syncing disks. [root@yinzhengjie.com ~]#
4>.為/dev/sdd磁盤新增一個200G的分區
[root@yinzhengjie.com ~]# fdisk /dev/sdd Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0xed751bb5. Command (m for help): p Disk /dev/sdd: 859.0 GB, 858993459200 bytes, 1677721600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0xed751bb5 Device Boot Start End Blocks Id System Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): Using default response p Partition number (1-4, default 1): First sector (2048-1677721599, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-1677721599, default 1677721599): +200G Partition 1 of type Linux and of size 200 GiB is set Command (m for help): t Selected partition 1 Hex code (type L to list all codes): fd Changed type of partition 'Linux' to 'Linux raid autodetect' Command (m for help): p Disk /dev/sdd: 859.0 GB, 858993459200 bytes, 1677721600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0xed751bb5 Device Boot Start End Blocks Id System /dev/sdd1 2048 419432447 209715200 fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@yinzhengjie.com ~]#
5>.為/dev/sde磁盤新增一個200G的分區
[root@yinzhengjie.com ~]# fdisk /dev/sde Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x487f53ac. Command (m for help): p Disk /dev/sde: 1073.7 GB, 1073741824000 bytes, 2097152000 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x487f53ac Device Boot Start End Blocks Id System Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): Using default response p Partition number (1-4, default 1): First sector (2048-2097151999, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-2097151999, default 2097151999): +200G Partition 1 of type Linux and of size 200 GiB is set Command (m for help): P Disk /dev/sde: 1073.7 GB, 1073741824000 bytes, 2097152000 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x487f53ac Device Boot Start End Blocks Id System /dev/sde1 2048 419432447 209715200 83 Linux Command (m for help): t Selected partition 1 Hex code (type L to list all codes): fd Changed type of partition 'Linux' to 'Linux raid autodetect' Command (m for help): p Disk /dev/sde: 1073.7 GB, 1073741824000 bytes, 2097152000 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x487f53ac Device Boot Start End Blocks Id System /dev/sde1 2048 419432447 209715200 fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@yinzhengjie.com ~]#
二.創建RAID 5塊設備實戰案例
1>.創建一個RAID 5設備
[root@yinzhengjie.com ~]# cat /proc/mdstat Personalities : [raid0] md1 : active raid0 sdc1[1] sdb1[0] 209582080 blocks super 1.2 512k chunks unused devices: <none> [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mdadm -C -a yes /dev/md2 -l 5 -n 3 -x 1 -c 1M /dev/sd{b2,c2,d1,e1} #"-C -a yes"表示創建一個RAID設備，名稱為"/dev/md2"。"-x"表示備用的設備，"-c"表示組塊的大小為1M，默認最后一塊設備(/dev/sde1)為備用塊設備。 mdadm: /dev/sdb2 appears to contain an ext2fs file system size=209715200K mtime=Thu Jan 1 08:00:00 1970 Continue creating array? yes mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md2 started. [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# cat /proc/mdstat #當創建RAID設備時，由於塊設備較大，因此需要等待一段時間，此時我們就可以看到創建過程的進度，如下所示，可以看到當前的進度時6.2%。 Personalities : [raid0] [raid6] [raid5] [raid4] md2 : active raid5 sdd1[4] sde1[3](S) sdc2[1] sdb2[0] 419166208 blocks super 1.2 level 5, 1024k chunk, algorithm 2 [3/2] [UU_] [=>...................] recovery = 6.2% (13109760/209583104) finish=7.2min speed=452060K/sec bitmap: 0/2 pages [0KB], 65536KB chunk md1 : active raid0 sdc1[1] sdb1[0] 209582080 blocks super 1.2 512k chunks unused devices: <none> [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# cat /proc/mdstat #目前已經創建完畢啦~之前的進度信息也沒啦！ Personalities : [raid0] [raid6] [raid5] [raid4] md2 : active raid5 sdd1[4] sde1[3](S) sdc2[1] sdb2[0] 419166208 blocks super 1.2 level 5, 1024k chunk, algorithm 2 [3/3] [UUU] bitmap: 0/2 pages [0KB], 65536KB chunk md1 : active raid0 sdc1[1] sdb1[0] 209582080 blocks super 1.2 512k chunks unused devices: <none> [root@yinzhengjie.com ~]#
2>.查看創建設備的詳細信息
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 03:35:01 2020 State : clean Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 93 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 1 8 34 1 active sync /dev/sdc2 4 8 49 2 active sync /dev/sdd1 3 8 65 - spare /dev/sde1 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]#
3>.對RAID 5設備進行格式化，指定為XFS文件系統
[root@yinzhengjie.com ~]# blkid /dev/sdc1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="78244c8b-f961-577f-9d32-af6c474e9f2b" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member" /dev/sdc2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="142f3a98-d654-1abd-00bd-a50a66fec7fc" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member" /dev/sdd1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="f1699604-e5f0-f054-31f6-a56f90d452e3" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member" /dev/sdb1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="595fb659-a792-aaac-3a2d-800b3e0795ac" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member" /dev/sdb2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="5a6802d5-93af-680d-dca1-64d02c5b615b" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member" /dev/sde1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="e6b567f9-0b34-3acd-6d7d-39540d51878f" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member" /dev/sda1: PARTUUID="8f500d9c-87b9-421d-8ce0-8c385b73bf08" /dev/sda2: UUID="1865a93f-6113-4097-89dc-8c4ea5fdf68c" TYPE="xfs" PARTUUID="e85154fa-bbc8-43c1-90f0-676000195a97" /dev/sda3: UUID="lNmzXe-Whhd-NTtI-C55A-7m4x-CudG-1Q556q" TYPE="LVM2_member" PARTUUID="fa2e251a-d220-415f-acc7-da94b7a2d1b4" /dev/sda4: PARTUUID="814caa31-3a7e-499d-8b7d-cc3017473ba8" /dev/sr0: UUID="2018-11-25-21-21-31-00" LABEL="CentOS 7 x86_64" TYPE="iso9660" PTTYPE="dos" /dev/mapper/centos-root: UUID="c0160ecb-06e4-497e-813c-53690b4a6109" TYPE="xfs" /dev/md1: UUID="bc7d4ffa-439e-4544-b59e-58d10228569c" TYPE="xfs" /dev/mapper/centos-yinzhengjie: UUID="6a2643fb-8d55-44cd-b41e-b5f2709ef855" TYPE="xfs" [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mkfs.xfs /dev/md2 #格式化RAID 5設備，將其格式化為XFS文件系統。 meta-data=/dev/md2 isize=512 agcount=16, agsize=6549248 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=104787968, imaxpct=25 = sunit=256 swidth=512 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=51168, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# blkid /dev/sdc1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="78244c8b-f961-577f-9d32-af6c474e9f2b" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member" /dev/sdc2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="142f3a98-d654-1abd-00bd-a50a66fec7fc" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member" /dev/sdd1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="f1699604-e5f0-f054-31f6-a56f90d452e3" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member" /dev/sdb1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="595fb659-a792-aaac-3a2d-800b3e0795ac" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member" /dev/sdb2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="5a6802d5-93af-680d-dca1-64d02c5b615b" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member" /dev/sde1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="e6b567f9-0b34-3acd-6d7d-39540d51878f" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member" /dev/sda2: UUID="1865a93f-6113-4097-89dc-8c4ea5fdf68c" TYPE="xfs" PARTUUID="e85154fa-bbc8-43c1-90f0-676000195a97" /dev/sda3: UUID="lNmzXe-Whhd-NTtI-C55A-7m4x-CudG-1Q556q" TYPE="LVM2_member" PARTUUID="fa2e251a-d220-415f-acc7-da94b7a2d1b4" /dev/sr0: UUID="2018-11-25-21-21-31-00" LABEL="CentOS 7 x86_64" TYPE="iso9660" PTTYPE="dos" /dev/mapper/centos-root: UUID="c0160ecb-06e4-497e-813c-53690b4a6109" TYPE="xfs" /dev/md1: UUID="bc7d4ffa-439e-4544-b59e-58d10228569c" TYPE="xfs" /dev/mapper/centos-yinzhengjie: UUID="6a2643fb-8d55-44cd-b41e-b5f2709ef855" TYPE="xfs" /dev/sda1: PARTUUID="8f500d9c-87b9-421d-8ce0-8c385b73bf08" /dev/sda4: PARTUUID="814caa31-3a7e-499d-8b7d-cc3017473ba8" /dev/md2: UUID="eb94af80-9d27-4c8a-9c7b-1f54bb40048f" TYPE="xfs" [root@yinzhengjie.com ~]#
4>.掛載設備
[root@yinzhengjie.com ~]# vim /etc/fstab [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mkdir /yinzhengjie/raid5 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# tail -1 /etc/fstab UUID=eb94af80-9d27-4c8a-9c7b-1f54bb40048f /yinzhengjie/raid5 xfs defaults,noatime,nodiratime 0 0 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mount -a [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 500G 14G 486G 3% / devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 2.0G 12M 2.0G 1% /run tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup /dev/sda2 1014M 133M 882M 14% /boot /dev/mapper/centos-yinzhengjie 1.6T 2.4G 1.6T 1% /yinzhengjie /dev/md1 200G 1.1G 199G 1% /yinzhengjie/raid tmpfs 394M 0 394M 0% /run/user/0 /dev/md2 400G 33M 400G 1% /yinzhengjie/raid5 #為什么4塊200GB的設備，但其大小僅有400G呢？ [root@yinzhengjie.com ~]#
5>.測試RAID 5設備的I/O性能
[root@yinzhengjie.com ~]# free -h total used free shared buff/cache available Mem: 3.8G 259M 3.4G 11M 143M 3.4G Swap: 0B 0B 0B [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# dd if=/dev/zero of=/yinzhengjie/raid5/test.log bs=1M count=1024 #測試RAID 5設備的寫性能 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 1.86494 s, 576 MB/s [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# free -h total used free shared buff/cache available Mem: 3.8G 263M 2.4G 11M 1.2G 3.3G Swap: 0B 0B 0B [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# sync; echo 1 > /proc/sys/vm/drop_caches [root@yinzhengjie.com ~]# sync; echo 2 > /proc/sys/vm/drop_caches [root@yinzhengjie.com ~]# sync; echo 3 > /proc/sys/vm/drop_caches [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# free -h total used free shared buff/cache available Mem: 3.8G 261M 3.4G 11M 143M 3.4G Swap: 0B 0B 0B [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# dd if=/yinzhengjie/raid5/test.log of=/dev/null #測試RAID 5設備的讀性能 2097152+0 records in 2097152+0 records out 1073741824 bytes (1.1 GB) copied, 2.85041 s, 377 MB/s [root@yinzhengjie.com ~]#
三.測試RAID 5設備
1>.備份mdadm的配置信息
[root@yinzhengjie.com ~]# mdadm -Ds ARRAY /dev/md/001 metadata=1.2 name=yinzhengjie.com:001 UUID=e7e1a4a7:f73b2990:a52b16b3:92be0561 ARRAY /dev/md2 metadata=1.2 spares=1 name=yinzhengjie.com:2 UUID=7bdb7eab:498a99bb:f69ee4ce:7d5c010f [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mdadm -Ds >/etc/mdadm.conf #將mdadm管理的RAID設備信息導出到一個配置文件中，如果人為損壞設備恢復時可以用到對應的UUID。 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# cat /etc/mdadm.conf ARRAY /dev/md/001 metadata=1.2 name=yinzhengjie.com:001 UUID=e7e1a4a7:f73b2990:a52b16b3:92be0561 ARRAY /dev/md2 metadata=1.2 spares=1 name=yinzhengjie.com:2 UUID=7bdb7eab:498a99bb:f69ee4ce:7d5c010f [root@yinzhengjie.com ~]#
2>.手動將RAID 5中的一塊設備下線，觀察備用設備是否會自動構建
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #觀察操作之前的RAID 5設備信息 /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 05:40:34 2020 State : clean Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 93 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 1 8 34 1 active sync /dev/sdc2 4 8 49 2 active sync /dev/sdd1 3 8 65 - spare /dev/sde1 [root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm /dev/md2 -f /dev/sdc2 #手動下線"/dev/sdc2"這塊設備，請仔細觀察"RAID 5"設備的信息，你會發現空閑設備自動就頂替上來啦，如下圖所示。 mdadm: set /dev/sdc2 faulty in /dev/md2 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mdadm -D /dev/md2 /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 05:51:07 2020 State : clean, degraded, recovering Active Devices : 2 Working Devices : 3 Failed Devices : 1 Spare Devices : 1 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Rebuild Status : 0% complete Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 95 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 3 8 65 1 spare rebuilding /dev/sde1 4 8 49 2 active sync /dev/sdd1 1 8 34 - faulty /dev/sdc2 [root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #當備用(Spare)設備替換了"faulty"狀態的設備后，其狀態"active"，中間狀態為"spare rebuilding"(我測試環境的200G磁盤，需要5分鍾左右才完成) /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 05:59:00 2020 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 1 Spare Devices : 0 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 191 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 3 8 65 1 active sync /dev/sde1 4 8 49 2 active sync /dev/sdd1 1 8 34 - faulty /dev/sdc2 [root@yinzhengjie.com ~]#
3>.刪除有錯誤的(faulty)設備
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 05:59:00 2020 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 1 Spare Devices : 0 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 191 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 3 8 65 1 active sync /dev/sde1 4 8 49 2 active sync /dev/sdd1 1 8 34 - faulty /dev/sdc2 [root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm /dev/md2 -r /dev/sdc2 #刪除有錯誤的設備 mdadm: hot removed /dev/sdc2 from /dev/md2 [root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 3 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 06:07:15 2020 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 192 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 3 8 65 1 active sync /dev/sde1 4 8 49 2 active sync /dev/sdd1 [root@yinzhengjie.com ~]#
4>.往磁盤陣列中添加設備
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 3 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 06:07:15 2020 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 192 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 3 8 65 1 active sync /dev/sde1 4 8 49 2 active sync /dev/sdd1 [root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm /dev/md2 -a /dev/sdc2 #我們將設備添加到RAID 5磁盤陣列中 mdadm: re-added /dev/sdc2 [root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 06:10:38 2020 State : clean Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 193 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 3 8 65 1 active sync /dev/sde1 4 8 49 2 active sync /dev/sdd1 1 8 34 - spare /dev/sdc2 [root@yinzhengjie.com ~]#
5>.驗證數據是否存在
[root@yinzhengjie.com ~]# ll -h /yinzhengjie/raid5/ #當磁盤損壞時，我們發現文件依舊是可以正常訪問的，原因是我們有備用的磁盤設備。 total 1.0G -rw-r--r-- 1 root root 1.0G Aug 21 05:39 test.log [root@yinzhengjie.com ~]#
6>.禁用RAID設備
[root@yinzhengjie.com ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 500G 14G 486G 3% / devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 2.0G 12M 2.0G 1% /run tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup /dev/sda2 1014M 133M 882M 14% /boot /dev/mapper/centos-yinzhengjie 1.6T 3.4G 1.6T 1% /yinzhengjie /dev/md1 200G 1.1G 199G 1% /yinzhengjie/raid tmpfs 394M 0 394M 0% /run/user/0 /dev/md2 400G 1.1G 399G 1% /yinzhengjie/raid5 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# umount /yinzhengjie/raid5/ #禁用RAID陣列之前，先要卸載對應的掛載點 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 500G 14G 486G 3% / devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 2.0G 12M 2.0G 1% /run tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup /dev/sda2 1014M 133M 882M 14% /boot /dev/mapper/centos-yinzhengjie 1.6T 3.4G 1.6T 1% /yinzhengjie /dev/md1 200G 1.1G 199G 1% /yinzhengjie/raid tmpfs 394M 0 394M 0% /run/user/0 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #查看磁盤陣列信息 /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 06:25:31 2020 State : clean Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 193 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 3 8 65 1 active sync /dev/sde1 4 8 49 2 active sync /dev/sdd1 1 8 34 - spare /dev/sdc2 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mdadm -S /dev/md2 #禁用磁盤戰列 mdadm: stopped /dev/md2 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mdadm -D /dev/md2 #禁用后將無法查看磁盤陣列信息啦~ mdadm: cannot open /dev/md2: No such file or directory [root@yinzhengjie.com ~]#
7>.啟用RAID設備
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 #未啟動磁盤陣列之前是無法查看相關信息的 mdadm: cannot open /dev/md2: No such file or directory [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mdadm -A /dev/md2 #啟動磁盤陣列 mdadm: /dev/md2 has been started with 3 drives and 1 spare. [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]# mdadm -D /dev/md2 #啟用磁盤陣列后就可以查看該陣列的相關信息啦~ /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 06:25:31 2020 State : clean Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 193 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 3 8 65 1 active sync /dev/sde1 4 8 49 2 active sync /dev/sdd1 1 8 34 - spare /dev/sdc2 [root@yinzhengjie.com ~]# [root@yinzhengjie.com ~]#
8>.清除磁盤的RAID信息
[root@yinzhengjie.com ~]# mdadm -D /dev/md2 /dev/md2: Version : 1.2 Creation Time : Fri Aug 21 03:27:31 2020 Raid Level : raid5 Array Size : 419166208 (399.75 GiB 429.23 GB) Used Dev Size : 209583104 (199.87 GiB 214.61 GB) Raid Devices : 3 Total Devices : 4 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Aug 21 06:25:31 2020 State : clean Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 1024K Consistency Policy : bitmap Name : yinzhengjie.com:2 (local to host yinzhengjie.com) UUID : 7bdb7eab:498a99bb:f69ee4ce:7d5c010f Events : 193 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 3 8 65 1 active sync /dev/sde1 4 8 49 2 active sync /dev/sdd1 1 8 34 - spare /dev/sdc2 [root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm -S /dev/md2 #禁用磁盤陣列 mdadm: stopped /dev/md2 [root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# blkid /dev/sd{b,c,d,e}* #查看"/dev/md2"磁盤陣列對應的分區文件，注意觀察對應的UUID
/dev/sdb: PTTYPE="dos"
/dev/sdb1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="595fb659-a792-aaac-3a2d-800b3e0795ac" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdb2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="5a6802d5-93af-680d-dca1-64d02c5b615b" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sdc: PTTYPE="dos"
/dev/sdc1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="78244c8b-f961-577f-9d32-af6c474e9f2b" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdc2: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="142f3a98-d654-1abd-00bd-a50a66fec7fc" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sdd: PTTYPE="dos"
/dev/sdd1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="f1699604-e5f0-f054-31f6-a56f90d452e3" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
/dev/sde: PTTYPE="dos"
/dev/sde1: UUID="7bdb7eab-498a-99bb-f69e-e4ce7d5c010f" UUID_SUB="e6b567f9-0b34-3acd-6d7d-39540d51878f" LABEL="yinzhengjie.com:2" TYPE="linux_raid_member"
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# mdadm --zero-superblock /dev/sdb2 /dev/sdc2 /dev/sdd1 /dev/sde1 #我們將"/dev/md2"對應的RAID信息進行清空操作
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]# blkid /dev/sd{b,c,d,e}* #再次觀察各個分區對應的UUID，發現"/dev/md2"的UUID被成功刪除啦~
/dev/sdb: PTTYPE="dos"
/dev/sdb1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="595fb659-a792-aaac-3a2d-800b3e0795ac" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdb2: LABEL="/mnt/sdb2" UUID="15028503-abd5-4bc7-9e35-2e745662d42d" TYPE="ext4"
/dev/sdc: PTTYPE="dos"
/dev/sdc1: UUID="e7e1a4a7-f73b-2990-a52b-16b392be0561" UUID_SUB="78244c8b-f961-577f-9d32-af6c474e9f2b" LABEL="yinzhengjie.com:001" TYPE="linux_raid_member"
/dev/sdc2: UUID="2307c9c5-f7b1-4a2b-b9e9-46d89a9c7c81" TYPE="xfs"
/dev/sdd: PTTYPE="dos"
/dev/sde: PTTYPE="dos"
[root@yinzhengjie.com ~]#
[root@yinzhengjie.com ~]#