multipath多路徑實驗02-配置多路徑軟件
在上一篇文章《multipath多路徑實驗01-構建iSCSI模擬環境》，我構建了iSCSI的模擬環境，在文章最后，已經成功配置並在主機上認出多個鏈路分別映射出的盤。而本文我就要對這樣的多路徑環境，通過Linux自帶multipath多路徑軟件聚合成可以方便我們后期使用的形式。
1.安裝多路徑軟件包
查詢是否安裝了多路徑軟件包：rpm -qa |grep device-mapper-multipath
如果沒有安裝，則用yum安裝即可：
yum install device-mapper-multipath
[root@jyrac1 ~]# yum install device-mapper-multipath
Loaded plugins: product-id, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package device-mapper-multipath.x86_64 0:0.4.9-72.el6 will be installed
--> Processing Dependency: device-mapper-multipath-libs = 0.4.9-72.el6 for package: device-mapper-multipath-0.4.9-72.el6.x86_64
--> Processing Dependency: libmultipath.so()(64bit) for package: device-mapper-multipath-0.4.9-72.el6.x86_64
--> Processing Dependency: libmpathpersist.so.0()(64bit) for package: device-mapper-multipath-0.4.9-72.el6.x86_64
--> Running transaction check
---> Package device-mapper-multipath-libs.x86_64 0:0.4.9-72.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==================================================================================================================
Package Arch Version Repository Size
==================================================================================================================
Installing:
device-mapper-multipath x86_64 0.4.9-72.el6 ISO 116 k
Installing for dependencies:
device-mapper-multipath-libs x86_64 0.4.9-72.el6 ISO 180 k
Transaction Summary
==================================================================================================================
Install 2 Package(s)
Total download size: 297 k
Installed size: 653 k
Is this ok [y/N]: y
Downloading Packages:
------------------------------------------------------------------------------------------------------------------
Total 2.8 MB/s | 297 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : device-mapper-multipath-libs-0.4.9-72.el6.x86_64 1/2
Installing : device-mapper-multipath-0.4.9-72.el6.x86_64 2/2
Verifying : device-mapper-multipath-libs-0.4.9-72.el6.x86_64 1/2
Verifying : device-mapper-multipath-0.4.9-72.el6.x86_64 2/2
Installed:
device-mapper-multipath.x86_64 0:0.4.9-72.el6
Dependency Installed:
device-mapper-multipath-libs.x86_64 0:0.4.9-72.el6
Complete!
如果不方便配置yum，也可以直接rpm安裝這兩個包：
[root@jyrac2 Packages]# rpm -ivh device-mapper-multipath-libs-0.4.9-72.el6.x86_64.rpm
warning: device-mapper-multipath-libs-0.4.9-72.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing... ########################################### [100%]
1:device-mapper-multipath########################################### [100%]
[root@jyrac2 Packages]# rpm -ivh device-mapper-multipath-0.4.9-72.el6.x86_64.rpm
warning: device-mapper-multipath-0.4.9-72.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing... ########################################### [100%]
1:device-mapper-multipath########################################### [100%]
2.設置服務開機啟動
查詢當前開機自啟用設置：chkconfig --list|grep multipathd
確認內核成功加載：
modprobe -l |grep multipath
確認開啟開機自啟動：
[root@jyrac1 ~]# chkconfig multipathd on
[root@jyrac1 ~]# chkconfig --list|grep multipathd
multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
3.生成multipath配置文件
生成multipath配置文件:
/sbin/mpathconf --enable
確認服務狀態：
service multipathd status
4.multipath的常用命令
常用命令：--生成multipath配置文件
/sbin/mpathconf --enable
--顯示多路徑的布局
multipath -ll
--重新刷取
multipath -v2
--清空所有多路徑
multipath -F
針對上述常用命令，實際操作的記錄，供參考：
[root@jyrac1 ~]# multipath -ll
Jul 18 15:57:18 | DM multipath kernel driver not loaded
Jul 18 15:57:18 | /etc/multipath.conf does not exist, blacklisting all devices.
Jul 18 15:57:18 | A sample multipath.conf file is located at
Jul 18 15:57:18 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Jul 18 15:57:18 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
Jul 18 15:57:18 | DM multipath kernel driver not loaded
[root@jyrac1 ~]# chkconfig --list|grep multipathd
multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@jyrac1 ~]# modprobe -l |grep multipath
kernel/drivers/md/dm-multipath.ko
[root@jyrac1 ~]# service multipathd status
multipathd is stopped
[root@jyrac1 ~]# service multipathd start
Starting multipathd daemon: [ OK ]
[root@jyrac1 ~]# multipath -ll
Jul 18 15:59:00 | /etc/multipath.conf does not exist, blacklisting all devices.
Jul 18 15:59:00 | A sample multipath.conf file is located at
Jul 18 15:59:00 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Jul 18 15:59:00 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
[root@jyrac1 ~]# /sbin/mpathconf --enable
[root@jyrac1 ~]# multipath -v2
Jul 18 16:02:52 | mpatha: ignoring map
Jul 18 16:02:52 | mpathb: ignoring map
Jul 18 16:02:52 | mpathc: ignoring map
Jul 18 16:02:52 | mpathd: ignoring map
Jul 18 16:02:52 | mpathe: ignoring map
Jul 18 16:02:52 | mpathf: ignoring map
create: mpathg (1IET 00010001) undef IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=undef
|-+- policy='round-robin 0' prio=1 status=undef
| `- 10:0:0:1 sdi 8:128 undef ready running
`-+- policy='round-robin 0' prio=1 status=undef
`- 11:0:0:1 sdj 8:144 undef ready running
[root@jyrac1 ~]# multipath -ll
mpathg (1IET 00010001) dm-2 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 10:0:0:1 sdi 8:128 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 11:0:0:1 sdj 8:144 active ready running
[root@jyrac1 ~]#
這里測試沒有問題后，為了后續方便測試ASM磁盤組，在iSCSI服務端又分了一個10g的lv，模擬存儲端分出的一個LUN，最后在這邊就應該有兩個多路徑聚合盤了。同樣的方法在RAC的另一個節點也都配置好。
5.udev綁定權限
我這里系統是RHEL6.5，對於multipath的權限，手工去修改幾秒后會變回root。所以需要使用udev去綁定好權限。 搜索對應的配置文件模板：[root@jyrac1 ~]# find / -name 12-*
/usr/share/doc/device-mapper-1.02.79/12-dm-permissions.rules
根據模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面：
vi /etc/udev/rules.d/12-dm-permissions.rules
# MULTIPATH DEVICES
#
# Set permissions for all multipath devices
ENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
# Set permissions for first two partitions created on a multipath device (and detected by kpartx)
# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"
查看多路徑對應的底層dm設備：
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw---- 1 root disk 253, 0 Jul 19 16:40 /dev/dm-0
brw-rw---- 1 root disk 253, 1 Jul 19 16:40 /dev/dm-1
brw-rw---- 1 root disk 253, 2 Jul 19 21:20 /dev/dm-2
brw-rw---- 1 root disk 253, 3 Jul 19 21:20 /dev/dm-3
--啟動start_udev
[root@jyrac2 rules.d]# start_udev
Starting udev: [ OK ]
--查看權限：
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0
brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1
brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2
brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3
--等30s后再查，權限固定：
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0
brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1
brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2
brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3
發現權限固定不變沒有問題了，在RAC的另一個節點也同樣設置。
這時多路徑的信息再查看下：
[root@jyrac1 ~]# multipath -ll
mpathi (1IET 00010002) dm-2 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:2 sdk 8:160 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 12:0:0:2 sdl 8:176 active ready running
mpathg (1IET 00010001) dm-3 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:1 sdi 8:128 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 12:0:0:1 sdj 8:144 active ready running
[root@jyrac1 ~]#
在另一個節點也要達到同樣效果。
6.新建ASM磁盤組測試
使用grid用戶登錄xmanager圖形界面調出asmca，創建一個新的磁盤組data2，使用這兩個盤。冗余度選擇Normal冗余： 最終加磁盤組成功，對應查詢磁盤組和磁盤的相關信息如下：SQL> select group_number, disk_number, name, path, mode_status, voting_file from v$asm_disk order by 1,2;
GROUP_NUMBER DISK_NUMBER NAME PATH MODE_ST V
------------ ----------- ------------------------------ -------------------------------------------------- ------- -
1 0 DATA1_0000 /dev/asm-diske ONLINE N
1 1 DATA1_0001 /dev/asm-diskf ONLINE N
1 2 DATA1_0002 /dev/asm-diskg ONLINE N
2 0 FRA1_0000 /dev/asm-diskh ONLINE N
3 0 OCR1_0000 /dev/asm-diskb ONLINE Y
3 1 OCR1_0001 /dev/asm-diskc ONLINE Y
3 2 OCR1_0002 /dev/asm-diskd ONLINE Y
4 0 DATA2_0000 /dev/mapper/mpathg ONLINE N
4 1 DATA2_0001 /dev/mapper/mpathi ONLINE N
9 rows selected.
SQL> select group_number, name, total_mb, free_mb, usable_file_mb, offline_disks, state, type from v$asm_diskgroup;
GROUP_NUMBER NAME TOTAL_MB FREE_MB USABLE_FILE_MB OFFLINE_DISKS STATE TYPE
------------ ------------------------------ ---------- ---------- -------------- ------------- ----------- ------
1 DATA1 15360 13204 13204 0 CONNECTED EXTERN
2 FRA1 5120 3773 3773 0 CONNECTED EXTERN
3 OCR1 3072 2146 561 0 MOUNTED NORMAL
4 DATA2 20480 20290 10145 0 MOUNTED NORMAL
至此，有關Linux自帶的multipath多路徑軟件相關的實驗已經全部完成。