CentOS配置multipath

可以通過2種方式查看HBA的WWN信息：

1. 查看sys文件系統

查看HBA卡型號：
[root@localhost ~]# lspci  | grep -i fibre
13:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
13:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
查看HBA卡WWN信息：
[root@localhost ~]# ls -l /sys/class/fc_host/
total 0
drwxr-xr-x 3 root root 0 Jun  7 23:55 host3
drwxr-xr-x 3 root root 0 Jun  7 23:58 host4
[root@localhost ~]# cat /sys/class/fc_host/host3/port_name 
0x21000024ff462814

 

2. 采用systool

systool所在軟件包：

yum -y install sysfsutils

systool -c fc_host -v
-c <class_name>  /  -b <bus_name> /  -m <module_name> 
[root@localhost ~]# systool -v -c fc_host
Class = "fc_host"
  Class Device = "host3"
  Class Device path = "/sys/class/fc_host/host3"
    fabric_name         = "0x1000000533746b8f"
    issue_lip           = <store method only>
    node_name           = "0x20000024ff462814"
    port_id             = "0x011300"
    port_name           = "0x21000024ff462814"
    port_state          = "Online"
    port_type           = "NPort (fabric via point-to-point)"
    speed               = "8 Gbit"
    supported_classes   = "Class 3"
    supported_speeds    = "1 Gbit, 2 Gbit, 4 Gbit, 8 Gbit"
    symbolic_name       = "QLE2562 FW:v5.03.16 DVR:v8.03.07.03.05.07-k"
    system_hostname     = ""
    tgtid_bind_type     = "wwpn (World Wide Port Name)"
    uevent              = <store method only>
    Device = "host3"
    Device path = "/sys/devices/pci0000:00/0000:00:07.0/0000:13:00.0/host3"
      ct                  = 
      edc                 = <store method only>
      els                 = 
      fw_dump             = 
      nvram               = "ISP "
      optrom_ctl          = <store method only>
      optrom              = 
      reset               = <store method only>
      sfp                 = ""
      uevent              = <store method only>
      vpd                 = "‚("

  Class Device = "host4"
  Class Device path = "/sys/class/fc_host/host4"
    fabric_name         = "0x10000005337638b9"
    issue_lip           = <store method only>
    node_name           = "0x20000024ff462815"
    port_id             = "0x021300"
    port_name           = "0x21000024ff462815"
    port_state          = "Online"
    port_type           = "NPort (fabric via point-to-point)"
    speed               = "8 Gbit"
    supported_classes   = "Class 3"
    supported_speeds    = "1 Gbit, 2 Gbit, 4 Gbit, 8 Gbit"
    symbolic_name       = "QLE2562 FW:v5.03.16 DVR:v8.03.07.03.05.07-k"
    system_hostname     = ""
    tgtid_bind_type     = "wwpn (World Wide Port Name)"
    uevent              = <store method only>
    Device = "host4"
    Device path = "/sys/devices/pci0000:00/0000:00:07.0/0000:13:00.1/host4"
      ct                  = 
      edc                 = <store method only>
      els                 = 
      fw_dump             = 
      nvram               = "ISP "
      optrom_ctl          = <store method only>
      optrom              = 
      reset               = <store method only>
      sfp                 = ""
      uevent              = <store method only>
      vpd                 = "‚("

 

Multipath相關軟件介紹

1. device-mapper-multipath

　　即multipath-tools，主要提供multipathd和multipath等工具和 multipath.conf等配置文件。這些工具通過device mapper的ioctr的接口創建和配置multipath，設備創建的多路徑設備映射會在/dev /mapper中。

2. device-mapper

　　主要包括兩大部分：內核部分和用戶部分。內核部分主要由device mapper核心（dm.ko）和一些target driver（md-multipath.ko）。核心完成設備的映射，而target根據映射關系和自身特點具體處理從mappered device 下來的i/o。同時，在核心部分，提供了一個接口，用戶通過ioctr可和內核部分通信，以指導內核驅動的行為，比如如何創建mappered device，這些divece的屬性等。linux device mapper的用戶空間部分主要包括device-mapper這個包。其中包括dmsetup工具和一些幫助創建和配置mappered device的庫。這些庫主要抽象，封裝了與ioctr通信的接口，以便方便創建和配置mappered device。multipath-tool的程序中就需要調用這些庫

3. dm-multipath.ko和dm.ko：dm.ko是device mapper驅動。它是實現multipath的基礎。dm-multipath其實是dm的一個target驅動。

 

 https://www.cnblogs.com/jianqiang2010/archive/2011/09/01/2162574.html

AIX

http://blog.163.com/herod_xiao/blog/static/871883992011819112227689/

 

 

1. 檢查multipath模塊，如果沒有相關模塊就說明沒有安裝相關軟件包

lsmod | grep dm_multipath

yum -y install device-mapper device-mapper-multipath

 

2. 加載相關模塊

modprobe dm-multipath

modprobe dm-round-robin

 

3. 生成multipath配置文件，並配置

/sbin/mpathconf --enable

配置multipath.conf文件

 

4. 啟動multipath服務

service multipathd start

chkcofig multipathd on

[root@tyhr ~]# multipath -ll
mpathc (36005076801820709e000000000000087) dm-3 IBM,2145
size=300G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 4:0:0:1 sdc 8:32  active ready running
  |- 4:0:2:1 sde 8:64  active ready running
  |- 5:0:2:1 sdg 8:96  active ready running
  `- 5:0:4:1 sdi 8:128 active ready running
mpathb (36005076801820709e000000000000086) dm-2 IBM,2145
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 4:0:0:0 sdb 8:16  active ready running
  |- 4:0:2:0 sdd 8:48  active ready running
  |- 5:0:2:0 sdf 8:80  active ready running
  `- 5:0:4:0 sdh 8:112 active ready running

 

 

如果完成配置后，不能使用multipath -ll查看到任何多路徑信息，就需要手工重新掃描HBA信息

一般新增LUN映射給主機后，系統無法直接更新掛載的存儲盤，需要重新掃描。

# cd /sys/class/scsi_host/host4/
# echo '- - - ' > scan
或
echo "1" > /sys/class/fc_host/host/issue_lip  //某些存儲或系統沒有scan文件，可以通issue_lip文件識別

 

RHEL5.x 操作系統，新增multipath后，能夠識別磁盤，但是通過fdisk分區后，不能顯示分區信息（/dev/mapper/dm-app 磁盤不顯示/dev/mapper/dm-appp1）。partprob無報錯但是沒有效果。fdisk w保存后有報錯： 

WARNING: Re-reading the partition table failed with error 22: Invalid argument.

Run "kpartx -a" after FDISK is completed to add all partition mappings on the 
newly-created multipath device



[root@TYEDI1 host4]# ls -l /dev/mapper/dm-*
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app
brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db

[root@TYEDI1 host4]# kpartx -a /dev/mapper/dm-app 


[root@TYEDI1 host4]# ls -l /dev/mapper/dm-*
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app
brw-rw---- 1 root disk 253, 3 Jan 21 23:01 /dev/mapper/dm-app1
brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db

　　

 

 

 其他配置：

1. udev綁定權限

RHEL6.5，對於multipath的權限，手工去修改幾秒后會變回root。所以需要使用udev去綁定好權限。

1)搜索對應的配置文件模板：
[root@jyrac1 ~]# find / -name 12-*
/usr/share/doc/device-mapper-1.02.79/12-dm-permissions.rules\

2)根據模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面：
vi /etc/udev/rules.d/12-dm-permissions.rules
# MULTIPATH DEVICES
#
# Set permissions for all multipath devices
ENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

# Set permissions for first two partitions created on a multipath device (and detected by kpartx)
# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"
3)查看多路徑對應的底層dm設備：
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw---- 1 root disk 253, 0 Jul 19 16:40 /dev/dm-0
brw-rw---- 1 root disk 253, 1 Jul 19 16:40 /dev/dm-1
brw-rw---- 1 root disk 253, 2 Jul 19 21:20 /dev/dm-2
brw-rw---- 1 root disk 253, 3 Jul 19 21:20 /dev/dm-3

4)啟動start_udev
[root@jyrac2 rules.d]# start_udev 
Starting udev: [  OK  ]

5)查看權限：
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw---- 1 root disk     253, 0 Jul 19 21:35 /dev/dm-0
brw-rw---- 1 root disk     253, 1 Jul 19 21:35 /dev/dm-1
brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2
brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3

6)等30s后再查，權限固定：
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw---- 1 root disk     253, 0 Jul 19 21:35 /dev/dm-0
brw-rw---- 1 root disk     253, 1 Jul 19 21:35 /dev/dm-1
brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2
brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3

 

 

 

 

 

 可以使用lsscsi查看識別的磁盤信息：

[root@tyhr ~]#  lsscsi 
[0:0:0:0]    cd/dvd  TSSTcorp DVD-ROM TS-L333H ID03  /dev/sr0 
[2:0:0:0]    cd/dvd  IMM      Virtual CD/DVD   0316  /dev/sr1 
[3:0:0:0]    disk    IBM-ESXS MBF2300RC        SB27  -       
[3:0:1:0]    disk    IBM-ESXS MBF2300RC        SB27  -       
[3:1:1:0]    disk    LSILOGIC Logical Volume   3000  /dev/sda 
[4:0:0:0]    disk    IBM      2145             0000  /dev/sdb 
[4:0:0:1]    disk    IBM      2145             0000  /dev/sdc 
[4:0:2:0]    disk    IBM      2145             0000  /dev/sdd 
[4:0:2:1]    disk    IBM      2145             0000  /dev/sde 
[5:0:2:0]    disk    IBM      2145             0000  /dev/sdf 
[5:0:2:1]    disk    IBM      2145             0000  /dev/sdg 
[5:0:4:0]    disk    IBM      2145             0000  /dev/sdh 
[5:0:4:1]    disk    IBM      2145             0000  /dev/sdi 

 

 

 

 

1. multipath常用操作：

multipath -r （修改multipath.conf配置文件之后重新加載）
multipath -ll （查看多路徑狀態）
multipath -v2 （格式化路徑，檢測路徑，合並路徑）
multipath -v3 （查看多路徑詳情blacklist、whitelist和設備wwid）
multipath -F （刪除現有鏈路）

 2. multipath文件詳細介紹

接下來的工作就是要編輯/etc/multipath.conf的配置文件

multipath.conf主要包括blacklist、multipaths、devices三部份的配置

blacklist配置

blacklist {

　　devnode "^sda"

}

Multipaths部分配置multipaths和devices兩部份的配置。

multipaths {

　　multipath {

　　　　wwid **************** 　　　　　　#此值multipath -v3可以看到

　　　　alias iscsi-dm0 　　　　　　　　#映射后的別名,可以隨便取

　　　　path_grouping_policy multibus 　　#路徑組策略

　　　　path_checker tur 　　　　　　　　#決定路徑狀態的方法

　　　　path_selector "round-robin 0" 　　#選擇那條路徑進行下一個IO操作的方法

　　}

}

Devices部分配置

devices {

　　device {

　　　　vendor "iSCSI-Enterprise" 　　　　　　　　　　#廠商名稱

　　　　product "Virtual disk" 　　　　　　　　　　　　#產品型號

　　　　path_grouping_policy multibus 　　　　　　　　#默認的路徑組策略

　　　　getuid_callout "/sbin/scsi_id -g -u -s /block/%n"　#獲得唯一設備號使用的默認程序

　　　　prio_callout      "/sbin/acs_prio_alua %d" 　　　  #獲取有限級數值使用的默認程序

　　　　path_checker readsector0 　　　　　　　　　　#決定路徑狀態的方法

　　　　path_selector "round-robin 0" 　　#選擇那條路徑進行下一個IO操作的方法

　　　　failback        immediate 　　　　#故障恢復的模式

   　　　no_path_retry      queue 　　　　#在disable queue之前系統嘗試使用失效路徑的次數的數值

  　　　　rr_min_io       100 　　　　　　#在當前的用戶組中，在切換到另外一條路徑之前的IO請求的數目

　　}

}

相關參數的標准文檔的介紹：

Attribute	Description
wwid	Specifies the WWID of the multipath device to which the multipath attributes apply. This parameter is mandatory for this section of themultipath.conf file.
alias	Specifies the symbolic name for the multipath device to which themultipath attributes apply. If you are using user_friendly_names, do not set this value tompathn; this may conflict with an automatically assigned user friendly name and give you incorrect device node names.
path_grouping_policy	Specifies the default path grouping policy to apply to unspecified multipaths. Possible values include: failover = 1 path per priority group multibus = all valid paths in 1 priority group group_by_serial = 1 priority group per detected serial number group_by_prio = 1 priority group per path priority value group_by_node_name = 1 priority group per target node name
path_selector	Specifies the default algorithm to use in determining what path to use for the next I/O operation. Possible values include: round-robin 0: Loop through every path in the path group, sending the same amount of I/O to each. queue-length 0: Send the next bunch of I/O down the path with the least number of outstanding I/O requests. service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to each path by its relative throughput.
failback	Manages path group failback. A value of immediate specifies immediate failback to the highest priority path group that contains active paths. A value of manual specifies that there should not be immediate failback but that failback can happen only with operator intervention. A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node requested the failover. A numeric value greater than zero specifies deferred failback, expressed in seconds.
prio	Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitableprio value. Possible values include: const: Set a priority of 1 to all paths. emc: Generate the path priority for EMC arrays. alua: Generate the path priority based on the SCSI-3 ALUA settings. tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit. ontap: Generate the path priority for NetApp arrays. rdac: Generate the path priority for LSI/Engenio RDAC controller. hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode. hds: Generate the path priority for Hitachi HDS Modular storage arrays.
no_path_retry	A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing. A value of fail indicates immediate failure, without queueing. A value of queue indicates that queueing should not stop until the path is fixed.
rr_min_io	Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems should userr_min_io_rq. The default value is 1000.
rr_min_io_rq	Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1.
rr_weight	If set to priorities, then instead of sending rr_min_io requests to a path before callingpath_selector to choose the next path, the number of requests to send is determined byrr_min_io times the path's priority, as determined by the prio function. If set touniform, all path weights are equal.
flush_on_last_del	If set to yes, then multipath will disable queueing when the last path to a device has been deleted.

 

multipath與oracle ASM結合

https://www.cnblogs.com/jyzhao/p/7208620.html

 

已有配置文件：

defaults {
    polling_interval    30
    failback            immediate
    no_path_retry       5
    rr_min_io           100
    path_checker        tur
    user_friendly_names yes 
}
devnode_blacklist {
        devnode "sda"
}
devices {
    device {
        vendor                   "IBM"
        product                  "2145"
        path_grouping_policy     group_by_prio
        prio_callout             "/sbin/mpath_prio_alua /dev/%n"
    }
    device {
        vendor                   "IBM"
        product                  "1750500"
        path_grouping_policy     group_by_prio
        prio_callout             "/sbin/mpath_prio_alua /dev/%n"
    }
    device {
        vendor                   "IBM"
        product                  "2107900"
        path_grouping_policy     group_by_serial
    }
    device {
        vendor                   "IBM"
        product                  "2105800"
        path_grouping_policy     group_by_serial
    }
}
multipaths {
        multipath {
                wwid    36005076801820709e000000000000043
                alias   dm-vote01
        }
}