Device Mapper Multipath（DM-Multipath）能夠將server節點和存儲陣列之間的多條I/O鏈路配置為一個單獨的設備。這些I/O鏈路是由不同的線纜、交換機、控制器組成的SAN物理鏈路。
Multipath將這些鏈路聚合在一起，生成一個單獨的新的設備。
1.DM-Multipath概覽：
（1）數據冗余
DM-Multipath能夠實如今active/passive模式下的災難轉移。在active/passive模式下。僅僅有一半的鏈路在工作，假設鏈路上的某一部分（線纜、交換機、控制器）出現問題，DM-Multipath就會切換到還有一半鏈路上。
（2）提高性能
DM-Multipath也能夠配置為active/active模式，從而I/O任務以round-robin的方式分布到全部的鏈路上去。通過配置，DM-Multipath還能夠檢測鏈路上的負載情況，動態地進行負載均衡。
DM-Multipath的存儲陣列的支持情況能夠查看multipath.conf.default。對於支持列表中沒有的存儲，能夠在multipath的配置文件multipath.conf中自行加入。
2.DM-Multipath的組成：
（1）dm-multipath內核模塊實現I/O重定向、支持failover以及鏈路的聚合
（2）mutilpath命令查看和配置multipath設備。它一般由/etc/rc.sysinit啟動。也會在系統發現新的塊設備時由udev啟動，或者由initramfs在系統啟動時運行。
（3）multipathd后台進程監控鏈路。當鏈路失效或者恢復時。它會發起鏈路組的切換。它實現了對multipath設備的動態改動。只是multipath.conf改動后，它須要又一次啟動。
（4）kpartx命令依據設備上分區表創建device mapper設備。當須要在DM-MP設備上使用基於DOS的分區時，這個命令是必須。
不使用DM-Multipath，每一條從server到存儲的鏈路都被系統識別為一個單獨的設備。
DM-Multipath能夠在這些底層的設備之上創建一個單一的multipath設備。實現對這些鏈路的組織和管理。
3.Multipath設備的標志符
每個multipath設備都有一個WWID（World Wide Identifier），這個id是全球唯一並且不可更改的。默認情況下，multipath設備的名稱被設置為它的WWID。只是也能夠在配置文件里使用_friendly_names選項。為設備取一個別名，別名為mpath[n]。
比如。一台server節點有2塊HBA卡，通過一台沒有划分zone的交換機。連接到2個磁盤陣列控制器。server系統中會發現4個設備：/dev/sda、/dev/sdb、/dev/sdc、/dev/sdd。DM-Multipath會依照配置文件在這些底層設備上創建一個擁有一個唯一WWID號的multipath設備。假設配置文件里_friendly_names選項被設置為yes，則這個multipath設備會被命名為mpath[n]。
當新的設備被DM-Multipath接管之后。新的設備文件會在/dev/文件夾下3個不同的地方出現：/dev/mapper/mpath[n]、/dev/mpath/mpath[n]、/dev/dm-[n]。
（1）/dev/mapper/文件夾下的文件。是早在系統啟動的過程中就創建了。
訪問multipath設備時就使用這些文件。比如創建lvm。
（2）/dev/mpath/文件夾下的文件。是為了能夠方便的在同一個文件夾下查看全部的multipath設備。這個文件由udev創建。
假設系統在啟動過程中須要訪問multipath設備，不要使用這些文件。不用在這些設備文件上建立lvm；
（3）/dev/dm-[n]僅僅為了內部使用目的，永遠不要對這些文件進行操作。
4.統一multipath設備的命名
當配置文件里_friendly_names被設置為yes，在該server節點上這個設備名是唯一並且確定的。可是不能保證在使用這些鏈路的其他server節點上的multipath設備的名稱可以相互保持一致。假設僅僅是建立lvm。那么這個問題不會有什么影響。可是假設希望不同server節點上的multipath設備的名稱可以統一，必須使用以下當中一種方法：
（1）在配置文件里的multipaths段使用alias選項為設備設置別名，並在不同的server上保持一致；
（2）假設希望不同server上multipath設備的user-friendly名稱保持一致。首先在一台server上建立全部multipath設備，然后把bindings文件復制到全部其他系統統一命名的server。binds文件的位置是/var/lib/multipath/bindings。在配置文件能夠使用bindings_file參數改動bindings文件的位置。
5.Multipath設備上建立lv
創建multipath設備之后，能夠像使用物理設備文件一樣在multipath設備上建立pv。比如。假定multipath設備為/dev/mapper/mpath0，使用
pvscreate /dev/mapper/mpath0
就可以將mpath0建立為物理卷。
相同能夠繼續建立卷族和邏輯卷。
當在配置為active/passive模式的multipath設備上建立邏輯卷時。須要在lvm的配置文件lvm.conf中添加過濾器，將multipath設備下層的設備加入到過濾列表中。
這時由於DM-Multipath會自己主動切換數據鏈路，當遇到failover和failback的情況時，假設下層的設備沒有在配置文件里過濾。lvm會掃描這些passive狀態下的數據鏈路。Passive狀態的鏈路改變到active狀態須要運行一些命令。所以lvm就會在這個時候報錯。
為了過濾全部的SCSI設備。在lvm.conf中的devices段，加入以下的配置：
filter = [ "r/disk/", "r/sd.*/", "a/.*/"
6.部署DM-Multipath
6.1開始部署
（1）編輯/etc/multipath.conf，凝視掉以下幾行：
devnode_blacklist {
devnode "*"
}
（2）multipath的默認配置已經集成在系統之中，不須要在/etc/multipath.conf中又一次配置。
path_grouping_policy的默認值為failover。在原始配置中default段設置了multipath設備的默認名稱是mpath[n]的形式，假設沒有這一段配置（即_friendly_names=yes），設備的默認名稱是它的WWID號。
（3）保存配置文件並退出。
（4）運行下列命令：
modprobe dm-multipath
sevice multipathd start
multipath -v2
注：multipath -v2會打印出已經聚合的數據鏈路。
（5）使用 chkconfig multipathd on，讓multipath服務開機自己主動啟動。
6.2排除本地scsi磁盤
非常多系統都安裝有本地scsi磁盤。DM-Multipath是不建議在這些磁盤上使用的。
能夠依照以下的步驟取消對本地scsi磁盤的映射。
（1）使用 multipath -v2 確認本地磁盤的信息。如以下的演示樣例（sda為本地scsi磁盤）：
[root@localhost ~]# multipath -v2
create: SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1
[size=33 GB][features="0"][hwhandler="0"]
\_ round-robin 0
\_ 0:0:0:0 sda 8:0
device-mapper ioctl cmd 9 failed: Invalid argument
device-mapper ioctl cmd 14 failed: No such device or address
create: 3600a0b80001327d80000006d43621677
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0
\_ 2:0:0:0 sdb 8:16
\_ 3:0:0:0 sdf 8:80
create: 3600 a0b80001327510000009a436215ec
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0
\_ 2:0:0:1 sdc 8:32
\_ 3:0:0:1 sdg 8:96
create: 3600a0b80001327d800000070436216b3
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0
\_ 2:0:0:2 sdd 8:48
\_ 3:0:0:2 sdh 8:112
create: 3600a0b80001327510000009b4362163e
[size=12 GB][features="0"][hwhandler="0"]
\_ round-robin 0
\_ 2:0:0:3 sde 8:64
\_ 3:0:0:3 sdi 8:128
（2）為了防止DM-Multipath對/dev/sda做映射。編輯/etc/multipath.conf中的devnode_blacklist段。
能夠使用devnode的方式過濾sda，只是系統中sda的命名不一定是固定的，所以最好使用wwid的方式。從上面的輸出中能夠看到/dev/sda的wwid為“SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1”，在配置文件加入：
devnode_blacklist{
wwid SIBM-ESXSST336732LC____F3ET0EP0Q000072428BX1
}
（3）運行以下的命令使配置生效，並又一次打印multipath設備列表。
multipath -F
multipath -v2
6.3在DM-Multipath中添加新的設備類型
DM-Multipath支持大部分的存儲陣列。默認的配置。能夠查看multipath.conf.default文件。
假設希望加入默認不支持的存儲設備，能夠在/etc/multipath.conf中加入對應的信息。
比如在配置為鑒中加入HP Open-V：
devices {
device {
vendor "HP"
product "OPEN-V"
getuid_callout "/sbin/scsi_id -g -u -p0x80 -s /block/%n"
}
}
7.DM-Multipath配置文件
7.1概覽
DM-Multipath的配置文件分為下面幾個部分：
devnode_blacklist
不使用DM-Multipath的設備列表。
默認的情況下，全部的設備都在列表中。啟用DM-Multipath的時候通常會將devnote_blacklist段凝視掉。
defaults
DM-Multipath的默認通用配置；
multipaths
單獨配置每個multipath設備的屬性。這些配置會覆蓋在defaults段和devices段的配置。
devices
單獨配置每個存儲控制器。
這些配置會覆蓋defaults段的配置。假設使用的存儲控制器不被DM-Multipath支持，那么就須要為這樣的類型的控制器加一個devices subsection。
DM-Multipath確定multipath設備的屬性時，會首先使用multipaths段的內容，然后是devices段，最好讀取defaults段。
7.2配置blacklist
devnode_blacklist指定了系統在配置multipath設備時不使用的設備，默認情況全部的設備都在這個列表中。凝視掉默認的一行之后，能夠在列表中增加某一種類型的設備或者某一特定的設備。禁用設備有2種方法：
（1）使用wwid：
能夠使用wwid指定特定的設備，如：
blacklist {
wwid 26353900f02796769
}
（2）使用設備名：
如：
devnode_blacklist {
devnode "^sd[a-z]"
}
這一段配置會禁用全部的SCSI磁盤設備。盡管能夠使用這樣的方法禁用單一特定的設備。可是並不建議這樣做。由於除非是使用了udev固定了設備的設備名。則設備的名稱在每次重新啟動之后是有可能發生變化的。
由於一些設備並不支持DM-Multipath，所以以下列出的設備是默認禁用的：
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^cciss!c[0-9]d[0-9]*"
}
device-mapper-multipathing。須要全然的升級操作系統。早期的紅帽企業Linux 4
不包括這個功能。編輯文件/etc/multipath.conf。在文件頭部凝視掉例如以下內容：
devnode_blacklist {
devnode "*"
}
例如以下例：
# devnode_blacklist {
# devnode "*"
# }
取消文件 /etc/multipath.conf中這段的凝視，這段內容能讓 device-mapper multipathing 不用掃描全部的設備。
etc/multipath.conf file:
defaults {
multipath_tool "/sbin/multipath -v0"
udev_dir /dev
polling_interval 10
default_selector "round-robin 0"
default_path_grouping_policy multibus
default_getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
default_prio_callout "/bin/true"
default_features "0"
rr_wmin_io 100
failback immediate
}
devnode_blacklist {
wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z][[0-9]*]"
devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}
這段設置了默認的 device-mapper 的動作。而且去出了通常不會有多路徑的設備。如IDE硬盤和軟盤。
默認的hd*設備的黑名單存在這一個排印錯誤。須要改動。
devnode "^hd[a-z][[0-9]*]"
把上行改動為例如以下:
devnode "^hd[a-z][0-9]*"
為了實現簡單的failover功能，下例中，defaults 組中的默認的 default_path_grouping_policy 選項被設置成為 failover。
defaults {
multipath_tool "/sbin/multipath -v0"
udev_dir /dev
polling_interval 10
default_selector "round-robin 0"
default_path_grouping_policy failover
default_getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
default_prio_callout "/bin/true"
default_features "0"
rr_wmin_io 100
failback immediate
}
退出編輯並保存設置，運行例如以下命令
modprobe dm-multipath
modprobe dm-round-robin
service multipathd start
multipath -v2
命令 multipath -v2 能顯示多路徑，從而能知道那些設備工作在多路徑下。假設沒有全部輸出。確認全部的SAN連接被正確的設置。系統有沒有正確的開啟多路徑功能。
運行例如以下命令確認多路徑服務有無啟動。
chkconfig multipathd on
設備的設備名會被生成，/dev/dm-#，#指代的是多路徑組，假設/dev/sda是唯一的多路徑設備，/dev/dm-0將會是
/dev/sda和/dev/sdb的多路徑設備。
注意：fdisk不能用於設備/dev/dm-#。使用fdisk僅僅能操作基礎磁盤，要在設備映射多路
徑映射設備上創建/dev/dm-#分區的操作. 運行一下命令。
kpartx -a /dev/dm-#
注意： dmsetup ls ?
target=multipath
是個協助偵測系統上多路徑設備的命令。假設在多路徑設備數據庫中沒有發現硬件，請查看文章“How can I add moreproducts
into the mutipathing database?”
device-mapper-multipath-0.4.7
=============================
RHEL4 U3 Device Mapper Multipath Usage
Maintainer
------------
Benjamin Marzinski
bmarzins@redhat.com
Overview
------------
Device Mapper Multipath (DM-MP) allows nodes to route I/O over multiple paths to
a storage controller. A path refers to the connection from an HBA port to a
storage controller port. As paths fail and new paths come up, DM-MP reroutes
the I/O over the available paths.
When there are multiple paths to a storage controller, each path
appears as a separate device. DM-MP creates a new device on top of
those devices. For example, a node with two HBAs attached to a storage
controller with two ports via a single unzoned FC switch sees four
devices: /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd. DM-MP creates a
single device, /dev/mpath/mpath1 that reroutes I/O to those four
underlying devices.
DM-MP consists of the
following components:
o dm-multipath kernel module -- This module reroutes I/O and fails
over paths and path groups.
o multipath command -- This command configures, lists, and removes multipath
devices. The command is run in rc.sysinit during startup, and by
udev, whenever a block device is added.
o multipathd daemon -- This daemon monitors paths, checking to see if faulty
paths have been fixed. As paths come back up, multipathd may also
initiate path group switches to ensure that the optimal path group
is being used. Also, it is possible to interactively modify a
multipath device.
o kpartx command -- This command creates Device Mapper devices for the
partitions on a device. It is necessary to use this command for DOS-
based partitions with DM-MP.
DM-MP works with a variety of storage arrays. It
auto-configures the following storage arrays:
o 3PARdata VV
o Compaq HSV110
o Compaq MSA1000
o DDN SAN DataDirector
o DEC HSG80
o EMC SYMMETRIX
o EMC CLARiiON
o FSC CentricStor
o GNBD
o HITACHI DF400
o HITACHI DF500
o HITACHI DF600
o HP HSV110
o HP HSV210
o HP A6189A
o HP Open-
o IBM 3542
o IBM ProFibre 4000R
o NETAPP
o SGI TP9100
o SGI TP9300
o SGI TP9400
o SGI TP9500
o STK OPENstroage D280
o SUN StorEdge 3510
o SUN T4
Storage arrays not included in the list may require entries in the
/etc/multipath.conf file.
NOTE: Some storage arrays require special handling of I/O errors and
path-group switching. Those require separate hardware handler
kernel modules.
Terms and Concepts
---------------------
Hardware Handler:
A kernel module that performs hardware-specific actions when switching
path groups and dealing with I/O errors.
Path:
The connection from an HBA port to a storage controller port for a LUN.
Each path appears as a separate device. Paths can be in
various states (refer to "Path States").
Path States:
ready - Path is able to handle I/O requests.
shaky - Path is up, but temporarily not available for normal
operations.
faulty - Path is unable to handle I/O requests.
ghost - Path is a passive path, on an active/passive controller.
NOTE: The shaky and ghost states only exist for certain
storage arrays.
Path Group:
A grouping of paths. With DM-MP, only one path group--the
active path group--receives I/O at any time. Within a path
group, DM-MP selects which ready path should receive I/O in a
round robin fashion. Path groups can be in various states (refer to
"Path Group States").
Path Group States:
active - Path group currently receiving I/O requests.
enabled - Path groups to try if the active path group has no paths
in the ready state.
disabled - Path groups to try if the active path group and all
enabled path groups have no paths in the active state.
NOTE: The disabled state only exists for certain storage arrays.
Path Priority:
Each path can have a priority assigned to it by a callout program.
Path priorities can be used to group paths by priority and change
their relative weights for the round robin path selector.
Path Group Priority:
Each path group has a priority that is equal to the sum of the
priorities of all the non-faulty paths in the group. By default, the
multipathd daemon tries to ensure that the path group with the
highest priority is always in the active state.
Failover:
When I/O to a path fails, the dm-multipath module tries to switch to
an enabled path group. If there are no enabled path groups with
any paths in the ready state, dm-multipath tries to switch to a disabled
path group. If necessary, dm-multipath runs the hardware handler for the
multipath device.
Failback:
At regular intervals, multipathd checks the current priority of
all path groups. If the current path group is not the highest priority
path group, multipathd reacts according to the failback mode.
By default, multipathd immediately switches to the highest priority
path group. Other options for multipathd are to (a) wait for a
user-defined length of time (for the path groups to stabilize)
and then switch or (b) for multipathd to do nothing and wait for
manual intervention. Failback can be forced at any time by
running the multipath command.
Multipath device:
The multipath device is the device mapper device created by
dm-multipath. A multipath device can be identified by either
its WWID or its alias. A multipath device has one or more path
groups. It also has numerous attributes defined in the
following file:
/usr/share/doc/device-mapper-multipathd-0.4.5/multipath.conf.annotated
alias:
The alias is the name of a multipath device. By default, the
alias is set to the WWID. However, by setting the
"user_friendly_names" configuration option, the alias is set to a
unique name of the form mpath<n>. The alias name can also be
explicitly set for each multipath device in the configuration file.
NOTE: While the alias in guaranteed to be unique on a node, it
is not guaranteed to be the same on all nodes using the
multipath device. Also, it may change.
WWID:
The WWID (World Wide Identifier) is an identifier for the
multipath device that is guaranteed to be globally unique and
unchanging. It is determined by the getuid callout program.
Using DM-MP
------------------------------------
Initial setup:
1. If it is not already installed. Install the device-mapper-multipath
package.
2. Edit /etc/multipath.conf. For new installations, all devices are blacklisted.
The default blacklist is listed in the commented out section of
/etc/multipath.conf. If you comment out or delete the following lines in
/etc/multipath.conf, the default blacklist takes effect:
devnode_blacklist {
devnode "*"
}
For some conditions, that may not be sufficient. If DM-MP is
multipathing devices that you do not want it to work on, you can
blacklist the devices by either device name or WWID.
NOTE: It is safest to blacklist individual devices by WWID, because
their device names may change.
Several other configuration options are detailed later in this
document. To check the effects of configuration changes, you can
do a dry run with the following command:
# multipath -v2 -d
3. Set the multipathd init script to run at boot time. by issuing the commands
# chkconfig --add multipathd
# chkconfig multipathd on
4. start dm-multipath (This is only necessary the first time. On
reboot, this should happen automatically).
# multipath
# /etc/init.d/multipathd start
After initial setup, all access to the multipathed storage should go through the
multipath device.
GNBD devices will not be automatically multipathed after they are imported.
The command
# multipath
must be run after every time the devices are imported. Otherwise the Multipath
devices will not be created.
Configuration File:
Many features of DM-MP are configurable using the configuration file,
/etc/multipath.conf.
For a complete list of all options with descriptions, refer to
/usr/share/doc/device-mapper-multipathd-0.4.5/multipath.conf.annotated
The configuration file is divided into four sections: system defaults,
blacklisted devices (devnode_blacklist), per storage array model settings
(devices), and per multipath device settings (multipaths). The per multipath
device settings are used for the multipath device with a matching "wwid"
value. The per storage array model settings are used for all multipath devices
with matching "vendor" and "product" values. To determine the attributes of a
multipath device, first the per multipath settings are checked, then the per
controller settings, then the system defaults. The blacklisted device section
is described setup step 2.
NOTE: There are compiled-in defaults for the "defaults", "devnode_blacklist",
and "devices" sections of the configuration file. To see what these
are, refer to the following file:
/usr/share/doc/device-mapper-multipathd-0.4.5/multipath.conf.defaults
If you are using one of the storage arrays listed in the preceding
text (in "Overview"), you probably do not need to modify the "devices"
subsection. If you are using a simple disk enclosure, the defaults
should work. If you are using a storage array that is not
listed, you may need to create a "devices" subsection for your array.
Reconfiguring a running system:
If any changes are make to the configuration file that will effect multipathd
(check /usr/share/doc/device-mapper-multipathd-0.4.5/multipath.conf.annotated
to see if mulipathd is in the options scope), you must restart multipathd
for these to take effect. To do that, run
# /etc/init.d/multipathd restart
Explanation of output
-----------------------
When you create, modify, or list a multipath device, you get a printout of
the current device setup. The format is as follows.
For each multipath device:
action_if_any: alias (wwid_if_different_from_alias)
[size][features][hardware_handler]
For each path group:
\_ scheduling_policy [path_group_priority_if_known][path_group_status_if_known]
For each path:
\_ host:channel:id:lun devnode major:minor [dm_status_if_known][path_status]
The dm status (dm_status_if_known) is like the path status
(path_status), but from the kernel's point of view. The dm status has two
states: "failed", which is analogous to "faulty", and "active" which
covers all other path states. Occasionally, the path state and the
dm state of a device will temporarily not agree.
NOTE: When a multipath device is being created or modified, the path group
status and the dm status are not known. Also, the features are not always
correct. When a multipath device is being listed, the path group priority is not
known.
Restrictions
---------------
DM-MP cannot be run on either the root or boot device.
Other Sources of information
----------------------------
Configuration file explanation:
/usr/share/doc/device-mapper-multipathd-0.4.5/multipath.conf.annotated
Upstream documentation:
http://christophe.varoqui.free.fr/wiki/wakka.php?
mailing list:
dm-devel@redhat.com
Subscribe to this from https://www.redhat.com/mailman/listinfo/dm-devel.
The list archives are at https://www.redhat.com/archives/dm-devel/
Man pages:
multipath.8, multipathd.8, kpartx.8 mpath_ctl.8