數據存儲技術介紹
DSA(Direct Attacted Storage 直接附加存儲) 本地磁盤接口:IDE SATA SAS SCSI
NAS(NetWork Attacted Storage 網絡附加存儲) 網絡服務共享: 共享的是文件夾,如 samba NFS
SAN(Storage Attacted Network 存儲附加網絡) 網絡服務共享: 共享的是設備,如iscsi
ISCSI介紹
ISCSI(internet SCSI)是一個協議,通過TCP/IP網絡的方式將存儲設備共享到客戶端,仿真為本地設備,讓本地計算機像管理直連存儲一樣,對共享的設備存儲進行分區,格式化,掛載使用等.ISCSI之間通信時一種C/S架構
客戶端C:為發起者
服務端S:為目標,包含多個LUN(Logical Unit Number邏輯單元),真正提供服務的是LUN,一個LUN為一個設備,可用同時共享多個設備給客戶端
ISCSI原理
·SCSI層根據客戶端發出的請求建立SCSI CDB(命令描述塊）,並且傳給ISCSI層。同時接收來自ISCSI層的CDB,並向應用返回數據。
·ISCSI對SCSICDB進行封裝，以便能夠在基於TCP/IP協議的網絡上進行傳輸，完成SCSI到TCP/IP的協議映射。這一層是ISCS協議的核心層。
·TCP/IP層對IP報文進行路由和轉發，並且提供端到端的透明可靠傳輸。
SAN/IPSAN
IPSAN是在SAN后產生的，
SAN默認指FCSAN,以光纖通道構建存儲網絡，
IPSAN則以IP網絡構建存儲網絡，較FCSAN,具有更經濟、自由擴展等特點。
ISCSI服務端部署
安裝啟動ISCSI服務
# 安裝ISCSI服務
yum install -y targetcli
systemctl enable target
systemctl start target
准備工作:准備共享硬盤設備或磁盤文件
[root@centos2 ~]# lsblk # 查看設備
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 10G 0 disk
└─sdb1 8:17 0 5G 0 part
sdc 8:32 0 10G 0 disk
sdd 8:48 0 20G 0 disk
sde 8:64 0 10G 0 disk
sdf 8:80 0 10G 0 disk
sr0 11:0 1 1G 0 rom
# 可用dd創建一個5G的文件
dd if=/dev/zero of=/opt/sdb2 bs=1M count=5000
ISCSI服務端部署
工作流:將設備添加到backstores存儲庫--->設置全球唯一標識名IQN ---> 設置TPG組,定義誰可用重哪個IP即端口訪問IQN標識對應的哪些設備
[root@centos2 ~]# targetcli # targetcli 進行交互操作
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> help
/> ls # 如下4類存儲可用 塊設備,文件,物理磁盤,划分的磁盤
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 0]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 0]
o- loopback ..................................................... [Targets: 0]
/>
- 添加存儲設備到存儲庫
/> cd backstores #進入存儲庫
/backstores> block/ create block1 /dev/sde # 創建一個塊設備,名稱block1,使用系統上/dev/sde設備
Created block storage object block1 using /dev/sde.
/backstores> ls
o- backstores ............................................................ [...]
o- block ................................................ [Storage Objects: 1]
| o- block1 ...................... [/dev/sde (10.0GiB) write-thru deactivated] #剛創建的設備
| o- alua ................................................. [ALUA Groups: 1]
| o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
o- fileio ............................................... [Storage Objects: 0]
o- pscsi ................................................ [Storage Objects: 0]
o- ramdisk .............................................. [Storage Objects: 0]
/backstores>
/backstores> block/ delete block1 #刪除設備
Deleted storage object block1.
/backstores>
/backstores> cd block #進入塊設備
/backstores/block> create block002 /dev/sdb1 # 創建塊設備block002
Created block storage object block002 using /dev/sdb1
/backstores/block>
/backstores/block> create block001 /dev/sde # 重新創建設備block001
Created block storage object block001 using /dev/sde.
/backstores/block>
/> cd backstores/fileio
/backstores/fileio> create block003 /opt/sdb2 # 進入文件設備庫,並創建文件設備
Created fileio block003 with size 5242880000
/backstores/fileio> ls # 查看創建的設備
o- fileio ................................................. [Storage Objects: 1]
o- block003 ...................... [/opt/sdb2 (4.9GiB) write-back deactivated]
o- alua ................................................... [ALUA Groups: 1]
o- default_tg_pt_gp ....................... [ALUA state: Active/optimized]
/backstores/fileio> cd ..
/backstores> ls
o- backstores ............................................................ [...]
o- block ................................................ [Storage Objects: 2]
| o- block001 .................... [/dev/sde (10.0GiB) write-thru deactivated]
| | o- alua ................................................. [ALUA Groups: 1]
| | o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
| o- block002 .................... [/dev/sdb1 (5.0GiB) write-thru deactivated]
| o- alua ................................................. [ALUA Groups: 1]
| o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
o- fileio ............................................... [Storage Objects: 1]
| o- block003 .................... [/opt/sdb2 (4.9GiB) write-back deactivated]
| o- alua ................................................. [ALUA Groups: 1]
| o- default_tg_pt_gp ..................... [ALUA state: Active/optimized]
o- pscsi ................................................ [Storage Objects: 0]
o- ramdisk .............................................. [Storage Objects: 0]
/backstores>
- 設置IQN(創建全球唯一標識設備名),必須符合統一格式
# IQN格式: iqn.年-月.二級域名倒寫:共享名
/backstores> cd /iscsi
/iscsi> create iqn.2021-05.com.iscsitest:storage # 設置iqn
Created target iqn.2021-05.com.iscsitest:storage.
Created TPG 1. # 自動設置TPG組為 1
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260. #默認監聽的IP和端口號
/iscsi> ls
o- iscsi .......................................................... [Targets: 1]
o- iqn.2021-05.com.iscsitest:storage ............................... [TPGs: 1]
o- tpg1 ............................................. [no-gen-acls, no-auth] # TPG1組
o- acls ........................................................ [ACLs: 0] # 誰可訪問
o- luns ........................................................ [LUNs: 0] # 可訪問哪些設備
o- portals .................................................. [Portals: 1]
o- 0.0.0.0:3260 ................................................... [OK] # 或者配置服務端自己的IP
/iscsi>
/iscsi> cd iqn.2021-05.com.iscsitest:storage/tpg1/acls/ # 進入訪問權限設置
/iscsi/iqn.20.../tpg1/portals> delete 192.168.198.129 3260
Deleted network portal 192.168.198.132:3260
/iscsi/iqn.20.../tpg1/portals> create 192.168.198.129 3260
- 設置可以訪問設備的用戶為名
# 設置用戶名 iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/acls> create iqn.2021-05.com.iscsitest:cliente001
Created Node ACL for iqn.2021-05.com.iscsitest:cliente001
- 將設備共享出來
/iscsi/iqn.20...age/tpg1/acls> ../luns/ create /backstores/block/block001 # 將block001設備共享出來
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/acls> cd ../luns/
/iscsi/iqn.20...age/tpg1/luns> create /backstores/block/block002 # 將block002設備共享出來
Created LUN 1.
Created LUN 1->1 mapping in node ACL iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/luns> create /backstores/fileio/block003 # 將block003文件設備共享出來
Created LUN 2.
Created LUN 2->2 mapping in node ACL iqn.2021-05.com.iscsitest:cliente001
/iscsi/iqn.20...age/tpg1/luns> cd ..
/iscsi/iqn.20...:storage/tpg1> ls
o- tpg1 ................................................. [no-gen-acls, no-auth]
o- acls ............................................................ [ACLs: 1]
| o- iqn.2021-05.com.iscsitest:cliente001 ................... [Mapped LUNs: 3]
| o- mapped_lun0 ................................ [lun0 block/block001 (rw)]
| o- mapped_lun1 ................................ [lun1 block/block002 (rw)]
| o- mapped_lun2 ............................... [lun2 fileio/block003 (rw)]
o- luns ............................................................ [LUNs: 3]
| o- lun0 ..................... [block/block001 (/dev/sde) (default_tg_pt_gp)]
| o- lun1 .................... [block/block002 (/dev/sdb1) (default_tg_pt_gp)]
| o- lun2 ................... [fileio/block003 (/opt/sdb2) (default_tg_pt_gp)]
o- portals ...................................................... [Portals: 1]
o- 0.0.0.0:3260 ....................................................... [OK]
/iscsi/iqn.20...:storage/tpg1>
- 相應的權限及訪問設置
/iscsi/iqn.20...:storage/tpg1> portals/ delete 0.0.0.0 3260 # 也可以對訪問的IP和端口進行刪除和創建
Deleted network portal 0.0.0.0:3260
/iscsi/iqn.20...:storage/tpg1> portals/ create 0.0.0.0 3260
Using default IP port 3260
Binding to INADDR_ANY (0.0.0.0)
Created network portal 0.0.0.0:3260.
/iscsi/iqn.20...:storage/tpg1> exit # 保存退出
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup/.
Configuration saved to /etc/target/saveconfig.json
[root@centos2 ~]#
[root@centos2 ~]# vi /etc/target/saveconfig.json # 剛才設置保存的文件
[root@centos2 ~]# systemctl restart target # 設置完后重啟服務
ISCSI客戶端部署
工作流程:安裝客戶端 ---> 設置客戶端iqn設備名稱 ---> 安裝客戶端服務並啟動 ---> 訪問共享 ---> 分區,格式化,自動掛載
1. 安裝客戶端
[root@centos ~]# yum install -y iscsi-initiator-utils
2. 設置iscsi名稱
[root@centos ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2021-05.com.iscsitest:cliente001
3. 啟動iscsi客戶端服務
[root@centos ~]# systemctl start iscsi
[root@centos ~]# systemctl enable iscsi
[root@centos ~]# man iscsiadm
#iscsid 為iscsi服務的守護進程,會隨iscsi自動起來,配置文件/etc/iscsi/iscsid.conf
4. 發現共享設備
[root@centos ~]# iscsiadm --mode discoverydb --type sendtargets --portal 192.168.198.129 --discover
192.168.198.129:3260,1 iqn.2021-05.com.iscsitest:storage
[root@centos ~]# ll /var/lib/iscsi/nodes #查看
total 0
drw------- 3 root root 36 May 23 02:36 iqn.2021-05.com.iscsitest:storage
5. 連接共享設備
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.129 --login
Logging in to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] (multiple)
Login to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] successful.
[root@centos ~]# lsblk # 查看掛載成功
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk # sdc/sdd/sde為新添加
sdd 8:48 0 5G 0 disk
sde 8:64 0 4.9G 0 disk
sr0 11:0 1 1G 0 rom
6. 進行分區格式化掛載
[root@centos ~]# fdisk /dev/sdc
[root@centos ~]# vi /etc/fstab
/dev/sdc1 /mnt/block1 ext4 _netdev 0 0 # _netdev 網絡文件,不要用默認
[root@centos ~]# mkfs.ext4 /dev/sdc1
[root@centos ~]# mount -a
[root@centos ~]#
[root@centos ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 899M 0 899M 0% /dev
tmpfs 910M 0 910M 0% /dev/shm
tmpfs 910M 9.5M 901M 2% /run
tmpfs 910M 0 910M 0% /sys/fs/cgroup
/dev/mapper/centos-root 17G 4.7G 13G 28% /
/dev/sda1 1014M 181M 834M 18% /boot
tmpfs 182M 0 182M 0% /run/user/0
192.168.198.130:/test 17G 4.7G 13G 28% /mnt/nfs
/dev/sdc1 9.7G 37M 9.2G 1% /mnt/block1 # 查看已經掛載成功
客戶端停止ISCSI 服務
# 查看服務名
[root@centos ~]# iscsiadm --mode node
192.168.198.129:3260,1 iqn.2021-05.com.iscsitest:storage
# 客戶端停止使用ISCSI共享
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.129:3260 --logout
# 徹底刪除,如果login登錄不了,需要刪除設備,重新發現並登錄
[root@centos ~]# rm -rf /var/lib/iscsi/nodes/iqn.2021-05.com.iscsitest:storage/
[root@centos ~]# rm -rf /var/lib/iscsi/send_targets/192.168.198.129,3260/
# 以上相關命令可用簡寫
IPSAN多鏈路共享
解決單點故障
1. 准備工作
服務端和客戶端都配備兩塊網卡進行連接
添加網卡,配置IP地址
# 服務端,添加新網卡
[root@centos2 ~]# ip a
192.168.198.129/24
192.168.198.132/24
# 客戶端也添加新網卡
[root@centos ~]# ip a
192.168.198.133/24
192.168.198.130/24
# nmcli配置IP地址
nmcli con add con-name eth1 ifname ens36 type 802-3-ethernet ipv4.method manual ipv4.add 192.168.198.136/24
# 檢查以上網卡是否相互ping通
[root@centos2 ~]# ping 192.168.198.133
[root@centos2 ~]# ping 192.168.198.130
[root@centos ~]#192.168.198.132
[root@centos ~]#192.168.198.129
2. 服務端設置
參考:ISCSI服務端部署,配置成如下配置
[root@centos2 ~]# targetcli
/> ls
o- / ...................................................................... [...]
o- backstores ........................................................... [...]
| o- block ............................................... [Storage Objects: 1]
| | o- block001 ..................... [/dev/sdc (10.0GiB) write-thru activated]
| | o- alua ................................................ [ALUA Groups: 1]
| | o- default_tg_pt_gp .................... [ALUA state: Active/optimized]
| o- fileio .............................................. [Storage Objects: 0]
| o- pscsi ............................................... [Storage Objects: 0]
| o- ramdisk ............................................. [Storage Objects: 0]
o- iscsi ......................................................... [Targets: 1]
| o- iqn.2021-05.com.iscsitest:storage1 ............................. [TPGs: 1]
| o- tpg1 ............................................ [no-gen-acls, no-auth]
| o- acls ....................................................... [ACLs: 1]
| | o- iqn.2021-05.com.iscsitest:clientemultipath ........ [Mapped LUNs: 1]
| | o- mapped_lun0 ........................... [lun0 block/block001 (rw)]
| o- luns ....................................................... [LUNs: 1]
| | o- lun0 ................ [block/block001 (/dev/sdc) (default_tg_pt_gp)] # LUN 把設備共享出來
| o- portals ................................................. [Portals: 2]
| o- 192.168.198.129:3260 .......................................... [OK] # 配置服務端的IP進行共享
| o- 192.168.198.132:3260 .......................................... [OK]
o- loopback ...................................................... [Targets: 0]
3. 客戶端設置
工作流程: 安裝客戶端並啟動 ---> 連接共享設備從兩條線路 ---> 分區格式化 ---> 安裝多路徑軟件 ---> 實現多路徑負載均衡
客戶端配置參考:ISCSI客戶端部署
# 客戶端發現兩個設備,並進行掛載
[root@centos ~]# iscsiadm --mode discoverydb --type sendtargets --portal 192.168.198.129 --discover
192.168.198.129:3260,1 iqn.2021-05.com.iscsitest:storage
192.168.198.132:3260,1 iqn.2021-05.com.iscsitest:storage
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.129 --login
Logging in to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] (multiple)
Login to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.129,3260] successful.
[root@centos ~]#
[root@centos ~]# iscsiadm --mode node --targetname iqn.2021-05.com.iscsitest:storage --portal 192.168.198.132 --login
Logging in to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.132,3260] (multiple)
Login to [iface: default, target: iqn.2021-05.com.iscsitest:storage, portal: 192.168.198.132,3260] successful.
分區及格式化
(多路徑不能格式化為xfs格式)
[root@centos ~]# fdisk /dev/sdc
[root@centos ~]# mkfs.ext4 /dev/sdc1
[root@centos ~]# partprobe #更新分區標信息
[root@centos ~]#
[root@centos ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
└─sdc1 8:33 0 10G 0 part
sdd 8:48 0 10G 0 disk
└─sdd1 8:49 0 10G 0 part
sr0 11:0 1 1G 0 rom
安裝multipath多路徑軟件並啟動
# 安裝多路徑軟件
yum install -y device-mapper-multipath
# 拷貝配置文件
cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/
# 啟動服務
systemctl enable multipathd; systemctl start multipathd
# 查看
[root@centos ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
└─mpatha 253:2 0 10G 0 mpath
└─mpatha1 253:3 0 10G 0 part
sdd 8:48 0 10G 0 disk
└─mpatha 253:2 0 10G 0 mpath
└─mpatha1 253:3 0 10G 0 part
sr0 11:0 1 1G 0 rom
配置multipath負載均衡模式
[root@centos ~]# multipath -ll # 多路徑block001設備id 360014051f73df963d734b45aa3f450e9
mpatha (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active # 默認是提供AB模式,可修改配置文件,變成負載均衡模式
| `- 21:0:0:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 22:0:0:0 sdd 8:48 active ready running
[root@centos ~]# vi /etc/multipath.conf
#defaults {
# user_friendly_names yes
# find_multipaths yes
#}
multipaths {
multipath {
wwid 360014051f73df963d734b45aa3f450e9 # 設備編號
alias multipathtest # 別名
path_grouping_policy multibus # 多路徑組策略
path_selector "round-robin 0" # 負載均衡模式
failback manual
rr_weight priorities # 按優先級輪詢
no_path_retry 5 # 重試時間
}
multipath {
wwid 1DEC_____321816758474
alias red
}
}
[root@centos ~]# systemctl start multipathd
[root@centos ~]# systemctl start iscsi
[root@centos ~]# multipath -ll
multipathtest (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 21:0:0:0 sdc 8:32 active ready running # 再次查看,多路徑已經變成rr負載均衡模式
`- 22:0:0:0 sdd 8:48 active ready running
# multipath -F #刪除現有路徑,新的路徑就會被刪除
# multipath -v2 #格式化路徑,格式化后又出現
[root@centos ~]# ll /dev/mapper
total 0
lrwxrwxrwx 1 root root 7 May 23 16:24 centos-root -> ../dm-0
lrwxrwxrwx 1 root root 7 May 23 16:24 centos-swap -> ../dm-1
crw------- 1 root root 10, 236 May 23 14:28 control
lrwxrwxrwx 1 root root 7 May 23 16:56 multipathtest -> ../dm-2
lrwxrwxrwx 1 root root 7 May 23 16:56 multipathtest1 -> ../dm-3
[root@centos ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 17G 0 lvm /
└─centos-swap 253:1 0 2G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sdc 8:32 0 10G 0 disk
└─multipathtest 253:2 0 10G 0 mpath # 這里是剛剛配置文件中設置的別名,也可以不設置別名
└─multipathtest1 253:3 0 10G 0 part
sdd 8:48 0 10G 0 disk
└─multipathtest 253:2 0 10G 0 mpath
└─multipathtest1 253:3 0 10G 0 part
sr0 11:0 1 1G 0 rom
# 掛載使用(也可以umount后重新格式化)
[root@centos ~]# mount /dev/mapper/multipathtest1 /mnt/multi/
multipath多鏈路容災測試
#在客戶端可進行讀寫操作
[root@centos ~]# touch /mnt/multi/file1
# 將服務端網卡down掉
[root@centos2 ~]# ifdown ens37
# 在客戶端查看,掉線一條鏈路,另一條正常運行
[root@centos ~]# multipath -ll
multipathtest (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 21:0:0:0 sdc 8:32 active faulty running
`- 22:0:0:0 sdd 8:48 active ready running
# 將服務端網卡啟動
[root@centos2 ~]# ifup ens37
# 在客戶端查看,兩條鏈路都正常運行
[root@centos ~]# multipath -ll
multipathtest (360014051f73df963d734b45aa3f450e9) dm-2 LIO-ORG ,block001
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 21:0:0:0 sdc 8:32 active ready running
`- 22:0:0:0 sdd 8:48 active ready running