本文結構：
一.存儲介紹
1.1 常見存儲類型的比較
1.2 NAS存儲
1.3 SAN存儲
1.4 NAS與SAN的主要區別
二.配置IPSAN存儲
2.1 存儲服務端配置
2.2 存儲客戶端10配置
2.3 存儲客戶端11配置
一.存儲介紹
1.1 常見存儲類型
直連式存儲(Direct-Attached Storage，DAS)：消耗服務器本身的IO資源和CPU資源，使用方便，服務器直接識別為一個塊設備
非直連式存儲：將IO資源和CPU資源消耗轉移到后端專門的存儲設備上，通常是通過網線或者光纖與需要使用存儲的服務器相連接，因此這種存儲類型對網絡要求很高，常見的兩種類型是SAN和NAS
1.2 NAS存儲
NAS：網絡附屬存儲基於標准網絡協議（Tcp/IP）實現數據傳輸，為網絡中的Windows / Linux / Mac OS 等各種不同操作系統的計算機提供文件共享和數據備份，NAS本身能夠支持多種協議，如NFS、CIFS、FTP、HTTP等，由此可以看出NAS對於主機而言是可以直接使用的文件系統，不需要直接面對塊設備(就更不用格式化和掛載塊設備了)。
1.3 SAN存儲
SAN:采用網狀通道（Fibre Channel ，簡稱FC）技術，通過FC交換機連接存儲陣列和服務器主機，建立專用於數據存儲的區域網絡。
SAN的存儲類型：
(1). IPSAN: 利用IP網絡構建存儲網絡, 使用TCP/IP協議的iscsi協議封裝構建的存儲區域網絡
(2). FSCAN: 利用光纖線, 通過高速FC交換機組成的存儲區域網絡，需要使用到光纖交換機和HBA卡，相對而言成本高
1.4 NAS和SAN的主要區別
第一，從網絡架構來說，本質區別在於：NAS直接使用TCP/IP傳輸數據。SAN使用SCSI或iSCSI協議傳輸數據。
第二，從文件讀寫實現方法上來說，本質區別在於：NAS采用了NFS和 CIFS技術實現文件共享。說明NAS是基於操作系統的“文件級”讀寫操作。SAN中計算機和存儲間的接口是底層的塊協議，它按照協議頭的“塊地址+偏移地址”來定位。共享的存儲和前端的操作系統類型沒有關系，任何服務器操作系統，都可以正常識別。
各類存儲類型特點總結如下：
| 存儲結構/性能對比 |
DAS |
NAS |
FC-SAN |
IP-SAN |
| 成本 |
低 |
較低 |
高 |
較高 |
| 數據傳輸速度 |
快 |
慢 |
極快 |
較快 |
| 擴展性 |
無擴展性 |
較低 |
易於擴展 |
最易擴展 |
| 服務器訪問存儲方式 |
直接訪問存儲數據塊 |
以文件方式訪問 |
直接訪問存儲數據塊 |
直接訪問存儲數據塊 |
| 服務器系統性能開銷 |
低 |
較低 |
低 |
較高 |
| 安全性 |
高 |
低 |
高 |
低 |
| 是否集中管理存儲 |
否 |
是 |
是 |
是 |
| 備份效率 |
低 |
較低 |
高 |
較高 |
| 網絡傳輸協議 |
無 |
TCP/IP |
Fibre Channel |
TCP/IP |
二.配置IPSAN存儲
2.1 存儲服務端配置
(1). 實驗環境
| 主機名 |
IP地址 |
身份 |
操作系統 |
| SAN |
192.168.0.105 |
SAN存儲 |
CentOS7 |
| 10 |
10.4.7.10 |
客戶端1 |
CentOS7 |
| 11 |
10.4.7.11 |
客戶端2 |
CentOS7 |
[root@SAN ~]# iptables -F #清空防火牆規則
[root@SAN ~]# systemctl stop firewalld && systemctl disable firewalld #關閉防火牆
[root@SAN ~]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config #關閉selinux
(2).安裝配置工具：
[root@SAN ~]# yum -y install targetcli
(3).配置存儲
[root@SAN ~]# targetcli
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.fb49
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> help #">"符號前的"/"表示當前路徑為根目錄，根Linux shell類似
······ AVAILABLE COMMANDS #可用命令 ================== The following commands are available in the current path: #一下命令在當前路徑 - bookmarks action [bookmark] - cd [path] #切換路徑 - clearconfig [confirm] #清空配置信息 - exit #退出 - get [group] [parameter...] - help [topic] #幫助 - ls [path] [depth] #列出當前路徑的文件和目錄 - pwd #查看當前路徑 - refresh - restoreconfig [savefile] [clear_existing] - saveconfig [savefile] #保存配置，默認文件路徑為/etc/target/saveconfig.json - sessions [action] [sid] #查看當前的客戶端與存儲服務器的連接情況 - set [group] [parameter=value...] #設置全局和局部屬性 - status #狀態 - version #版本
/> ls #查看當前根目錄下的文件信息
o- / ............................................................................................................. [...] #根目錄路徑 o- backstores .................................................................................................. [...] #根目錄路徑下的backstores目錄 | o- block ...................................................................................... [Storage Objects: 0] #根目錄路徑下backstores目錄下的目錄文件block文件路徑，顯示當前系統上的磁盤設備 | o- fileio ..................................................................................... [Storage Objects: 0] | o- pscsi ...................................................................................... [Storage Objects: 0] | o- ramdisk .................................................................................... [Storage Objects: 0] o- iscsi ................................................................................................ [Targets: 0] #根目錄下的iscsi目錄，跟backstores同級別目錄 o- loopback ............................................................................................. [Targets: 0] #根目錄下的loopbak目錄，跟backstores和iscsi目錄同級別
/> /backstores/ #/backstores/+空格+TAB鍵即可顯示/backstores/ 可搭配使用的命令參數，其他目錄下的搭配命令雷同
bookmarks cd exit get help ls pwd refresh set status #當前目錄下可使用的命令
/> /backstores/block #/backstores/block+空格+TAB鍵即可顯示/backstores/block可搭配使用的命令參數，其他目錄下的搭配命令雷同，不再例舉
bookmarks cd create delete exit get help ls pwd refresh
set status
/> /backstores/block create #/backstores/block create+空格+TAB鍵可顯示當前命令可帶的參數
dev= name= readonly= wwn= #"dev="指定當前系統上的磁盤設備，"name="指定前面"dev="的自定義名稱，"readonly="指定是否只讀模式，“wwn=”指定"dev="的wwn號
/> /backstores/block create dev=/dev/sdb name=sun1 #把當前系統上的/dev/sdb磁盤命名為sun1的塊存儲設備
Created block storage object sun1 using /dev/sdb. #提示使用/dev/sdb創建塊設備對象sun1成功
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...]
| o- block ...................................................................................... [Storage Objects: 1]
| | o- sun1 .............................................................. [/dev/sdb (20.0GiB) write-thru deactivated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 0]
o- loopback ............................................................................................. [Targets: 0
/> /iscsi create iqn.2020-03.cn.192:server01 #配置ISCSITarget命名，命名在同一子網內確保是唯一的，命名格式為：iqn.yyyy-mm.<主機名反寫>:自定義名稱（自定義名稱內不能有下划線）
Created target iqn.2020-03.cn.192:server01. Created TPG 1. Global pref auto_add_default_portal=true Created default portal listening on all IPs (0.0.0.0), port 3260.
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...] #定義后端存儲設備，這個是物理磁盤
| o- block ...................................................................................... [Storage Objects: 1]
| | o- sun1 .............................................................. [/dev/sdb (20.0GiB) write-thru deactivated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 1] #定義客戶端連接信息
| o- iqn.2020-03.cn.192:server01 ........................................................................... [TPGs: 1]
| o- tpg1 ................................................................................... [no-gen-acls, no-auth]
| o- acls .............................................................................................. [ACLs: 0] #定義允許哪些客戶端來連接該塊設備
| o- luns .............................................................................................. [LUNs: 0] #定義lun映射信息
| o- portals ........................................................................................ [Portals: 1] #定義端口信息
| o- 0.0.0.0:3260 ......................................................................................... [OK] #默認監聽在本機的3260端口，可手動更改
o- loopback ............................................................................................. [Targets: 0]
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/acls create iqn.2020-03.cn.192:clinet #創建ACL允許ISCSI客戶機連接
Created Node ACL for iqn.2020-03.cn.192:clinet
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...]
| o- block ...................................................................................... [Storage Objects: 1] #定義后端存儲
| | o- sun1 .............................................................. [/dev/sdb (20.0GiB) write-thru deactivated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 1] #定義客戶端如何來連接后端存儲
| o- iqn.2020-03.cn.192:server01 ........................................................................... [TPGs: 1]
| o- tpg1 ................................................................................... [no-gen-acls, no-auth]
| o- acls .............................................................................................. [ACLs: 1]
| | o- iqn.2020-03.cn.192:clinet ................................................................ [Mapped LUNs: 0]
| o- luns .............................................................................................. [LUNs: 0]
| o- portals ........................................................................................ [Portals: 1]
| o- 0.0.0.0:3260 ......................................................................................... [OK] #默認監聽在本機的3260端口，可更改
o- loopback ............................................................................................. [Targets: 0]
配置驗證用戶名和密碼(該項為非必須)
/> cd iscsi/iqn.2020-03.cn.192:server01/tpg1/acls/iqn.2020-03.cn.192:clinet
/iscsi/iqn.20...cn.192:clinet> ls
o- iqn.2020-03.cn.192:clinet .......................................................................... [Mapped LUNs: 1] o- mapped_lun0 ................................................................................ [lun0 block/sun1 (rw)]
/iscsi/iqn.20...cn.192:clinet> set auth userid=admin
Parameter userid is now 'admin'.
/iscsi/iqn.20...cn.192:clinet> set auth password=Admin
Parameter password is now 'Admin'.
/iscsi/iqn.20...cn.192:clinet> cd /
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/luns create /backstores/block/sun1 #創建lun（target塊設備的邏輯單元）
Created LUN 0. Created LUN 0->0 mapping in node ACL iqn.2020-03.cn.192:clinet
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...]
| o- block ...................................................................................... [Storage Objects: 1]
| | o- sun1 ................................................................ [/dev/sdb (20.0GiB) write-thru activated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 1]
| o- iqn.2020-03.cn.192:server01 ........................................................................... [TPGs: 1]
| o- tpg1 ................................................................................... [no-gen-acls, no-auth]
| o- acls .............................................................................................. [ACLs: 1]
| | o- iqn.2020-03.cn.192:clinet ................................................................ [Mapped LUNs: 1]
| | o- mapped_lun0 ...................................................................... [lun0 block/sun1 (rw)]
| o- luns .............................................................................................. [LUNs: 1]
| | o- lun0 ........................................................... [block/sun1 (/dev/sdb) (default_tg_pt_gp)]
| o- portals ........................................................................................ [Portals: 1]
| o- 0.0.0.0:3260 ......................................................................................... [OK] #默認監聽在3260端口上，可手段更改
o- loopback ............................................................................................. [Targets: 0
#創建IP和端口，可默認不更改
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/portals/ delete 0.0.0.0 3260 #刪除默認的3260端口
Deleted network portal 0.0.0.0:3260
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/portals/ create 0.0.0.0 13260 #x修改默認3260端口為13260
Binding to INADDR_ANY (0.0.0.0) Created network portal 0.0.0.0:13260.
/> saveconfig #保存上面所作的配置信息
Last 10 configs saved in /etc/target/backup/. Configuration saved to /etc/target/saveconfig.json #保存配置信息的路徑
/> exit #退出
Global pref auto_save_on_exit=true Last 10 configs saved in /etc/target/backup/. Configuration saved to /etc/target/saveconfig.json
[root@SAN ~]# systemctl restart target && systemctl enable target
2.2 存儲客戶端10配置(即使用后端存儲的服務器)
[root@10 ~]# yum install -y iscsi-initiator-utils
[root@10 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-03.cn.192:clinet #iqn.2020-03.cn.192:clinet為在存儲服務器定義的允許來連接的名稱
[root@10 ~]# vim /etc/iscsi/iscsid.conf #取消下面三行內容的注釋，並將username和password修改成我們在存儲服務器上設置的用戶名和密碼，若未在存儲服務器上設置用戶名和密碼則跳過該步驟
57 node.session.auth.authmethod = CHAP 61 node.session.auth.username = admin 62 node.session.auth.password = Admin
[root@10 ~]# lsblk #此時只有主機10本地的磁盤
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 19G 0 part ├─centos-root 253:0 0 15G 0 lvm / └─centos-swap 253:1 0 4G 0 lvm [SWAP] sr0 11:0 1 1024M 0 rom
[root@10 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.0.105:13260 #尋找連接測試
192.168.0.105:13260,1 iqn.2020-03.cn.192:server01 #找到一個目標taget
[root@10 ~]# tree /var/lib/iscsi/ #發現target服務后，目標target的信息會寫入/var/lib/iscsi 目錄下
/var/lib/iscsi/
├── ifaces
├── isns
├── nodes
│ └── iqn.2020-03.cn.192:server01
│ └── 192.168.0.105,13260,1
│ └── default
├── send_targets
│ └── 192.168.0.105,13260
│ ├── iqn.2020-03.cn.192:server01,192.168.0.105,13260,1,default -> /var/lib/iscsi/nodes/iqn.2020-03.cn.192:server01/192.168.0.105,13260,1
│ └── st_config
├── slp
└── static
[root@10 ~]# iscsiadm -m node -l #登錄，登錄后就可以看到/dev/sdb磁盤了；登出為iscsiadm -m node --logout，登出后在本機使用lsblk就看不到/dev/sdb了
Logging in to [iface: default, target: iqn.2020-03.cn.192:server01, portal: 192.168.0.105,13260] (multiple) Login to [iface: default, target: iqn.2020-03.cn.192:server01, portal: 192.168.0.105,13260] successful. #提示登錄成功
[root@10 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 19G 0 part
├─centos-root 253:0 0 15G 0 lvm /
└─centos-swap 253:1 0 4G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sr0 11:0 1 1024M 0 rom
[root@10 ~]# systemctl start iscsid && systemctl enbale iscsid
此時我們在存儲服務器上查看連接情況：
[root@SAN ~]# targetcli
/> sessions
alias: 10 sid: 1 type: Normal session-state: LOGGED_IN
2.3 存儲客戶端11配置
#連接第2台服務器，使得10和11共用存儲 ，由於我們之前只定義了一個target，因此我們登陸的到存儲服務器在定義一個target--iqn.2020-03.cn.192:clinet1
(1).在存儲服務器上再添加一個target
[root@SAN ~]# targetcli
/> /iscsi/iqn.2020-03.cn.192:server01/tpg1/acls/ create iqn.2020-03.cn.192:clinet1
Created Node ACL for iqn.2020-03.cn.192:clinet1 Created mapped LUN 0.
/> ls
o- / ............................................................................................................. [...]
o- backstores .................................................................................................. [...]
| o- block ...................................................................................... [Storage Objects: 1]
| | o- sun1 ................................................................ [/dev/sdb (20.0GiB) write-thru activated]
| | o- alua ....................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ........................................................... [ALUA state: Active/optimized]
| o- fileio ..................................................................................... [Storage Objects: 0]
| o- pscsi ...................................................................................... [Storage Objects: 0]
| o- ramdisk .................................................................................... [Storage Objects: 0]
o- iscsi ................................................................................................ [Targets: 1]
| o- iqn.2020-03.cn.192:server01 ........................................................................... [TPGs: 1]
| o- tpg1 ................................................................................... [no-gen-acls, no-auth]
| o- acls .............................................................................................. [ACLs: 2]
| | o- iqn.2020-03.cn.192:clinet ................................................................ [Mapped LUNs: 1]
| | | o- mapped_lun0 ...................................................................... [lun0 block/sun1 (rw)]
| | o- iqn.2020-03.cn.192:clinet1 ............................................................... [Mapped LUNs: 1]
| | o- mapped_lun0 ...................................................................... [lun0 block/sun1 (rw)]
| o- luns .............................................................................................. [LUNs: 1]
| | o- lun0 ........................................................... [block/sun1 (/dev/sdb) (default_tg_pt_gp)]
| o- portals ........................................................................................ [Portals: 1]
| o- 0.0.0.0:13260 ........................................................................................ [OK]
o- loopback ............................................................................................. [Targets: 0]
/> saveconfig #保存配置
/> exit
[root@SAN ~]# systemctl restart target #重啟服務
(2).在主機11客戶端上的配置(跟2.2雷同)：
[root@11 ~]# yum install -y iscsi-initiator-utils
[root@11 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-03.cn.192:clinet1 #名稱是新定義的target
[root@11 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.0.105:13260
192.168.0.105:13260,1 iqn.2020-03.cn.192:server01
[root@11 ~]# iscsiadm -m node -l
#如果有多個硬盤，則需要指定登出的磁盤，格式為iscsiadm -m node -T iqn.2020-03.cn.192:server01 -u
Logging in to [iface: default, target: iqn.2020-03.cn.192:server01, portal: 192.168.0.105,13260] (multiple) Login to [iface: default, target: iqn.2020-03.cn.192:server01, portal: 192.168.0.105,13260] successful.
[root@11 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 49G 0 part
├─centos-root 253:0 0 45G 0 lvm /
└─centos-swap 253:1 0 4G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
└─sdb1 8:17 0 20G 0 part #可以發現此處已經做好了分區，在第二台服務器上只需要掛載使用即可
sr0 11:0 1 1024M 0 rom
[root@11 ~]# systemctl start iscsid && systemctl enable iscsid
[root@11 ~]# mkdir /iscsi_data
[root@11 ~]# mount /dev/sdb1 /iscsi_data/
[root@11 ~]# ls /iscsi_data/
hello.txt
此時我們在存儲服務器上查看連接情況：
[root@SAN ~]# targetcli
/> sessions
alias: 10 sid: 1 type: Normal session-state: LOGGED_IN alias: 11 sid: 2 type: Normal session-state: LOGGED_IN
此時我們在主機11上新建文件，並查看主機10是否能看到新文件
[root@11 ~]# mkdir /iscsi_data/from_11
[root@10 ~]# ls /data/
hello.txt #並未顯示主機11上新建的目錄
why??是因為我們使用的XFS 文件系統，XFS文件系統不支持多個客戶端同時使用， 使用GFS文件系統就可以同步。