序言
更換系統后需要一個網絡文件存儲用於備份文件,本想用NFS多方便,但是timeshift不支持網絡存儲,備份路徑必須是一個塊存儲設備,
但是你還必須分好文件系統,這不是多此一舉???反正我只用rsync進行同步
於是使用iscsi滿足它的需求,因為iscsi連接后獲取到的就是一個塊存儲設備
在配置過程中參考了很多網友寫的博客 很有參考價值 非常感謝
《Linux下搭建iSCSI共享存儲的方法 Linux-IO Target 方式 Debian9.5下實現》
他寫的很全面,我在這里只記錄我用到的部分
簡述
iSCSI,通過TCP/IP網絡傳輸SCSI命令提供對存儲設備的塊級訪問,屬於SAN存儲,因此又叫IP-SAN,默認端口3260/TCP。
也就是說通過iSCS獲取到的是一個真實的或者虛擬的存儲設備,連接之后會多出一個硬件設備,就像一塊本機硬盤一樣,你需要在上面建立文件系統后才能使用。
而NFS、SMB等獲取到的只是一個掛載點,與文件系統無關,連接后即可以使用。
客戶端稱為initiators,服務器上的存儲目標稱為target,客戶端發現服務器上存儲目標的過程叫discovery。
安裝與配置
基於Debian/Ubuntu的發行版可以直接通過apt命令安裝
服務器 | 客戶端 | |
安裝 | sudo apt install targetcli-fb | sudo apt install open-iscsi |
配置程序 | targetcli | iscsiadm |
相關服務 | targetclid.server | iscsid.service |
配置文件目錄 | /etc/rtslib-fb-target/ | /etc/iscsi |
服務器配置 以root權限運行targetcli
sudo targetcli
該工具的使用類似linux下shell的使用
ls查看目錄
set 配置參數
saceconfig 保存配置文件 默認位置 /etc/rtslib-fb-target/saveconfig.json
默認設置每次退出后會自動保存配置到該位置,之前的配置會備份到/etc/rtslib-fb-target/backup中,
並且使用gzip壓縮歸檔,文件后綴名為gz,如果需要使用targetctl恢復,需要先使用gzip解壓后才能導入配置
/> ls
o- / .................................................................................... [...]
o- backstores ......................................................................... [...]
| o- block ............................................................. [Storage Objects: 0]
| o- fileio ............................................................ [Storage Objects: 0]
| o- pscsi ............................................................. [Storage Objects: 0]
| o- ramdisk ........................................................... [Storage Objects: 0]
o- iscsi ....................................................................... [Targets: 0]
o- loopback .................................................................... [Targets: 0]
o- vhost ....................................................................... [Targets: 0]
o- xen-pvscsi .................................................................. [Targets: 0]
/>
簡單的使用只需要了解兩個目錄backstorages和iscsi
其中backstorages是后端存儲
block是塊存儲設備,簡單的說就是連接到本機的硬盤驅動器
fileio是文件存儲,簡單地說就像虛擬機的鏡像文件,raw,qcow2那樣的鏡像
pscsi是連接到本機的scsi設備,我還沒用過這種設備,只在書上見過
ramdisk就是內存盤,linux上的tmpfs就屬於這種,如果服務器內存很大,可以划出一部分內存做臨時存儲用
iscsi里面就是要配置的存儲目標target
1. 添加存儲設備
我使用qemu-img創建一塊磁盤鏡像作為存儲設備
qemu-img create -f qcow2 -o preallocation=falloc /mnt/ext4linux/var/iscsi/storage-1.qcow2 20G
其中 preallocation=falloc 參數的作用是立即分配空間 但不填充 參考 《qemu-img create創建磁盤》
使用targetcli 他提供了彩色的文字方便檢查配置 綠色的配置為有效配置 紅色的為無效配置 白色的為默認配置
要注意不同的目錄下可用的命令和選項不相同
進入/backstores/fileio目錄下,使用create命令添加存儲設備
/> backstores/fileio create name=demo1 file_or_dev=/mnt/ext4linux/var/iscsi/storage-1.qcow2
Created fileio demo1 with size 21478375424
/> ls
o- / .................................................................................... [...]
o- backstores ......................................................................... [...]
| o- block ............................................................. [Storage Objects: 0]
| o- fileio ............................................................ [Storage Objects: 1]
| | o- demo1 .... [/mnt/ext4linux/var/iscsi/storage-1.qcow2 (20.0GiB) write-back deactivated]
| | o- alua .............................................................. [ALUA Groups: 1]
| | o- default_tg_pt_gp .................................. [ALUA state: Active/optimized]
| o- pscsi ............................................................. [Storage Objects: 0]
| o- ramdisk ........................................................... [Storage Objects: 0]
o- iscsi ....................................................................... [Targets: 0]
o- loopback .................................................................... [Targets: 0]
o- vhost ....................................................................... [Targets: 0]
o- xen-pvscsi .................................................................. [Targets: 0]
/>
2. 添加存儲目標
進入/iscsi目錄后,使用create命令添加存儲目標
/> iscsi/ create iqn.2021-12.cn.erika.iscsi:erika-ge.storage
Created target iqn.2021-12.cn.erika.iscsi:erika-ge.storage.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/> ls
o- / .................................................................................... [...]
o- backstores ......................................................................... [...]
| o- block ............................................................. [Storage Objects: 0]
| o- fileio ............................................................ [Storage Objects: 1]
| | o- demo1 .... [/mnt/ext4linux/var/iscsi/storage-1.qcow2 (20.0GiB) write-back deactivated]
| | o- alua .............................................................. [ALUA Groups: 1]
| | o- default_tg_pt_gp .................................. [ALUA state: Active/optimized]
| o- pscsi ............................................................. [Storage Objects: 0]
| o- ramdisk ........................................................... [Storage Objects: 0]
o- iscsi ....................................................................... [Targets: 1]
| o- iqn.2021-12.cn.erika.iscsi:erika-ge.storage .................................. [TPGs: 1]
| o- tpg1 .......................................................... [no-gen-acls, no-auth]
| o- acls ..................................................................... [ACLs: 0]
| o- luns ..................................................................... [LUNs: 0]
| o- portals ............................................................... [Portals: 1]
| o- 0.0.0.0:3260 ................................................................ [OK]
o- loopback .................................................................... [Targets: 0]
o- vhost ....................................................................... [Targets: 0]
o- xen-pvscsi .................................................................. [Targets: 0]
/>
創建的時候如果不加IQN名稱 則會生成一個隨機的 如果指定一個 需要遵循一定的格式 具體參考RFC3720
前面的iqn是固定的 必須要有 后面的日期通常為創建該存儲目標的日期 然后是反向域名 就和DNS的差不多
沒有域名就自己編一個 又不用去注冊 只為了作為標識 然后跟個冒號 冒號后面是該存儲目標的識別名稱
3. 關聯存儲設備
進入/iscsi/你設置的iqn目標/tpg1/luns目錄下,使用create命令關聯存儲設備
/> iscsi/iqn.2021-12.cn.erika.iscsi:erika-ge.storage/tpg1/luns create /backstores/fileio/demo1
Created LUN 0.
/> ls
o- / .................................................................................... [...]
o- backstores ......................................................................... [...]
| o- block ............................................................. [Storage Objects: 0]
| o- fileio ............................................................ [Storage Objects: 1]
| | o- demo1 ...... [/mnt/ext4linux/var/iscsi/storage-1.qcow2 (20.0GiB) write-back activated]
| | o- alua .............................................................. [ALUA Groups: 1]
| | o- default_tg_pt_gp .................................. [ALUA state: Active/optimized]
| o- pscsi ............................................................. [Storage Objects: 0]
| o- ramdisk ........................................................... [Storage Objects: 0]
o- iscsi ....................................................................... [Targets: 1]
| o- iqn.2021-12.cn.erika.iscsi:erika-ge.storage .................................. [TPGs: 1]
| o- tpg1 .......................................................... [no-gen-acls, no-auth]
| o- acls ..................................................................... [ACLs: 0]
| o- luns ..................................................................... [LUNs: 1]
| | o- lun0 [fileio/demo1 (/mnt/ext4linux/var/iscsi/storage-1.qcow2) (default_tg_pt_gp)]
| o- portals ............................................................... [Portals: 1]
| o- 0.0.0.0:3260 ................................................................ [OK]
o- loopback .................................................................... [Targets: 0]
o- vhost ....................................................................... [Targets: 0]
o- xen-pvscsi .................................................................. [Targets: 0]
/>
這個時候還不能使用客戶端訪問存儲,因為沒有配置認證,先寫一下客戶端發現並連接服務器的方法
# 發現目標 如果服務器端口號是默認的3260 則可以省略
sudo iscsiadm -m discovery -t sendtargets -p <ip:port>
# 登錄目標節點 iqn寫目標節點的iqn
sudo iscsiadm -m node -T <iqn> -p <ip:port> -l
# 登出目標節點 記得先umount 防止數據丟失
sudo iscsiadm -m node -T <iqn> -p <ip:port> -u
# 刪除目標節點 這里是刪除本機對該目標節點的記錄
sudo iscaiadm -m node -T <iqn> -o delete
這里我們先嘗試連接看看會如何
e@kvm-ubuntu-01:/etc/iscsi$ sudo iscsiadm -m discovery -t sendtargets -p 172.20.0.65
172.20.0.65:3260,1 iqn.2021-12.cn.erika.iscsi:erika-ge.storage
e@kvm-ubuntu-01:/etc/iscsi$ sudo iscsiadm -m node -T iqn.2021-12.cn.erika.iscsi:erika-ge.storage -l
Logging in to [iface: default, target: iqn.2021-12.cn.erika.iscsi:erika-ge.storage, portal: 172.20.0.65,3260] (multiple)
iscsiadm: Could not login to [iface: default, target: iqn.2021-12.cn.erika.iscsi:erika-ge.storage, portal: 172.20.0.65,3260].
iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
iscsiadm: Could not log into all portals
需要在服務器上配置一下認證,即使不想使用認證也需要配置
命令很簡單,進入/iscsi/你設置的iqn/tpg1目錄下,使用set命令設置generate_node_acls為1
/> iscsi/iqn.2021-12.cn.erika.iscsi:erika-ge.storage/tpg1/ set attribute generate_node_acls=1
Parameter generate_node_acls is now '1'.
/> ls
o- / .................................................................................... [...]
o- backstores ......................................................................... [...]
| o- block ............................................................. [Storage Objects: 0]
| o- fileio ............................................................ [Storage Objects: 1]
| | o- demo1 ...... [/mnt/ext4linux/var/iscsi/storage-1.qcow2 (20.0GiB) write-back activated]
| | o- alua .............................................................. [ALUA Groups: 1]
| | o- default_tg_pt_gp .................................. [ALUA state: Active/optimized]
| o- pscsi ............................................................. [Storage Objects: 0]
| o- ramdisk ........................................................... [Storage Objects: 0]
o- iscsi ....................................................................... [Targets: 1]
| o- iqn.2021-12.cn.erika.iscsi:erika-ge.storage .................................. [TPGs: 1]
| o- tpg1 ............................................................. [gen-acls, no-auth]
| o- acls ..................................................................... [ACLs: 0]
| o- luns ..................................................................... [LUNs: 1]
| | o- lun0 [fileio/demo1 (/mnt/ext4linux/var/iscsi/storage-1.qcow2) (default_tg_pt_gp)]
| o- portals ............................................................... [Portals: 1]
| o- 0.0.0.0:3260 ................................................................ [OK]
o- loopback .................................................................... [Targets: 0]
o- vhost ....................................................................... [Targets: 0]
o- xen-pvscsi .................................................................. [Targets: 0]
/>
設置完后查看當前配置,會發現該iqn下的acls后面的中括號里 no-gen-acls變成了gen-acls,而且文字顏色變成了綠色,說明可以訪問了
但要注意,只配置generate_node_acls的話該目標是只讀的狀態,應該是為了安全,畢竟不需要認證就能訪問
在該iqn下的tpg1目錄下查看屬性 get attribute 就能看到有一條
generate_node_acls=1
--------------------
If set to 1, allow all initiators to login (i.e. demo mode).
如果設置成1,則該節點為demo模式,如果需要寫入,還需要設置屬性demo_mode_write_protect=0,這樣才能寫入,然后再次連接
e@kvm-ubuntu-01:/etc/iscsi$ sudo iscsiadm -m node -T iqn.2021-12.cn.erika.iscsi:erika-ge.storage -l
Logging in to [iface: default, target: iqn.2021-12.cn.erika.iscsi:erika-ge.storage, portal: 172.20.0.65,3260] (multiple)
Login to [iface: default, target: iqn.2021-12.cn.erika.iscsi:erika-ge.storage, portal: 172.20.0.65,3260] successful.
客戶端查看當前的磁盤列表
e@kvm-ubuntu-01:/$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 86.9M 1 loop /snap/core/4917
sda 8:0 0 20G 0 disk
sr0 11:0 1 1024M 0 rom
vda 252:0 0 30G 0 disk
├─vda1 252:1 0 1M 0 part
└─vda2 252:2 0 30G 0 part /
上面20G未分區的sda就是連接到的目標節點,之后可以使用fdisk之類的軟件進行分區,然后掛載使用,就像本地硬盤一樣使用
4. 配置認證
請注意,iscsi的認證可能有緩存,因此如果下面的步驟如果發生認證錯誤,請在客戶端刪除發現記錄后重新發現然后嘗試連接
在上面的操作完成后,任何一台機器都能掛載該節點,完全沒有安全性可言,因此我們需要添加認證
認證分兩類:發現認證和登錄(會話)認證,這兩類認證又各自有單向認證和雙向認證
雙向認證是在單向認證的基礎上,服務器向客戶端確認認證信息用的,在一定程度上能避免中間人攻擊
4.1. 發現認證
發現認證用於客戶端發現服務器上可用的節點,屬於全局認證,需要在/iscsi目錄下進行
/> cd iscsi/
/iscsi> get discovery_auth
DISCOVERY_AUTH CONFIG GROUP
===========================
enable=False # 用來啟用或者禁用發現認證
-----------
The enable discovery_auth parameter.
mutual_password= # 雙向認證的密碼
----------------
The mutual_password discovery_auth parameter.
mutual_userid= # 雙向認證的用戶名
--------------
The mutual_userid discovery_auth parameter.
password= # 單向認證的密碼
-------------------
The password discovery_auth parameter.
userid= # 雙向認證的密碼
-------------
The userid discovery_auth parameter.
我這里只配置單向認證
/iscsi> set discovery_auth enable=1
Parameter enable is now 'True'.
/iscsi> set discovery_auth userid=admin
Parameter userid is now 'admin'.
/iscsi> set discovery_auth password=admin@r00t
Parameter password is now 'admin@r00t'.
/iscsi> get discovery_auth
DISCOVERY_AUTH CONFIG GROUP
===========================
enable=True
-----------
The enable discovery_auth parameter.
mutual_password=
----------------
The mutual_password discovery_auth parameter.
mutual_userid=
--------------
The mutual_userid discovery_auth parameter.
password=admin@r00t
-------------------
The password discovery_auth parameter.
userid=admin
------------
The userid discovery_auth parameter.
配置完后客戶端也需要做相應的配置,修改文件/etc/iscsi/iscsid.conf
# To enable CHAP authentication for a discovery session to the target
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
discovery.sendtargets.auth.authmethod = CHAP # 去掉注釋 啟用發現認證
# To set a discovery session CHAP username and password for the initiator
# authentication by the target(s), uncomment the following lines:
discovery.sendtargets.auth.username = admin # 去掉注釋 寫上用戶名
discovery.sendtargets.auth.password = admin@r00t # 去掉注釋 寫上密碼
# To set a discovery session CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#discovery.sendtargets.auth.username_in = username_in
#discovery.sendtargets.auth.password_in = password_in
# 這兩行是發現認證的雙向認證的配置
重啟iscsid服務,執行發現命令,成功后結果如下
e@kvm-ubuntu-01:~$ sudo iscsiadm -m discovery -t sendtargets -p 172.20.0.65
172.20.0.65:3260,1 iqn.2021-12.cn.erika.iscsi:erika-ge.storage
如果發現認證失敗,則會如下
iscsiadm: Login failed to authenticate with target
iscsiadm: discovery login to 172.20.0.65 rejected: initiator failed authorization
iscsiadm: Could not perform SendTargets discovery: iSCSI login failed due to authorization failure
4.2. 配置登錄認證
一個服務器下可以配置多個目標節點(target),如果不配置登錄認證,則通過發現認證的客戶端可以連接所有的目標節點
舉個例子,一個小公司使用iscsi做中央存儲服務器,分給每個員工1T空間作為網絡存儲。
如果不設置登錄認證,則每個員工都可以隨意連接該服務器上的節點。
配置過程如下,進入/iscsi/你設置的iqn目標/tpg1/ 目錄下
使用set命令設置屬性 set attribute generate_node_acls=0,因為要自定義訪問規則,因此要把自動生成的訪問規則去掉
使用set命令設置屬性 set attribute authentication=1,目的是啟用登錄認證
然后進入acls下,使用create命令創建允許訪問的客戶端的iqn,這個iqn一會要填在客戶端的/etc/iscsi/initiatorname.iscsi文件中
然后執行ls命令會發現該iqn后面的中括號里面的文字是紅色的,說明缺少配置信息,配置無效,因為這時還沒有配置認證信息,繼續
進入到創建的客戶讀的iqn目錄下,使用set命令設置單向認證信息
/> cd iscsi/iqn.2021-12.cn.erika.iscsi:erika-ge.storage/tpg1/
/iscsi/iqn.20....storage/tpg1> ls
o- tpg1 ........................................................... [no-gen-acls, auth per-acl]
o- acls ........................................................................... [ACLs: 0]
o- luns ........................................................................... [LUNs: 1]
| o- lun0 ...... [fileio/demo1 (/mnt/ext4linux/var/iscsi/storage-1.qcow2) (default_tg_pt_gp)]
o- portals ..................................................................... [Portals: 1]
o- 0.0.0.0:3260 ...................................................................... [OK]
/iscsi/iqn.20....storage/tpg1> set attribute authentication=1
Parameter authentication is now '1'.
/iscsi/iqn.20....storage/tpg1> set attribute generate_node_acls=0
Parameter generate_node_acls is now '0'.
/iscsi/iqn.20....storage/tpg1> acls/ create iqn.2021-12.cn.erika.iscsi:kvm-ubuntu.client
Created Node ACL for iqn.2021-12.cn.erika.iscsi:kvm-ubuntu.client
Created mapped LUN 0.
/iscsi/iqn.20....storage/tpg1> cd acls/iqn.2021-12.cn.erika.iscsi:kvm-ubuntu.client/
/iscsi/iqn.20...ubuntu.client> set auth userid=e
Parameter userid is now 'e'.
/iscsi/iqn.20...ubuntu.client> set auth password=admin@r00t
Parameter password is now 'admin@r00t'.
/iscsi/iqn.20...ubuntu.client> cd iscsi/iqn.2021-12.cn.erika.iscsi:erika-ge.storage/tpg1/
/iscsi/iqn.20....storage/tpg1> ls
o- tpg1 ........................................................... [no-gen-acls, auth per-acl]
o- acls ........................................................................... [ACLs: 1]
| o- iqn.2021-12.cn.erika.iscsi:kvm-ubuntu.client .............. [1-way auth, Mapped LUNs: 1]
| o- mapped_lun0 ................................................. [lun0 fileio/demo1 (rw)]
o- luns ........................................................................... [LUNs: 1]
| o- lun0 ...... [fileio/demo1 (/mnt/ext4linux/var/iscsi/storage-1.qcow2) (default_tg_pt_gp)]
o- portals ..................................................................... [Portals: 1]
o- 0.0.0.0:3260 ...................................................................... [OK]
/iscsi/iqn.20....storage/tpg1> cd /
/> ls
o- / .................................................................................... [...]
o- backstores ......................................................................... [...]
| o- block ............................................................. [Storage Objects: 0]
| o- fileio ............................................................ [Storage Objects: 1]
| | o- demo1 ...... [/mnt/ext4linux/var/iscsi/storage-1.qcow2 (20.0GiB) write-back activated]
| | o- alua .............................................................. [ALUA Groups: 1]
| | o- default_tg_pt_gp .................................. [ALUA state: Active/optimized]
| o- pscsi ............................................................. [Storage Objects: 0]
| o- ramdisk ........................................................... [Storage Objects: 0]
o- iscsi ...................................................... [1-way disc auth, Targets: 1]
| o- iqn.2021-12.cn.erika.iscsi:erika-ge.storage .................................. [TPGs: 1]
| o- tpg1 ..................................................... [no-gen-acls, auth per-acl]
| o- acls ..................................................................... [ACLs: 1]
| | o- iqn.2021-12.cn.erika.iscsi:kvm-ubuntu.client ........ [1-way auth, Mapped LUNs: 1]
| | o- mapped_lun0 ........................................... [lun0 fileio/demo1 (rw)]
| o- luns ..................................................................... [LUNs: 1]
| | o- lun0 [fileio/demo1 (/mnt/ext4linux/var/iscsi/storage-1.qcow2) (default_tg_pt_gp)]
| o- portals ............................................................... [Portals: 1]
| o- 0.0.0.0:3260 ................................................................ [OK]
o- loopback .................................................................... [Targets: 0]
o- vhost ....................................................................... [Targets: 0]
o- xen-pvscsi .................................................................. [Targets: 0]
/>
設置完成后,配置的客戶端iqn后面的中括號里面的文字會變成綠色
然后修改客戶端的配置文件/etc/iscsi/initiatorname.iscsi
將InitiatorName=后面的改成上面配置的客戶端的iqn
然后修改客戶端的配置文件/etc/iscsi/iscsid.conf
# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
node.session.auth.authmethod = CHAP # 去掉注釋 啟用登錄認證
# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
node.session.auth.username = e # 去掉注釋 寫上用戶名
node.session.auth.password = admin@r00t # 去掉注釋 寫上密碼
重啟服務iscsid,刪除節點記錄后重新發現並登錄
e@kvm-ubuntu-01:/etc/iscsi$ sudo iscsiadm -m node -T iqn.2021-12.cn.erika.iscsi:erika-ge.storage -o delete
e@kvm-ubuntu-01:/etc/iscsi$ sudo iscsiadm -m discovery -t sendtargets -p 172.20.0.65
172.20.0.65:3260,1 iqn.2021-12.cn.erika.iscsi:erika-ge.storage
e@kvm-ubuntu-01:/etc/iscsi$ sudo iscsiadm -m node -T iqn.2021-12.cn.erika.iscsi:erika-ge.storage -l
Logging in to [iface: default, target: iqn.2021-12.cn.erika.iscsi:erika-ge.storage, portal: 172.20.0.65,3260] (multiple)
Login to [iface: default, target: iqn.2021-12.cn.erika.iscsi:erika-ge.storage, portal: 172.20.0.65,3260] successful.
到這里就結束了 謝謝觀看