分布式安裝部署MinIO

官方文檔地址：http://docs.minio.org.cn/docs/master/distributed-minio-quickstart-guide

前提條件：分布式Minio至少需要4個硬盤，使用分布式Minio自動引入了糾刪碼功能。

注意:

• 分布式Minio里所有的節點需要有同樣的access秘鑰和secret秘鑰，這樣這些節點才能建立聯接。為了實現這個，你需要在執行minio server命令之前，先將access秘鑰和secret秘鑰export成環境變量。
• 分布式Minio使用的磁盤里必須是干凈的，里面沒有數據。
• 下面示例里的IP僅供示例參考，你需要改成你真實用到的IP和文件夾路徑。
• 分布式Minio里的節點時間差不能超過3秒，你可以使用NTP 來保證時間一致。
• 在Windows下運行分布式Minio處於實驗階段，請悠着點使用。

這里演示使用一台主機上的4塊磁盤，每個磁盤1G

官方步驟：

export MINIO_ACCESS_KEY=<ACCESS_KEY>
export MINIO_SECRET_KEY=<SECRET_KEY>
minio server http://192.168.1.11/export1 http://192.168.1.12/export2 \
               http://192.168.1.13/export3 http://192.168.1.14/export4 \
               http://192.168.1.15/export5 http://192.168.1.16/export6 \
               http://192.168.1.17/export7 http://192.168.1.18/export8

實際操作步驟：
1.在一台主機上創建4個不同的路徑模擬4個不同的磁盤來進行

從執行結果來看，這種方式行不通

export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=12345678

mkdir -p /mnt/{data1,data2,data3,data4}
/usr/local/minio/bin/minio server http://192.168.20.102/mnt/data1 http://192.168.20.102/mnt/data2 http://192.168.20.102/mnt/data3 http://192.168.20.102/mnt/data4 --console-address ":9001"

# 操作完畢后提示信息如下
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
Formatting 1st pool, 1 set(s), 4 drives per set.
WARNING: Host 192.168.20.102 has more than 2 drives of set. A host failure will result in data becoming unavailable.

API: SYSTEM()
Time: 09:24:00 UTC 03/10/2022
Error: Disk `/mnt/data1` is part of root disk, will not be used (*errors.errorString)
       7: cmd/erasure-sets.go:1257:cmd.markRootDisksAsDown()
       6: cmd/format-erasure.go:792:cmd.initFormatErasure()
       5: cmd/prepare-storage.go:222:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:306:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:640:cmd.newObjectLayer()
       1: cmd/server-main.go:491:cmd.serverMain()

API: SYSTEM()
Time: 09:24:00 UTC 03/10/2022
Error: Disk `/mnt/data2` is part of root disk, will not be used (*errors.errorString)
       7: cmd/erasure-sets.go:1257:cmd.markRootDisksAsDown()
       6: cmd/format-erasure.go:792:cmd.initFormatErasure()
       5: cmd/prepare-storage.go:222:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:306:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:640:cmd.newObjectLayer()
       1: cmd/server-main.go:491:cmd.serverMain()

API: SYSTEM()
Time: 09:24:00 UTC 03/10/2022
Error: Disk `/mnt/data3` is part of root disk, will not be used (*errors.errorString)
       7: cmd/erasure-sets.go:1257:cmd.markRootDisksAsDown()
       6: cmd/format-erasure.go:792:cmd.initFormatErasure()
       5: cmd/prepare-storage.go:222:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:306:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:640:cmd.newObjectLayer()
       1: cmd/server-main.go:491:cmd.serverMain()

API: SYSTEM()
Time: 09:24:00 UTC 03/10/2022
Error: Disk `/mnt/data4` is part of root disk, will not be used (*errors.errorString)
       7: cmd/erasure-sets.go:1257:cmd.markRootDisksAsDown()
       6: cmd/format-erasure.go:792:cmd.initFormatErasure()
       5: cmd/prepare-storage.go:222:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:306:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:640:cmd.newObjectLayer()
       1: cmd/server-main.go:491:cmd.serverMain()
Formatting 1st pool, 1 set(s), 4 drives per set.
WARNING: Host 192.168.20.102 has more than 2 drives of set. A host failure will result in data becoming unavailable.

改進步驟：

# 根據上一步的執行提示，更新設置的環境變量名
export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=12345678

# 新加四塊磁盤，每個磁盤空間1G，不分區，直接格式化后掛載

# 不重啟，直接刷新磁盤數據總線，獲取新加的磁盤
for host in $(ls /sys/class/scsi_host) ; do echo "- - -" > /sys/class/scsi_host/$host/scan; done

# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   40G  0 disk 
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   39G  0 part 
  ├─centos-root 253:0    0 35.1G  0 lvm  /
  └─centos-swap 253:1    0  3.9G  0 lvm  [SWAP]
sdb               8:16   0    1G  0 disk 
sdc               8:32   0    1G  0 disk 
sdd               8:48   0    1G  0 disk 
sde               8:64   0    1G  0 disk 

[root@myrabbit3 ~]# mkfs.ext4 /dev/sdb
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb is entire device, not just one partition!
無論如何也要繼續? (y,n) y
文件系統標簽=
OS type: Linux
塊大小=4096 (log=2)
分塊大小=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
第一個數據塊=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376

Allocating group tables: 完成                            
正在寫入inode表: 完成                            
Creating journal (8192 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

[root@myrabbit3 ~]# mkfs.ext4 /dev/sdc
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdc is entire device, not just one partition!
無論如何也要繼續? (y,n) y
文件系統標簽=
OS type: Linux
塊大小=4096 (log=2)
分塊大小=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
第一個數據塊=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376

Allocating group tables: 完成                            
正在寫入inode表: 完成                            
Creating journal (8192 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

[root@myrabbit3 ~]# mkfs.ext4 /dev/sdd
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdd is entire device, not just one partition!
無論如何也要繼續? (y,n) y
文件系統標簽=
OS type: Linux
塊大小=4096 (log=2)
分塊大小=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
第一個數據塊=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376

Allocating group tables: 完成                            
正在寫入inode表: 完成                            
Creating journal (8192 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

[root@myrabbit3 ~]# mkfs.ext4 /dev/sde
mke2fs 1.42.9 (28-Dec-2013)
/dev/sde is entire device, not just one partition!
無論如何也要繼續? (y,n) y
文件系統標簽=
OS type: Linux
塊大小=4096 (log=2)
分塊大小=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
第一個數據塊=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376

Allocating group tables: 完成                            
正在寫入inode表: 完成                            
Creating journal (8192 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

[root@myrabbit3 ~]# mount /dev/sdb /mnt/data1/
[root@myrabbit3 ~]# mount /dev/sdc /mnt/data2/
[root@myrabbit3 ~]# mount /dev/sdd /mnt/data3/
[root@myrabbit3 ~]# mount /dev/sde /mnt/data4/

# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   40G  0 disk 
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   39G  0 part 
  ├─centos-root 253:0    0 35.1G  0 lvm  /
  └─centos-swap 253:1    0  3.9G  0 lvm  [SWAP]
sdb               8:16   0    1G  0 disk /mnt/data1
sdc               8:32   0    1G  0 disk /mnt/data2
sdd               8:48   0    1G  0 disk /mnt/data3
sde               8:64   0    1G  0 disk /mnt/data4

# 記得把磁盤掛載信息寫道/etc/fstab文件中

# /usr/local/minio/bin/minio server http://192.168.20.102/mnt/data1 http://192.168.20.102/mnt/data2 http://192.168.20.102/mnt/data3 http://192.168.20.102/mnt/data4 --console-address ":9001"
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
Automatically configured API requests per node based on available memory on the system: 55
Status:         4 Online, 0 Offline. 
API: http://192.168.20.102:9000  http://127.0.0.1:9000   
RootUser: admin 
RootPass: 12345678 

Console: http://192.168.20.102:9001 http://127.0.0.1:9001  
RootUser: admin 
RootPass: 12345678 

Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc alias set myminio http://192.168.20.102:9000 admin 12345678

Documentation: https://docs.min.io

擴展現有的分布式集群

例如我們是通過區的方式啟動MinIO集群，命令行如下：

export MINIO_ACCESS_KEY=<ACCESS_KEY>
export MINIO_SECRET_KEY=<SECRET_KEY>
minio server http://host{1...32}/export{1...32}

MinIO支持通過命令，指定新的集群來擴展現有集群（糾刪碼模式），命令行如下：

export MINIO_ACCESS_KEY=<ACCESS_KEY>
export MINIO_SECRET_KEY=<SECRET_KEY>
minio server http://host{1...32}/export{1...32} http://host{33...64}/export{1...32}

新的對象上傳請求會自動分配到最少使用的集群上。通過以上擴展策略，您就可以按需擴展您的集群。重新配置后重啟集群，會立即在集群中生效，並對現有集群無影響。如上命令中，我們可以把原來的集群看做一個區，新增集群看做另一個區，新對象按每個區域中的可用空間比例放置在區域中。在每個區域內，基於確定性哈希算法確定位置。

說明: 您添加的每個區域必須具有與原始區域相同的磁盤數量（糾刪碼集）大小，以便維持相同的數據冗余SLA。 例如，第一個區有8個磁盤，您可以將集群擴展為16個、32個或1024個磁盤的區域，您只需確保部署的SLA是原始區域的倍數即可。

多個節點的存儲容量和就是分布式Minio的存儲容量。