Minio 集群擴容存儲空間，配合nginx 負載反向代理后端minio 集群服務器，提升高可用性

本文轉載自查看原文 2020-03-22 11:02 5542

環境：Centos 7

軟件：minio,Etcd

需求：通過聯盟兩個集群實例，實現水平擴容存儲空間問題；

服務器使用阿里雲，一共4台服務器(官方說明最好4台服務器做分布式，測試節省服務器所以我們使用兩台服務器每台服務器兩個目錄，做一個假服務器代替，真實環境條件允許最好4台服務器操作)：

集群	Minio-1	Minio-2
IP 地址	172.31.78.244 172.31.78.243	172.31.78.241 172.31.78.242
計算名稱地址	topsysfile.server.1.itusye.co topsysfile.server.2.itusye.co	topsysfile.server.3.itusye.co topsysfile.server.4.itusye.co
目錄地址	/Data/minio/{data1,data2}	/Data/minio/{data1,data2}

官方說明：
可擴容：不同MinIO集群可以組成聯邦，並形成一個全局的命名空間，並跨越多個數據中心，但不支持動態擴容；

https://docs.min.io/docs/minio-federation-quickstart-guide.html 官方參考文檔

1. 由於多個集群使用Etcd 聯邦，所以我們先需要安裝配置ETCD 服務，etcd 與zookeeper 服務器都是奇數性3，5，7 模式增加，etcd 安裝配置我就簡單跳過了，列出安裝與配置；

[root@Minio-Storage-Services ~]# yum -y install etcd
[root@Minio-Storage-Services ~]# cat /etc/etcd/etcd.conf | grep -v '#'
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://172.31.78.244:2380"  #內網通迅
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"     #外網通迅
ETCD_NAME="top-etcd-01"    #這里是集群連接地址，請注意與cluster 對應，與zookeeper myid  是一樣的原理
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.78.244:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_CLUSTER="top-etcd-01=http://172.31.78.244:2380,top-etcd-02=http://172.31.78.243:2380,top-etcd-03=http://172.31.78.241:2380"     #etcd 集群連接分配地址，注意三台服務器一樣
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_STRICT_RECONFIG_CHECK="true"

以上服務器配置是172.31.78.244 這台

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:16:3e:08:72:67 brd ff:ff:ff:ff:ff:ff
    inet 172.31.78.243/20 brd 172.31.79.255 scope global dynamic eth0
       valid_lft 315039419sec preferred_lft 315039419sec
[root@Minio-Storage-Services ~]# cat /etc/etcd/etcd.conf  | grep -v '#'
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://172.31.78.243:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_NAME="top-etcd-02"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.78.243:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_CLUSTER="top-etcd-01=http://172.31.78.244:2380,top-etcd-02=http://172.31.78.243:2380,top-etcd-03=http://172.31.78.241:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_STRICT_RECONFIG_CHECK="true"
[root@Minio-Storage-Services ~]#

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:16:3e:08:74:04 brd ff:ff:ff:ff:ff:ff
    inet 172.31.78.241/20 brd 172.31.79.255 scope global dynamic eth0
       valid_lft 315040516sec preferred_lft 315040516sec
[root@Minio-Storage-Services ~]# cat /etc/etcd/etcd.conf | grep -v '#'
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://172.31.78.241:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_NAME="top-etcd-03"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.78.241:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_CLUSTER="top-etcd-01=http://172.31.78.244:2380,top-etcd-02=http://172.31.78.243:2380,top-etcd-03=http://172.31.78.241:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_STRICT_RECONFIG_CHECK="true"
[root@Minio-Storage-Services ~]#

etcd 已安裝完成；

2. 下載minio , 4台配置服務器同樣的配置如下；

[root@Minio-Storage-Services ~]#  wget https://dl.min.io/server/minio/release/linux-amd64/minio
[root@Minio-Storage-Services ~]# chmod +x minio 
[root@Minio-Storage-Services ~]# mv minio /usr/local/bin/
[root@Minio-Storage-Services ~]# mkdir /usr/local/minio{bin,config} -p
[root@Minio-Storage-Services ~]# mkdir /Data/minio/{data1,data2} -p
[root@Minio-Storage-Services ~]# firewall-cmd --zone=public --add-port=2380/tcp --add-port=2379/tcp --permanent && firewall-cmd --reload
[root@Minio-Storage-Services ~]# firewall-cmd --zone=public --add-port=9000/tcp --permanent
[root@Minio-Storage-Services ~]# systemctl restart firewalld

3. 編輯啟動腳本，兩個集群，兩份配置不一樣，由於我配置了hosts，所以etcd 連接使用了域名

[root@Minio-Storage-Services ~]# more /etc/hosts
::1     localhost       localhost.localdomain   localhost6      localhost6.localdomain6
127.0.0.1       localhost       localhost.localdomain   localhost4      localhost4.localdomain4

172.31.78.241   Minio-Storage-Services  Minio-Storage-Services


172.31.78.244   topsysfile.server.1.itusye.co
172.31.78.243   topsysfile.server.2.itusye.co
172.31.78.241   topsysfile.server.3.itusye.co
172.31.78.242   topsysfile.server.4.itusye.co

[root@Minio-Storage-Services ~]#

Minio-1集群配置，兩台服務器同樣的配置；

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:16:3e:08:77:6b brd ff:ff:ff:ff:ff:ff
    inet 172.31.78.244/20 brd 172.31.79.255 scope global dynamic eth0
       valid_lft 315039437sec preferred_lft 315039437sec
[root@Minio-Storage-Services ~]# more /usr/local/minio/bin/run.sh 
#!/bin/bash
export MINIO_ACCESS_KEY=topsysfile
export MINIO_SECRET_KEY=Hao123baidu!

export MINIO_ETCD_ENDPOINTS="http://topsysfile.server.1.海itusye.co:2380,http://topsysfile.server.2.itusye.co:2380,http://topsysfile.server.3.itusye.co:2380"

export MINIO_DOMAIN=topfile.storage.minio.itusye.co
export MINIO_PUBLIC_IPS=172.31.78.244,172.31.78.243

minio server --config-dir /usr/local/minio/config/ \
http://172.31.78.244/Data/minio/data1 \
http://172.31.78.244/Data/minio/data2 \
http://172.31.78.243/Data/minio/data1 \
http://172.31.78.243/Data/minio/data2

Minio-2 集群配置，兩台服務器同樣的配置；

       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:16:3e:08:74:04 brd ff:ff:ff:ff:ff:ff
    inet 172.31.78.241/20 brd 172.31.79.255 scope global dynamic eth0
       valid_lft 315034589sec preferred_lft 315034589sec
[root@Minio-Storage-Services ~]# more /usr/local/minio/bin/run.sh 
#!/bin/bash
export MINIO_ACCESS_KEY=topsysfile   # 登錄賬號
export MINIO_SECRET_KEY=Hao123baidu!  # 登錄密碼

export MINIO_ETCD_ENDPOINTS="http://topsysfile.server.1.itusye.co:2380,http://topsysfile.server.2.itusye.co:2380,http://topsysfile.server.3.itusye.co:2380"   # 配置etcd 連接
export MINIO_DOMAIN=topfile.storage.minio.itusye.co  #此處需特別注意：此域名需要配置與nginx 配置反向代理時使用的域名一致，不然比如分享會直接訪問到xml, 開發連接也會有問題，
export MINIO_PUBLIC_IPS=172.31.78.241,172.31.78.242 # 此處需要配置上Minio-1 集群的所有ip ,可配置內網外網，因為我測試使用內網通迅，此處配置內網地址了


minio server --config-dir /usr/local/minio/config/ \   # 官方提示：--config-dir 選項要停止使用了，此選擇主要創建一個certs目錄存放ca 信息 
http://172.31.78.241/Data/minio/data1 \     # 以下信息是minio 分布式集群minio-1 啟動情況
http://172.31.78.241/Data/minio/data2 \
http://172.31.78.242/Data/minio/data1 \
http://172.31.78.242/Data/minio/data2
[root@Minio-Storage-Services ~]#

4 配置啟動腳本，所有服務器同樣的配置；

[root@Minio-Storage-Services ~]# more /usr/lib/systemd/system/minio.service 
[Unit]
Description=Minio service
Documentation=https://docs.minio.io/

[Service]
WorkingDirectory=/usr/local/minio/
ExecStart=/usr/local/minio/bin/run.sh

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
[root@Minio-Storage-Services ~]#

5.配置nginx , https://docs.min.io/docs/setup-nginx-proxy-with-minio.html 官方參考文檔

主要需要注意：客戶端上傳限制的問題

client_max_body_size 2048M;
client_body_buffer_size 500M;

以下為nginx Server 模塊配置

upstream topfilestorage-minio{
                ip_hash;
                server topsysfile.server.1.toptech.ph:9000 weight=3 max_fails=3 fail_timeout=10s;
                server topsysfile.server.2.toptech.ph:9000 weight=3 max_fails=3 fail_timeout=10s;
                server topsysfile.server.3.toptech.ph:9000 weight=3 max_fails=3 fail_timeout=10s;

                server topsysfile.server.4.toptech.ph:9000 weight=3 max_fails=3 fail_timeout=10s;
                #       check interval=3000 rise=2 fall=3 timeout=1000 default_down=true type=http;

}

server {
        listen 80;
        server_name localhost;
        if ($scheme = http ) {
                return 301 https://$host$request_uri;
                }
        }


server {
         server_name localhost;
                listen 443;
                ssl on;
                access_log  /home/logs/nginx/www_logs/web_access.log main;
                error_log  /home/logs/nginx/error.log;
                ssl_certificate /usr/local/nginx/ssl/topfile.crt;
                ssl_certificate_key     /usr/local/nginx/ssl/topfile.key;
                ssl_prefer_server_ciphers on;
                ssl_protocols   TLSv1 TLSv1.1 TLSv1.2;
                ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:ECDHE-RSA-AES128-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA128:DHE-RSA-A
ES128-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA128:ECDHE-RSA-AES128-SHA384:ECDHE-RSA-AES128-SHA128:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES
128-SHA128:DHE-RSA-AES128-SHA128:DHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA384:AES128-GCM-SHA128:AES128-SHA128:AES12
8-SHA128:AES128-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
                ssl_ecdh_curve secp384r1;
                error_page 403 404 /403.html;
                location = /403.html {
                        root  /var/www/;
                }
                ignore_invalid_headers off;
                proxy_buffering off;

                location / {
                        proxy_pass http://topfilestorage-minio;
                        include conf.d/proxy.conf;
                }

}

以下proxy.conf 文檔

[root@Minio-Storage-Services ~]# more /usr/local/nginx/config/conf.d/proxy.conf | grep -v '#'
proxy_http_version 1.1;


proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;

proxy_set_header Connection "";
chunked_transfer_encoding off;

[root@Minio-Storage-Services ~]#

以上集群已配置完成，測試新建文件夾會自動選擇集群存儲，並且測試如果固定用一個集群創建一個根目錄文件夾，再用另外一個集群或者反向代理去上傳都是可以把數據上傳存儲到文件夾所在的集群空間里，兩個集群內的數據並不會同步，但是在存儲時minio 會自動計算選擇集群存儲；

MinIO是一個開源的對象存儲解決方案，優點如下：

高性能：作為高性能對象存儲，在標准硬件條件下它能達到55GB/s的讀、35GG/s的寫速率
可擴容：不同MinIO集群可以組成聯邦，並形成一個全局的命名空間，並跨越多個數據中心
雲原生：容器化、基於K8S的編排、多租戶支持
兼容性：兼容S3 API這一事實上的對象存儲標准，最先支持S3 Select
簡單：這一設計原則讓MinIO不容易出錯、更快啟動
支持糾刪碼：MinIO使用糾刪碼、Checksum來防止硬件錯誤和靜默數據污染。在最高冗余度配置下，即使丟失1/2的磁盤也能恢復數據

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 文件服務器-docker-Minio分布式+nginx負載均衡+反向代理+nginx圖片剪切（一站式） Linux : 安裝MinIO集群+Nginx高可用/負載均衡+MC工具配置賬戶權限 minIO分布式集群搭建+nginx負載均衡 nginx做反向代理負載均衡 Java怎么獲取后端服務器獲取用戶IP 使用sidekick 負載均衡minio 集群使用Nginx實現服務器反向代理和負載均衡 Nginx服務器之反向代理與負載均衡 windows 2012配置DHCP服務器，進行高可用性負載均衡 nginx反向代理-后端服務器組設置 NGINX反向代理，后端服務器獲取真實IP