前面使用了哨兵集群搭建單個分片高可用的結構，實現了主從復制和故障轉移，但是沒有實現分布式。接下來使用Redis集群搭建，實現哨兵集群沒有實現的分布式高可用。
redis-cluster搭建需求
下面准備6個redis節點，搭建如圖所示的三主三從分布式redis集群，實現兩兩互聯，以master8001為例，它可以連接另外兩個主節點，以及三個從節點。搭建過程可以使用原生redis cluster命令（使用方便），也可以使用ruby自帶的腳本（需要先安裝ruby，還有安裝redis和ruby的接口，比較麻煩）。
ruby腳本輔助搭建
使用ruby腳本輔助安裝，即使用redis根目錄/src/redis-trib.rb腳本來完成，需要先安裝ruby。
-rwxrwxr-x. 1 root root 60852 Sep 21 2017 redis-trib.rb
（1）安裝ruby，這里解壓了壓縮包，進行編譯和安裝。但是安裝完只是最基本的一步，后面還有坑，比較麻煩。
# 解壓 [root@node01 /home/software]# tar -zxvf ruby-2.3.1.tar.gz # 檢查 [root@node01 /home/software/ruby-2.3.1]# . /configure # 編譯，安裝 [root@node01 /home/software/ruby-2.3.1]# make && make install # 省略具體安裝日志，安裝完成后使用ruby -v如果能查看到版本號，就安裝ok [root@node01 /home/software/ruby-2.3.1]# ruby -v ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_4-linux]
（2）先准備一個redis-cluster.conf啟動加載文件，基本配置參考前面博文https://www.cnblogs.com/youngchaolin/p/11983705.html#_label0 ，主要是bind設置不綁定ip、protected mode設置為no、daemonize設置為yes、其他還有pid、logfile、rdb等，保持原設置不變。
（3）配置appendonly為yes，啟動aof持久化方式，這是除了rdb外redis提供的第二種持久化方式。
589 # AOF and RDB persistence can be enabled at the same time without problems. 590 # If the AOF is enabled on startup Redis will load the AOF, that is the file 591 # with the better durability guarantees. 592 # 593 # Please check http://redis.io/topics/persistence for more information. 594 # 配置第二種持久化方式aof 595 appendonly yes 596 597 # The name of the append only file (default: "appendonly.aof") 598 # 標識aof持久化文件名，以端口號區分 599 appendfilename "appendonly6379.aof"
其中rdb和aof兩種持久化的主要區別為：
a.rdb保存的是具體的key-value數據，如name-messi，aof保存的是操作記錄，如set name messi。
b.因為aof保存的數據更全，redis啟動默認加載的是aof，rdb啟動可以單獨開啟。
c.rdb持久化頻率是按照持久化策略來的，容易造成數據丟失，而aof是每秒保存一次數據，數據不容易丟失。
d.如果對數據的可靠性要求高，使用aof持久化，如果需要一定的數據恢復能力，但是又不需要很高的可靠性，就選擇rdb。
（4）開啟集群模式，如果不開啟無法計算槽道號。
719 # Normal Redis instances can't be part of a Redis Cluster; only nodes that are 720 # started as cluster nodes can. In order to start a Redis instance as a 721 # cluster node enable the cluster support uncommenting the following: 722 # 需設置yes，否則無法計算槽道號，無法創建集群 723 cluster-enabled yes
（5）需要開啟集群節點狀態記錄文件，這個文件會自動更新，每個redis節點都需要一個這樣的文件。
725 # Every cluster node has a cluster configuration file. This file is not 726 # intended to be edited by hand. It is created and updated by Redis nodes. 727 # Every Redis Cluster node requires a different cluster configuration file. 728 # Make sure that instances running in the same system do not have 729 # overlapping cluster configuration file names. 730 # 開啟集群節點狀態記錄文件 731 cluster-config-file nodes-6379.conf
（6）將上面的配置文件保存，接下來復制到6個不同的文件夾下，其中文件夾的名字就是redis節點的端口號，根據上圖准備8000、8001、8002、8003、8004和8005六個文件夾。復制完成后，修改對應端口號，如8000文件夾的配置文件，將里面全是6379的部分替換為8000，其他類似修改即可。
# 創建目錄 [root@node01 /home/software/redis-3.2.11]# mkdir 8000 8001 8002 8003 8004 8005 # 復制文件 [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8000/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8001/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8002/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8003/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8004/ [root@node01 /home/software/redis-3.2.11]# cp redis.conf 8005/ # 遞歸查看是否copy成功 [root@node01 /home/software/redis-3.2.11]# ls -R 800* 8000: redis.conf 8001: redis.conf 8002: redis.conf 8003: redis.conf 8004: redis.conf 8005: redis.conf
# 修改端口略
（7）啟動各個節點，發現均是cluster的方式啟動，這是創建集群的基礎，另外查看集群狀態為fail，是因為槽道還沒分配的原因，且每個節點cluster nodes查看集群節點信息都只能找到自己一個節點，目前兩兩互聯還沒有建立。
# 加載修改后的配置文件啟動各個節點 [root@node01 /home/software/redis-3.2.11]# redis-server 8000/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8001/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8002/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8003/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8004/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8005/redis.conf # 查看是否以cluster方式啟動，ok [root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis root 16028 1 0 15:43 ? 00:00:00 redis-server *:8000 [cluster] root 16033 1 0 15:43 ? 00:00:00 redis-server *:8001 [cluster] root 16037 1 0 15:43 ? 00:00:00 redis-server *:8002 [cluster] root 16041 1 0 15:43 ? 00:00:00 redis-server *:8003 [cluster] root 16045 1 0 15:43 ? 00:00:00 redis-server *:8004 [cluster] root 16049 1 0 15:43 ? 00:00:00 redis-server *:8005 [cluster] root 16053 1409 0 15:43 pts/0 00:00:00 grep redis # 登錄一個節點，查看集群信息 [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 127.0.0.1:8000> cluster info cluster_state:fail # 只要有一個槽道沒分配，就是fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:1 # 只能發現自己一個節點，兩兩互聯暫未建立 cluster_size:0 cluster_current_epoch:0 # 集群的紀元 cluster_my_epoch:0 cluster_stats_messages_sent:0 # 兩兩互聯發送數據量 cluster_stats_messages_received:0 # 兩兩互聯接受數據量 # 查看集群節點，發現只有自己一個 127.0.0.1:8000> cluster nodes dc0e2e16888426089de31466ff398679eec81b86 :8000 myself,master - 0 0 0 connected
（8）啟動src/redis-trib.rb命令后想查看幫助信息發現報錯。這是正常的，這意味着踏入了一個大坑，接下來的需要填坑了，參考文末博文，完成了報錯解決。
# 報錯 /usr/local/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- redis (LoadError) from /usr/local/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require' from src/redis-trib.rb:25:in `<main>'
（9）上面報錯是需要redis的gem包來安裝ruby和redis的接口，先安裝gem包，網址https://rubygems.org/gems/redis/，選擇對應的版本下載，我這里是選擇3.2.1版本，下載好后上傳到了redis根目錄/src目錄下了，具體位置放哪不清楚，放src下是可以驗證通過的，另外查看博文放根目錄也是可以的。
-rw-r--r--. 1 root root 73728 Dec 12 16:51 redis-3.2.1.gem
（10）准備使用gem install redis命令安裝接口，發現報錯，提示需要安裝zlib，網址http://www.zlib.net，本次選擇了1.2.11版本。
ERROR: Loading command: install (LoadError) cannot load such file -- zlib ERROR: While executing gem ... (NoMethodError) undefined method `invoke_with_build_args' for nil:NilClass
（11）tar -zxvf解壓zlib壓縮包后，進入解壓目錄，先指定安裝目錄，后編譯安裝zlib。
# --prefix指定安裝目錄/usr/local/zlib [root@node01 /home/software/zlib-1.2.11]# ./configure --prefix=/usr/local/zlib Checking for gcc... Checking for shared library support... Building shared library libz.so.1.2.11 with gcc. Checking for size_t... Yes. Checking for off64_t... Yes. Checking for fseeko... Yes. Checking for strerror... Yes. Checking for unistd.h... Yes. Checking for stdarg.h... Yes. Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf(). Checking for vsnprintf() in stdio.h... Yes. Checking for return value of vsnprintf()... Yes. Checking for attribute(visibility) support... Yes. You have new mail in /var/spool/mail/root # 編譯后安裝 [root@node01 /home/software/zlib-1.2.11]# make && make install ...省略 安裝成功 chmod 644 /usr/local/zlib/include/zlib.h /usr/local/zlib/include/zconf.h
（12）還需要進入ruby根目錄/ext/zlib，先使用ruby extconf.rb命令，執行完后會在當前目錄下生成Makefile文件。
[root@node01 /home/software/ruby-2.3.1/ext/zlib]# ll total 892 -rw-r--r--. 1 root root 146 Apr 16 2013 depend -rw-r--r--. 1 root root 1447 Dec 16 2015 extconf.rb
# 使用ruby extconf.rb命令后會生成一個Makefile文件 -rw-r--r--. 1 root root 7656 Dec 12 17:14 Makefile -rw-r--r--. 1 root root 7090 Dec 12 17:14 mkmf.log -rw-r--r--. 1 root root 124168 Dec 19 2015 zlib.c -rw-r--r--. 1 root root 418576 Dec 12 17:18 zlib.o -rwxr-xr-x. 1 root root 336032 Dec 12 17:18 zlib.so
# 安裝時需添加配置信息，指定zlib目錄下的include和lib，zlib目錄就是前面編譯指定安裝的目錄 [root@node01 /home/software/ruby-2.3.1/ext/zlib]# ruby extconf.rb --with-zlib-include=/usr/local/zlib/include/ --with-zlib-lib=/usr/local/zlib/lib
# 省略，編譯安裝
[root@node01 /home/software/ruby-2.3.1/ext/zlib]# make && make install
接下來編譯，本次編譯順利通過了，如果沒有通過可能會出現如下報錯，還需要修改Makefile內容。
make: *** No rule to make target `/include/ruby.h', needed by `zlib.o'. Stop
修改Makefile內容，將$(top_srcdir)整個換成絕對路徑，我這里沒有報錯，因此沒有修改。
zlib.o: $(top_srcdir)/include/ruby.h
修改成：zlib.o: ../../include/ruby.h
最后編譯通過會提示如下內容。
[root@node01 /home/software/ruby-2.3.1/ext/zlib]# make && make install compiling zlib.c linking shared-object zlib.so /usr/bin/install -c -m 0755 zlib.so /usr/local/lib/ruby/site_ruby/2.4.0/x86_64-linux
（13）繼續安裝openssl，如果不安裝切換到redis目錄使用gem install redis會報''unable to require openssl...''的報錯，因此需要繼續像安裝zlib一樣安裝openssl。
網址https://www.openssl.org/source/，解壓后進入安裝目錄，執行以下命令，編譯安裝執行時間比較長。
[root@node01 /home/software/openssl-1.0.2t]# ./config -fPIC --prefix=/usr/local/openssl enable-shared [root@node01 /home/software/openssl-1.0.2t]# ./config -t # 這里會編譯很久 [root@node01 /home/software/openssl-1.0.2t]# make && make install
（14）還需進入ruby根目錄/ext/openssl下，執行ruby extconf.rb命令，也會在當前目錄下生成Makefile文件，執行后光榮的報錯了。
[root@node01 /home/software/ruby-2.3.1/ext/openssl]# ruby extconf.rb --with-openssl-include=/usr/local/openssl/include/ --with-openssl-lib=/usr/local/openssl/lib [root@node01 /home/software/ruby-2.3.1/ext/openssl]# make && make install
（15）類似上面安裝zlib，也將Makefile文件的\$(top_srcdir)全部改成"../.."，這里在使用了":%s/\${top_srcdir}/..\/../g"進行了全局替換。再次編譯安裝就通過了。
（16）到這里，就差不多可以了，切換目錄到redis根目錄，然后執行gem install redis，再次報錯。沒轍了，網上找了一種土辦法，先讓安裝成功。
[root@node01 /home/software/redis-3.2.11]# gem install redis ERROR: Could not find a valid gem 'redis' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)
臨時辦法，大概意思讓https連接改成http連接。
[root@node01 /home/software/redis-3.2.11]# gem sources -r https://rubygems.org source https://rubygems.org not present in cache [root@node01 /home/software/redis-3.2.11]# gem sources -a http://rubygems.org https://rubygems.org is recommended for security over http://rubygems.org Do you want to add this insecure source? [yn] y http://rubygems.org added to sources You have new mail in /var/spool/mail/root
繼續執行命令，終於ok了，可謂一波三折，這個方法只能作為參考，換一個redis或ruby又不知道會發生什么報錯。
# 繼續 [root@node01 /home/software/redis-3.2.11/src]# gem install redis Successfully installed redis-3.2.1 Parsing documentation for redis-3.2.1 Installing ri documentation for redis-3.2.1 Done installing documentation for redis after 0 seconds #成功了 WARNING: Unable to pull data from 'https://rubygems.org/': SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz) 1 gem installed You have new mail in /var/spool/mail/root # 試運行redis-trib.rb腳本，終於ok了 [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb Usage: redis-trib <command> <options> <arguments ...> create host1:port1 ... hostN:portN --replicas <arg> check host:port info host:port fix host:port --timeout <arg> reshard host:port --from <arg> --to <arg> --slots <arg> --yes --timeout <arg> --pipeline <arg> rebalance host:port --weight <arg> --auto-weights --use-empty-masters --timeout <arg> ...省略
（17）前面准備好了后，就可以使用ruby腳本來開始構建集群了。上面啟動后的redis節點都是各自為政互不聯系，接下來先將8000、8001、8002作為主節點連起來，並分配槽道號，本次使用默認分配。
# 使用create [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb create 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 >>> Creating cluster >>> Performing hash slots allocation on 3 nodes... Using 3 masters: 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 slots:0-5460 (5461 slots) master M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 slots:5461-10922 (5462 slots) master M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 slots:10923-16383 (5461 slots) master Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join >>> Performing Cluster Check (using node 192.168.200.140:8000) M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 slots:0-5460 (5461 slots) master 0 additional replica(s) M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 0 additional replica(s) M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 0 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. You have new mail in /var/spool/mail/root [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 # 查看發現集群狀態已變成ok，說明槽道沒有一個沒分配 127.0.0.1:8000> cluster into (error) ERR Wrong CLUSTER subcommand or number of arguments 127.0.0.1:8000> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:3 cluster_size:3 cluster_current_epoch:3 cluster_my_epoch:1 cluster_stats_messages_sent:68 cluster_stats_messages_received:68 # 三個主節點都分配了槽道 127.0.0.1:8000> cluster nodes
# 節點信息
# 第一個為節點id，用40位的十六進制表示
# 第二個為節點ip+端口
# 第三個為角色
# 第四個為主節點id，沒有主節點就用'-'表示
# 第五個是與操作相關的時間戳
# 第六個代表序號
# 第七個代表連接狀態
# 第八個代表槽道號區間 aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 master - 0 1576145625541 2 connected 5461-10922 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 master - 0 1576145623522 3 connected 10923-16383 dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460
（18）將8003掛接到8000，8004掛接到8001，8005掛接到8002成為從節點，使用add node命令，這里需要用到兩個選項，一個是--salve，代表新加的節點是從，一個是--master-id，指定需要添加的主是誰，最后依次添加slaveHost:slavePort existHost:existPort，后面的節點可以是集群中任意一個集群的已知節點。
# 以添加8003為例 ，注意主節點可以是任意一個主節點，不一定是當前從的主節點
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb add-node --slave --master-id dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8003 192.168.200.140:8000 >>> Adding node 192.168.200.140:8003 to cluster 192.168.200.140:8000 >>> Performing Cluster Check (using node 192.168.200.140:8000) M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 slots:0-5460 (5461 slots) master 0 additional replica(s) M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 0 additional replica(s) M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 0 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
# 底層還是調用redis原生命令cluster meet >>> Send CLUSTER MEET to node 192.168.200.140:8003 to make it join the cluster. Waiting for the cluster to join. >>> Configure node as replica of 192.168.200.140:8000. [OK] New node added correctly. You have new mail in /var/spool/mail/root
登錄集群客戶端查看集群信息，發現搭建成功，實現了三主三從redis集群的搭建。
127.0.0.1:8000> cluster nodes aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 master - 0 1576152242486 2 connected 5461-10922 dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460 824a999176c12eb6ebd944b75944fd97c65b579d 192.168.200.140:8005 slave 719d1dd412faf6b4e1eb348c65c329a262e393e1 0 1576152244002 3 connected 312accb9be27c48c107a19ab501a75a8c5d321c6 192.168.200.140:8004 slave aa79ace502e5369236b62ed61c0eb43733ddcbde 0 1576152239461 2 connected ada1903bcb5744900916151dd385936ceb6ce049 192.168.200.140:8003 slave dc0e2e16888426089de31466ff398679eec81b86 0 1576152243496 1 connected 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 master - 0 1576152244507 3 connected 10923-16383
ruby腳本簡化搭建
可以看出，這個搭建比較繁瑣， 如果解決了ruby的報錯問題，只需要一個命令就可以完成三主三從redis集群的搭建，下面記錄一下。
（1）停止redis所有進程。
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8001 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8002 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8004 shutdown [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8005 shutdown [root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis root 3657 18054 0 20:13 pts/2 00:00:00 grep redis
（2）刪除rdb和aof持久化文件，刪除集群節點狀態記錄文件。
[root@node01 /home/software/redis-3.2.11]# rm -rf appendonly800* [root@node01 /home/software/redis-3.2.11]# rm -rf dump800* [root@node01 /home/software/redis-3.2.11]# rm -rf nodes-800*
（3）再次啟動6個redis節點。
# 啟動 [root@node01 /home/software/redis-3.2.11]# redis-server 8000/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8001/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8002/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8003/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8004/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-server 8005/redis.conf [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 # 還沒分配槽道，redis集群狀態fail 127.0.0.1:8000> cluster info cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:1 cluster_size:0 cluster_current_epoch:0 cluster_my_epoch:0 cluster_stats_messages_sent:0 cluster_stats_messages_received:0
（4）使用ruby腳本src/redis-trib.rb create --replicas 1 六個的節點ip:端口，這一條命令就可以自動搭建了三主三從。
# 一個命令 [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb create --replicas 1 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 192.168.200.140:8003 192.168.200.140:8004 192.168.200.140:8005 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 Adding replica 192.168.200.140:8003 to 192.168.200.140:8000 Adding replica 192.168.200.140:8004 to 192.168.200.140:8001 Adding replica 192.168.200.140:8005 to 192.168.200.140:8002 M: 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000 slots:0-5460 (5461 slots) master M: 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001 slots:5461-10922 (5462 slots) master M: 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002 slots:10923-16383 (5461 slots) master S: a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003 replicates 6537ac1cb42f209600ea20b1d6310e7b789257d4 S: ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004 replicates 5da79737cf493085724a689bdb5bd878c9121c07 S: bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005 replicates 49f40898d2d18cc162b00e5f69593f0b91af5766 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 192.168.200.140:8000) M: 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005 slots: (0 slots) slave replicates 49f40898d2d18cc162b00e5f69593f0b91af5766 M: 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003 slots: (0 slots) slave replicates 6537ac1cb42f209600ea20b1d6310e7b789257d4 S: ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004 slots: (0 slots) slave replicates 5da79737cf493085724a689bdb5bd878c9121c07 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. You have new mail in /var/spool/mail/root [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 # 結果一樣 127.0.0.1:8000> cluster nodes 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002 master - 0 1576153283083 3 connected 10923-16383 bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005 slave 49f40898d2d18cc162b00e5f69593f0b91af5766 0 1576153285101 6 connected 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001 master - 0 1576153286109 2 connected 5461-10922 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460 a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003 slave 6537ac1cb42f209600ea20b1d6310e7b789257d4 0 1576153283585 4 connected ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004 slave 5da79737cf493085724a689bdb5bd878c9121c07 0 1576153284091 5 connected
redis cluster原生命令搭建
以上兩種方法，都是依賴ruby腳本，其實它還是基於redis cluster的命令來搭建完成，使用這種方式搭建能更好的理解搭建中經歷了什么。
（1）提前准備6個啟動的節點，前提也需刪除持久化文件，以及節點狀態記錄文件，跟上面一樣。
（2）登錄任何一個節點，和其他5個進行meet，這里使用8000的客戶端來完成，meet完后任意一個節點都認為自己是主節點。
# 使用cluster meet命令
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8001 OK 192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8002 OK 192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8003 OK 192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8004 OK 192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8005 OK
# 登錄任何一個節點查看，都能得到如下信息 192.168.200.140:8000> cluster nodes fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576155518818 0 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 myself,master - 0 0 4 connected 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576155517810 5 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576155514284 1 connected 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576155515791 2 connected 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576155516800 3 connected
（3）給8000，8001和8002分配槽道號，讓其作為主節點。分配槽道號的命令為cluster addslots 槽道號...。分配完成后不管登錄哪個節點都能看到槽道號信息，
# 槽道號之間有多個需要用空格隔開 [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 -h 192.168.200.140 192.168.200.140:8000> CLUSTER ADDSLOTS 0 1 2 3 4 5 OK
# 可以看到8000的節點分配了0-5這6個槽道號 192.168.200.140:8000> cluster nodes fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576155898158 0 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 myself,master - 0 0 4 connected 0-5 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576155896137 5 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576155894119 1 connected 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576155897147 2 connected 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576155895128 3 connected
一個個寫來分配會比較費力氣，可以寫一個shell腳本來批量分配槽道號，使用for循環來實現。
shell腳本
#分配槽道號的腳本 #!/bin/bash # 8000節點 for slot in {6..5460} do redis-cli -c -p 8000 -h 192.168.200.140 cluster addslots $slot done # 8001節點 for slot in {5461..10922} do redis-cli -c -p 8001 -h 192.168.200.140 cluster addslots $slot done # 8002節點 for slot in {10923..16383} do redis-cli -c -p 8002 -h 192.168.200.140 cluster addslots $slot done # 執行完打印出來結果 echo "分配槽道號結束"
運行腳本執行完結果，ok。
登錄一個節點查看集群槽道信息，發現跟分配的預想一樣。
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8001 -h 192.168.200.140 192.168.200.140:8001> cluster nodes
# 8001結果，與上面腳本一樣結果 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 myself,master - 0 0 1 connected 5461-10922 fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576157229386 0 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 0 1576157228376 4 connected 0-5460 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576157229386 2 connected 10923-16383 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576157227368 3 connected 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576157230393 5 connected
（4）將8003、8004和8005角色轉換為從節點，使用cluster replicate 主節點id 命令。最后查看集群信息，發現成功的分配了槽道號。
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140 # 成為8000的從節點 192.168.200.140:8003> cluster replicate ced952bc6a47c38756bbc48d85a5f601db620b79 OK 192.168.200.140:8003> quit [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8004 -h 192.168.200.140 # 成為8001的從節點 192.168.200.140:8004> cluster replicate 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 OK 192.168.200.140:8004> quit You have new mail in /var/spool/mail/root [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8005 -h 192.168.200.140 # 成為8002的從節點 192.168.200.140:8005> cluster replicate 7ce388bde879f686fc3c8491175397ca20405565 OK # 結果ok 192.168.200.140:8005> cluster nodes 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 myself,slave 7ce388bde879f686fc3c8491175397ca20405565 0 0 5 connected 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576157733223 2 connected 10923-16383 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 slave ced952bc6a47c38756bbc48d85a5f601db620b79 0 1576157732216 4 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 0 1576157735238 4 connected 0-5460 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576157736242 1 connected 5461-10922 fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576157734230 1 connected
最后集群狀態為ok，槽道全部分配完成，形成了三主三從的結構。
192.168.200.140:8005> cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0
# 6個節點 cluster_known_nodes:6 # 3個主
cluster_size:3 cluster_current_epoch:5 cluster_my_epoch:2 cluster_stats_messages_sent:5093 cluster_stats_messages_received:5093
redis cluster高可用
現在在8000節點上保存數據，然后將8000的主節點宕機，8003從節點會頂替上來成為主節點，並獲取到8000的槽道信息。
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 -h 192.168.200.140 192.168.200.140:8000> set age 28 OK 192.168.200.140:8000> get age "28" 192.168.200.140:8000> quit [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140 192.168.200.140:8003> get age -> Redirected to slot [741] located at 192.168.200.140:8000 "28" 192.168.200.140:8000> shutdown not connected> quit You have new mail in /var/spool/mail/root [root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140 # 8000宕機后，8003暫時還是從 192.168.200.140:8003> cluster nodes fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576158134892 1 connected 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 myself,slave ced952bc6a47c38756bbc48d85a5f601db620b79 0 0 3 connected ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 1576158120865 1576158119755 4 disconnected 0-5460 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576158132873 5 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576158133883 1 connected 5461-10922 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576158131864 2 connected 10923-16383 # 過一會，8003變為主，並且獲得槽道信息 192.168.200.140:8003> cluster nodes fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576158149019 1 connected 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 myself,master - 0 0 6 connected 0-5460 ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master,fail - 1576158120865 1576158119755 4 disconnected 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576158148008 5 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576158146999 1 connected 5461-10922 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576158150028 2 connected 10923-16383 # 數據也備份了 192.168.200.140:8003> get age "28"
redis cluster添加和刪除節點
再補充添加和刪除節點的操作，關於刪除節點，它只能刪除從節點或沒有槽道管理權的節點，由於8000已經沒有了槽道管理權，將8000重啟后，下面使用ruby腳本刪除，命令為del-node ip:port 節點id。
# 確認8000開啟
[root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis root 4170 1 0 20:51 ? 00:00:13 redis-server *:8001 [cluster] root 4174 1 0 20:51 ? 00:00:11 redis-server *:8002 [cluster] root 4178 1 0 20:51 ? 00:00:08 redis-server *:8003 [cluster] root 4182 1 0 20:51 ? 00:00:08 redis-server *:8004 [cluster] root 4186 1 0 20:51 ? 00:00:08 redis-server *:8005 [cluster] root 7231 1 0 22:09 ? 00:00:00 redis-server *:8000 [cluster] root 7298 18054 0 22:13 pts/2 00:00:00 grep redis
# 刪除8000 [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb del-node 192.168.200.140:8000 94f634b2d364d560d3f8fb6e88fa874cf2b493f6 >>> Removing node 94f634b2d364d560d3f8fb6e88fa874cf2b493f6 from cluster 192.168.200.140:8000 >>> Sending CLUSTER FORGET messages to the cluster...
# 關閉節點 >>> SHUTDOWN the node. You have new mail in /var/spool/mail/root
發現刪除節點后，還自動將節點關閉了，因為8000節點的狀態記錄文件nodes-8000.conf還記錄着8000是連接狀態，但是其他節點的這個文件已經沒了8000的信息，為了集群數據的不混亂，需要將8000立即關閉。
既然刪除了，嘗試啟動8000后添加到集群，需要使用add-node newHost:newPort existHost:existPort命令，類似添加從節點命令只是沒有選項。發現啟動8000后再添加進集群會報錯。
# 確認開啟8000 [root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis root 4170 1 0 20:51 ? 00:00:14 redis-server *:8001 [cluster] root 4174 1 0 20:51 ? 00:00:11 redis-server *:8002 [cluster] root 4178 1 0 20:51 ? 00:00:08 redis-server *:8003 [cluster] root 4182 1 0 20:51 ? 00:00:08 redis-server *:8004 [cluster] root 4186 1 0 20:51 ? 00:00:08 redis-server *:8005 [cluster] root 7400 1 0 22:21 ? 00:00:00 redis-server *:8000 [cluster] root 7422 18054 0 22:22 pts/2 00:00:00 grep redis You have new mail in /var/spool/mail/root # 添加8000到集群 [root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb add-node 192.168.200.140:8000 192.168.200.140:8001 >>> Adding node 192.168.200.140:8000 to cluster 192.168.200.140:8001 >>> Performing Cluster Check (using node 192.168.200.140:8001) M: 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slots: (0 slots) slave replicates 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 M: 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 slots:0-5460 (5461 slots) master 0 additional replica(s) S: 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slots: (0 slots) slave replicates 7ce388bde879f686fc3c8491175397ca20405565 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. # 提示報錯，提示8000可能有數據，或者8000里有其他集群的信息 [ERR] Node 192.168.200.140:8000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
這里刪除了8000的rdb和aof持久化文件，以及狀態文件后，重啟8000節點再次嘗試添加進去集群發現成功，但是添加進去的節點為沒有槽道管理權的主節點。
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb add-node 192.168.200.140:8000 192.168.200.140:8001 >>> Adding node 192.168.200.140:8000 to cluster 192.168.200.140:8001 >>> Performing Cluster Check (using node 192.168.200.140:8001) M: 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slots: (0 slots) slave replicates 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 M: 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 slots:0-5460 (5461 slots) master 0 additional replica(s) S: 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slots: (0 slots) slave replicates 7ce388bde879f686fc3c8491175397ca20405565 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 192.168.200.140:8000 to make it join the cluster. [OK] New node added correctly.
8000的信息
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576161337438 1 connected # 沒有槽道管理權的主節點 c41dbe9595ae83725d1322b032736fd198b26c49 192.168.200.140:8000 myself,master - 0 0 0 connected 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576161336432 2 connected 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576161336432 1 connected 5461-10922 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576161339456 6 connected 0-5460 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576161338448 2 connected 10923-16383
以上是redis-cluster集群搭建的基本知識，后面繼續補充槽道知識。
參考博文：
（1）《Redis設計與實踐》
（2）https://blog.csdn.net/qq_26710443/article/details/82724268 ruby安裝
（3）https://www.jianshu.com/p/c38369097448
（4）https://www.cnblogs.com/xuliangxing/p/7146868.html ruby腳本不能執行解決，報錯主要參考文
（5）https://blog.csdn.net/ck3207/article/details/90404952 openssl無法編譯