版本：redis-3.0.5 redis-3.2.0 redis-3.2.9 redis-4.0.11
參考：http://redis.io/topics/cluster-tutorial。
集群部署交互式命令行工具：https://github.com/eyjian/redis-tools/tree/master/deploy
集群運維命令行工具：https://github.com/eyjian/redis-tools/tree/master
批量操作工具：https://github.com/eyjian/libmooon/releases
目錄
5.3. OOM相關：vm.overcommit_memory 5
5.4. /sys/kernel/mm/transparent_hugepage/enabled 5
11.3. jedis（java cluster client） 13
11.4. r3c（C++ cluster client） 14
17.2. redis-trib.rb rebalance 18
1. 前言
本文參考官方文檔而成：http://redis.io/topics/cluster-tutorial。經測試，安裝過程也適用於redis-3.2.0、redis-4.0.11等。
Redis運維工具和部署工具：https://github.com/eyjian/redis-tools。
2. 部署計划
依據官網介紹，部署6個redis節點，為3主3從。3台物理機每台都創建2個redis節點：
| 服務端口 |
IP地址 |
配置文件名 |
| 6379 |
192.168.0.251 |
redis-6379.conf |
| 6379 |
192.168.0.252 |
redis-6379.conf |
| 6379 |
192.168.0.253 |
redis-6379.conf |
| 6380 |
192.168.0.251 |
redis-6380.conf |
| 6380 |
192.168.0.252 |
redis-6380.conf |
| 6380 |
192.168.0.253 |
redis-6380.conf |
疑問：3台物理機，會不會主和從節點分布在同一個物理機上？
3. 目錄結構
redis.conf為從https://raw.githubusercontent.com/antirez/redis/3.0/redis.conf下載的配置文件。redis-6379.conf和redis-6380.conf指定了服務端口，兩者均通過include復用（包含）了redis.conf。
本文將redis安裝在/data/redis（每台機器完全相同，同一台機器上的多個節點對應相同的目錄和文件，並建議將bin目錄加入到環境變量PATH中，以簡化后續的使用）：
| /data/redis |-- bin | |-- redis-benchmark | |-- redis-check-aof | |-- redis-check-dump | |-- redis-cli | |-- redis-sentinel -> redis-server | `-- redis-server |-- conf | |-- redis-6379.conf | |-- redis-6380.conf | `-- redis.conf `-- log
3 directories, 9 files |
4. 編譯安裝
打開redis的Makefile文件，可以看到如下內容：
| PREFIX?=/usr/local INSTALL_BIN=$(PREFIX)/bin INSTALL=install |
Makefile中的“?=”表示，如果該變量之前沒有定義過，則賦值為/usr/local，否則什么也不做。
如果不設置環境變量PREFIX或不修改Makefile中的值，則默認安裝到/usr/local/bin目錄下。建議不要使用默認配置，而是指定安裝目錄，如/data/redis-3.0.5：
| $ make $ make install PREFIX=/data/redis-3.0.5 $ ln -s /data/redis-3.0.5 /data/redis $ mkdir /data/redis/conf $ mkdir /data/redis/log $ mkdir /data/redis/data |
5. 修改系統參數
5.1. 修改最大可打開文件數
修改文件/etc/security/limits.conf，加入以下兩行：
| * soft nofile 102400 * hard nofile 102400
# End of file |
其中102400為一個進程最大可以打開的文件個數，當與RedisServer的連接數多時，需要設定為合適的值。
有些環境修改后，root用戶需要重啟機器才生效，而普通用戶重新登錄后即生效。如果是crontab，則需要重啟crontab，如：service crond restart，有些平台可能是service cron restart。
有些環境下列設置即可讓root重新登錄即生效，而不用重啟機器：
| root soft nofile 102400 root hard nofile 102400
# End of file |
但是要小心，有些環境上面這樣做，可能導致無法ssh登錄，所以在修改時最好打開兩個窗口，萬一登錄不了還可自救。
如何確認更改對一個進程生效？按下列方法（其中$PID為被查的進程ID）：
| $ cat /proc/$PID/limits |
系統關於/etc/security/limits.conf文件的說明：
| #This file sets the resource limits for the users logged in via PAM. #It does not affect resource limits of the system services. |
PAM：全稱“Pluggable Authentication Modules”，中文名“插入式認證模塊”。/etc/security/limits.conf實際為pam_limits.so（位置：/lib/security/pam_limits.so）的配置文件，只針對單個會話。要使用limits.conf生效，必須保證pam_limits.so被加入到了啟動文件中。
注釋說明只對通過PAM登錄的用戶生效，與PAM相關的文件（均位於/etc/pam.d目錄下）：
| /etc/pam.d/login /etc/pam.d/sshd /etc/pam.d/crond |
如果需要設置Linux用戶的密碼策略，可以修改文件/etc/login.defs，但這個只對新增的用戶有效，如果要影響已有用戶，可使用命令chage。
5.2. TCP監聽隊列大小
即TCP listen的backlog大小，“/proc/sys/net/core/somaxconn”的默認值一般較小如128，需要修改大一點，比如改成32767。立即生效還可以使用命令：sysctl -w net.core.somaxconn=32767。
要想永久生效，需要在文件/etc/sysctl.conf中增加一行：net.core.somaxconn = 32767，然后執行命令“sysctl -p”以生效。
Redis配置項tcp-backlog的值不能超過somaxconn的大小。
5.3. OOM相關：vm.overcommit_memory
“/proc/sys/vm/overcommit_memory”默認值為0，表示不允許申請超過CommitLimmit大小的內存。可以設置為1關閉Overcommit，設置方法請參照net.core.somaxconn完成。
5.4. /sys/kernel/mm/transparent_hugepage/enabled
默認值為“[always] madvise never”，建議設置為never，以開啟內核的“Transparent Huge Pages (THP)”特性，設置后redis進程需要重啟。為了永久生效，請將“echo never > /sys/kernel/mm/transparent_hugepage/enabled”加入到文件/etc/rc.local中。
什么是Transparent Huge Pages？為提升性能，通過大內存頁來替代傳統的4K頁，使用得管理虛擬地址數變少，加快從虛擬地址到物理地址的映射，以及摒棄內存頁面的換入換出以提高內存的整體性能。內核Kernel將程序緩存內存中，每頁內存以2M為單位。相應的系統進程為khugepaged。
在Linux中，有兩種方式使用Huge Pages，一種是2.6內核引入的HugeTLBFS，另一種是2.6.36內核引入的THP。HugeTLBFS主要用於數據庫，THP廣泛應用於應用程序。
一般可以在rc.local或/etc/default/grub中對Huge Pages進行設置。
6. 配置redis
從https://raw.githubusercontent.com/antirez/redis/3.0/redis.conf下載配置文件（也可直接復制源代碼包中的redis.conf，然后在它的基礎上進行修改），在這個基礎上，進行如下表所示的修改（配置文件名redis-6379.conf中的6379建議設置為實際使用的端口號）：
| 配置項 |
值 |
配置文件 |
說明 |
| include |
redis.conf |
redis-6379.conf |
引用公共的配置文件，建議為全路徑值 |
| port |
6379 |
客戶端連接端口，並且總有一個剛好大於10000的端口，這個大的端口用於主從復制和集群內部通訊。 |
|
| cluster-config-file |
nodes-6379.conf |
默認放在dir指定的目錄 |
|
| pidfile |
/var/run/redis-6379.pid |
只有當daemonize值為yes時，才有意義；並且這個要求對目錄/var/run有寫權限，否則可以考慮設置為/tmp/redis-6379.pid。 |
|
| dir |
/data/redis/data/6379 |
|
|
| dbfilename |
dump-6379.rdb |
位於dir指定的目錄下 |
|
| appendonly |
yes |
|
|
| appendfilename |
"appendonly-6379.aof" |
|
|
| logfile |
/data/redis/log/redis-6379.log |
日志文件，包含目錄和文件名，注意redis不會自動滾動日志文件 |
|
| include |
redis.conf |
redis-6380.conf |
引用公共的配置文件 |
| port |
6380 |
|
|
| cluster-config-file |
nodes-6380.conf |
默認放在dir指定的目錄 |
|
| pidfile |
/var/run/redis-6380.pid |
|
|
| dir |
/data/redis/data/6380 |
AOF和RDB文件存放目錄 |
|
| dbfilename |
dump-6380.rdb |
RDB文件名 |
|
| appendfilename |
appendonly-6380.aof |
AOF文件名 |
|
| logfile |
/data/redis/log/redis-6380.log |
|
|
| loglevel |
verbose |
redis.conf （公共配置文件） |
日志級別，建議為notice，另外注意redis是不會滾動日志文件的，每次寫日志都是先打開日志文件再寫日志再關閉方式 |
| maxclients |
10000 |
最大連接數 |
|
| timeout |
0 |
客戶端多長（秒）時間沒發包過來關閉它，0表示永不關閉 |
|
| cluster-enabled |
yes |
表示以集群方式運行，為no表示以非集群方式運行 |
|
| cluster-node-timeout |
15000 |
單位為毫秒： repl-ping-slave-period+ (cluster-node-timeout* cluster-slave-validity-factor) 判斷節點失效（fail）之前，允許不可用的最大時長（毫秒），如果master不可用時長超過此值，則會被failover。不能太小，建議默認值15000 |
|
| cluster-slave-validity-factor |
0 |
如果要最大的可用性，值設置為0。定義slave和master失聯時長的倍數，如果值為0，則只要失聯slave總是嘗試failover，而不管與master失聯多久。失聯最大時長：(cluster-slave-validity-factor*cluster-node-timeout) |
|
| repl-timeout |
10 |
該配置項的值要求大於repl-ping-slave-period的值 |
|
| repl-ping-slave-period |
1 |
定義slave多久（秒）ping一次master，如果超過repl-timeout指定的時長都沒有收到響應，則認為master掛了 |
|
| slave-read-only |
yes |
slave是否只讀 |
|
| slave-serve-stale-data |
yes |
當slave與master斷開連接，slave是否繼續提供服務 |
|
| slave-priority |
100 |
slave權重值，當master掛掉，只有權重最大的slave接替master |
|
| aof-use-rdb-preamble |
|
4.0新增配置項，用於控制是否啟用RDB-AOF混用，值為no表示關閉 |
|
| appendonly |
yes |
當同時寫AOF或RDB，則redis啟動時只會加載AOF，AOF包含了全量數據。如果當隊列使用，入隊壓力又很大，建議設置為no |
|
| appendfsync |
no |
可取值everysec，其中no表示由系統自動，當寫壓力很大時，建議設置為no，否則容易造成整個集群不可用 |
|
| daemonize |
yes |
相關配置項pidfile |
|
| protected-mode |
no |
3.2.0新增的配置項，默認值為yes，限制從其它機器登錄Redis server，而只能從127.0.0.1登錄。為保證redis-trib.rb工具的正常運行，需要設置為no，完成后可以改回yes，但每次使用redis-trib.rb都需要改回為no。要想從非127.0.0.1訪問也需要改為no。 |
|
| tcp-backlog |
32767 |
取值不能超過系統的/proc/sys/net/core/somaxconn |
|
| auto-aof-rewrite-percentage |
100 |
設置自動rewite AOF文件（手工rewrite只需要調用命令BGREWRITEAOF） |
|
| auto-aof-rewrite-min-size |
64mb |
觸發rewrite的AOF文件大小，只有大於此大小時才會觸發rewrite |
|
| no-appendfsync-on-rewrite |
yes |
子進程在做rewrite時，主進程不調用fsync（由內核默認調度） |
|
| stop-writes-on-bgsave-error |
yes |
如果因為磁盤故障等導致保存rdb失敗，停止寫操作，可設置為NO。 |
|
| cluster-require-full-coverage |
no |
為no表示有slots不可服務時其它slots仍然繼續服務 |
|
| maxmemory |
26843545600 |
設置最大的內存，單位為字節 |
|
| maxmemory-policy |
volatile-lru |
設置達到最大內存時的淘汰策略 |
|
| client-output-buffer-limit |
|
設置master端的客戶端緩存，三種：normal、slave和pubsub |
|
| cluster-migration-barrier |
1 |
最少slave數，用來保證集群中不會有裸奔的master。當某個master節點的slave節點掛掉裸奔后，會從其他富余的master節點分配一個slave節點過來，確保每個master節點都有至少一個slave節點，不至於因為master節點掛掉而沒有相應slave節點替換為master節點導致集群崩潰不可用。 |
|
| repl-backlog-size |
1mb |
當slave失聯時的，環形復制緩區大小，值越大可容忍更長的slave失聯時長 |
|
| repl-backlog-ttl |
|
slave失聯的時長達到該值時，釋放backlog緩沖區 |
|
| save |
save 900 1 save 300 10 save 60 10000 |
刷新快照（RDB）到磁盤的策略，根據實際調整值，“save 900 1”表示900秒后至少有1個key被修改才觸發save操作，其它類推。 注意執行flushall命令也會產生RDB文件，不過是空文件。 如果不想生成RDB文件，可以將save全注釋掉。 |
7. 啟動redis實例
登錄3台物理機，啟動兩個redis實例（啟動之前，需要創建好配置中的各目錄）：
1) redis-server redis-6379.conf
2) redis-server redis-6380.conf
可以寫一個啟動腳本start-redis-cluster.sh：
| #!/bin/sh
REDIS_HOME=/data/redis $REDIS_HOME/bin/redis-server $REDIS_HOME/conf/redis-6379.conf $REDIS_HOME/bin/redis-server $REDIS_HOME/conf/redis-6380.conf |
8. 創建和啟動redis cluster前的准備工作
上一步啟動的redis只是單機版本，在啟動redis cluster之前，需要完成如下一些依賴的安裝。在此之后，才可以創建和啟動redis cluster。
8.1. 安裝ruby
安裝命令：yum install ruby
安裝過程中，如提示“[y/d/N]”，請選“y”然后回車。
查看版本：
| $ ruby --version ruby 2.0.0p353 (2013-11-22) [x86_64-linux] |
也可以從Ruby官網https://www.ruby-lang.org下載安裝包（如ruby-2.3.1.tar.gz）來安裝Ruby。截至2016/5/13，Ruby的最新穩定版本為Ruby 2.3.1。
8.2. 安裝rubygems
安裝命令：yum install rubygems
如果不使用yum安裝，也可以手動安裝RubyGems，RubyGems是一個Ruby包管理框架，它的下載網址：https://rubygems.org/pages/download。
比如下載安裝包rubygems-2.6.4.zip后解壓，然后進入解壓生成的目錄，里面有個setup.rb文件，以root用戶執行：ruby setup.rb安裝RubyGems。
8.3. 安裝redis-3.0.0.gem
安裝命令：gem install -l redis-3.0.0.gem
安裝之前，需要先下載好redis-3.0.0.gem。
redis-3.0.0.gem官網：https://rubygems.org/gems/redis/versions/3.0.0
redis-3.0.0.gem下載網址：https://rubygems.org/downloads/redis-3.0.0.gem
redis-3.3.0.gem官網：https://rubygems.org/gems/redis/versions/3.3.0
redis-3.3.3.gem官網：https://rubygems.org/gems/redis/versions/3.3.3
redis-4.0.1.gem官網：https://rubygems.org/gems/redis/versions/4.0.1
集群的創建只需要在一個節點上操作，所以只需要在一個節點上安裝redis-X.X.X.gem即可。
| # gem install -l redis-3.3.3.gem Successfully installed redis-3.3.3 Parsing documentation for redis-3.3.3 Installing ri documentation for redis-3.3.3 Done installing documentation for redis after 1 seconds 1 gem installed |
9. redis-trib.rb
redis-trib.rb是redis官方提供的redis cluster管理工具，使用ruby實現。
10. 創建和啟動redis集群
10.1. 復制redis-trib.rb
將redis源代碼的src目錄下的集群管理程序redis-trib.rb復制到/data/redis/bin目錄，並將bin目錄加入到環境變量PATH中，以簡化后續的操作。
redis-trib.rb用法（不帶任何參數執行redis-trib.rb即顯示用法）：
| $ ./redis-trib.rb Usage: redis-trib <command> <options> <arguments ...>
rebalance host:port --auto-weights --timeout <arg> --pipeline <arg> --use-empty-masters --weight <arg> --threshold <arg> --simulate add-node new_host:new_port existing_host:existing_port --slave --master-id <arg> reshard host:port --timeout <arg> --pipeline <arg> --yes --slots <arg> --to <arg> --from <arg> check host:port set-timeout host:port milliseconds call host:port command arg arg .. arg fix host:port --timeout <arg> info host:port create host1:port1 ... hostN:portN --replicas <arg> import host:port --replace --copy --from <arg> help (show this help) del-node host:port node_id
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster. |
10.2. 創建redis cluster
創建命令（3主3從）：
| redis-trib.rb create --replicas 1 192.168.0.251:6379 192.168.0.252:6379 192.168.0.253:6379 192.168.0.251:6380 192.168.0.252:6380 192.168.0.253:6380 |
Ø 參數說明：
1) create
表示創建一個redis cluster集群。
2) --replicas 1
表示為集群中的每一個主節點指定一個從節點，即一比一的復制。\
運行過程中，會有個提示，輸入yes回車即可。從屏幕輸出，可以很容易地看出哪些是主（master）節點，哪些是從（slave）節點：
| >>> Creating cluster Connecting to node 192.168.0.251:6379: OK /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182: warning: wrong element type nil at 0 (expected array) /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182: warning: ignoring wrong elements is deprecated, remove them explicitly /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182: warning: this causes ArgumentError in the next release >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 192.168.0.251:6379 192.168.0.252:6379 192.168.0.253:6379 Adding replica 192.168.0.252:6380 to 192.168.0.251:6379 Adding replica 192.168.0.251:6380 to 192.168.0.252:6379 Adding replica 192.168.0.253:6380 to 192.168.0.253:6379 M: 150f77d1000003811fb3c38c3768526a0b25ec31 192.168.0.251:6379 slots:0-5460 (5461 slots) master M: de461d3337b17d2119b79024d57d8b119e7320a6 192.168.0.252:6379 slots:5461-10922 (5462 slots) master M: faf50658fb7b0bae64cee5371da782e0f4919eee 192.168.0.253:6379 slots:10923-16383 (5461 slots) master S: c567db02cc40eebf577f71f703214dd2f4f26dfb 192.168.0.251:6380 replicates de461d3337b17d2119b79024d57d8b119e7320a6 S: 284f8196b250ad9ac272316db84a07bebf661ab7 192.168.0.252:6380 replicates 150f77d1000003811fb3c38c3768526a0b25ec31 S: 39fdef9fd5778dc94d8add819789d7d73ca06899 192.168.0.253:6380 replicates faf50658fb7b0bae64cee5371da782e0f4919eee Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join.... >>> Performing Cluster Check (using node 192.168.0.251:6379) M: 150f77d1000003811fb3c38c3768526a0b25ec31 192.168.0.251:6379 slots:0-5460 (5461 slots) master M: de461d3337b17d2119b79024d57d8b119e7320a6 192.168.0.252:6379 slots:5461-10922 (5462 slots) master M: faf50658fb7b0bae64cee5371da782e0f4919eee 192.168.0.253:6379 slots:10923-16383 (5461 slots) master M: c567db02cc40eebf577f71f703214dd2f4f26dfb 192.168.0.251:6380 slots: (0 slots) master replicates de461d3337b17d2119b79024d57d8b119e7320a6 M: 284f8196b250ad9ac272316db84a07bebf661ab7 192.168.0.252:6380 slots: (0 slots) master replicates 150f77d1000003811fb3c38c3768526a0b25ec31 M: 39fdef9fd5778dc94d8add819789d7d73ca06899 192.168.0.253:6380 slots: (0 slots) master replicates faf50658fb7b0bae64cee5371da782e0f4919eee [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. |
10.3. ps aux|grep redis
| [test@test-168-251 ~]$ ps aux|grep redis test 3824 0.7 5.9 6742404 3885144 ? Ssl 2015 1639:13 /data/redis/bin/redis-server *:6379 [cluster] test 3831 0.5 3.9 6709636 2618536 ? Ssl 2015 1235:43 /data/redis/bin/redis-server *:6380 [cluster] |
停止redis實例，直接使用kill命令即可，如：kill 3831，重啟和單機版相同，經過上述一系列操作后，重啟會自動轉換成cluster模式。。
11. redis cluster client
11.1. 命令行工具redis-cli
官方提供的命令行客戶端工具，在單機版redis基礎上指定參數“-c”即可。以下是在192.168.0.251上執行redis-cli的記錄：
| $ ./redis-cli -c -p 6379 127.0.0.1:6379> set foo bar -> Redirected to slot [12182] located at 192.168.0.253:6379 OK 192.168.0.253:6379> set hello world -> Redirected to slot [866] located at 192.168.0.251:6379 OK 192.168.0.251:6379> get foo -> Redirected to slot [12182] located at 192.168.0.253:6379 "bar" 192.168.0.253:6379> get hello -> Redirected to slot [866] located at 192.168.0.251:6379 "world"
查看集群中的節點： 192.168.0.251:6379> cluster nodes |
11.2. 從slaves讀數據
默認不能從slaves讀取數據，但建立連接后，執行一次命令READONLY ，即可從slaves讀取數據。如果想再次恢復不能從slaves讀取數據，可以執行下命令READWRITE。
11.3. jedis（java cluster client）
官網：https://github.com/xetorthio/jedis
編程示例：
| Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>(); //Jedis Cluster will attempt to discover cluster nodes automatically jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7379)); JedisCluster jc = new JedisCluster(jedisClusterNodes); jc.set("foo", "bar"); String value = jc.get("foo"); |
11.4. r3c（C++ cluster client）
官網：https://github.com/eyjian/r3c
12. 新增節點
12.1. 添加一個新主（master）節點
先以單機版配置和啟動好redis-server，然后執行命令：
| ./redis-trib.rb add-node 127.0.0.1:7006 127.0.0.1:7000 |
執行上面這條命令時，可能遇到錯誤“[ERR] Sorry, can't connect to node 127.0.0.1:7006”。引起該問題的原因可能是因為ruby的版本過低（運行ruby -v可以查看ruby的版本），可以嘗試升級ruby再嘗試，比如ruby 1.8.7版本就需要升級。對於Redis 3.0.5和Redis 3.2.0，使用Ruby 2.3.1操作正常。請注意升級到最新版本的ruby也可能遇到這個錯誤。
另一個會引起這個問題的原因是從Redis 3.2.0版本開始引入了“保護模式（protected mode），防止redis-cli遠程訪問”，僅限redis-cli綁定到127.0.0.1才可以連接Redis server。
為了完成添加新主節點，可以暫時性的關閉保護模式，使用redis-cli，不指定-h參數（但可以指定-p參數，或者-h參數值為127.0.0.1）進入操作界面：CONFIG SET protected-mode no。
注意7006是新增的節點，而7000是已存在的節點（可為master或slave）。如果需要將7006變成某master的slave節點，執行命令：
| cluster replicate 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e |
新加入的master節點上沒有任何數據（slots，運行redis命令cluster nodes可以看到這個情況）。當一個slave想成為master時，由於這個新的master節點不管理任何slots，它不參與選舉。
可以使用工具redis-trib.rb的resharding特性為這個新master節點分配slots，如：
redis-trib.rb reshard 127.0.0.1:7000，其中7000為集群中任意一個節點即可，redis-trib.rb將自動發現其它節點。
在reshard過程中，將會詢問reshard多少slots：
How many slots do you want to move (from 1 to 16384)?，取值范圍為1~16384，其中16384為redis cluster的擁有的slots總數，比如想只移動100個，輸入100即可。如果遷移的slots數量多，應當設置redis-trib.rb的超時參數--timeout值大一點。否則，遷移過程中易遇到超時錯誤“[ERR] Calling MIGRATE: IOERR error or timeout reading to target instance”，導致只完成部分，可能會造成數據丟失。
接着，會提示“What is the receiving node ID?”，輸入新加入的master節點ID。過程中如果遇到錯誤“Sorry, can't connect to node 10.225.168.253:6380”，則可能需要暫時先關閉相應的保護模式。
如果在遷移過程遇到下面這樣的錯誤：
| >>> Check for open slots... [WARNING] Node 192.168.0.3:6379 has slots in importing state (5461). [WARNING] Node 192.168.0.5:6380 has slots in migrating state (5461). [WARNING] The following slots are open: 5461 |
可以考慮使用命令“redis-trib.rb fix 192.168.0.3:6379”嘗試修復。需要顯示有節點處於migrating或importing狀態，可以登錄到相應的節點，使用命令“cluster setslot 5461 stable”修改，參數5461為問題顯示的slot的ID。
12.2. 添加一個新從（slave）節點
| ./redis-trib.rb add-node --slave 127.0.0.1:7006 127.0.0.1:7000 |
注意這種方式，如果添加了多個slave節點，可能導致master的slaves不均衡，比如一些有3個slave，其它只1個slave。可以在slave節點上執行redis命令“CLUSTER REPLICATE”進行調整，讓它成為其它master的slave。“CLUSTER REPLICATE”帶一個參數，即master ID，注意使用redis-cli -c登錄到slave上執行。
上面方法沒有指定7006的master，而是隨機指定。下面方法可以明確指定為哪個master的slave：
| ./redis-trib.rb add-node --slave --master-id 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 127.0.0.1:7006 127.0.0.1:7000 |
其中“127.0.0.1:7006”是新節點，“127.0.0.1:7000”是集群中已有的節點。
13. 刪除節點
從集群中刪除一個節點：
| ./redis-trib.rb del-node 127.0.0.1:7000 <node-id> |
第一個參數為集群中任意一個節點，第二個參數為需要刪除節點的ID。
成功刪除后，提示：
| $./redis-trib.rb del-node 127.0.0.1:6380 f49a2bda05e81aa343adb9924775ba95a1f4236e >>> Removing node f49a2bda05e81aa343adb9924775ba95a1f4236e from cluster 127.0.0.1:6379 /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182: warning: wrong element type nil at 0 (expected array) 。。。 。。。 /usr/local/share/gems/gems/redis-3.0.0/lib/redis.rb:182: warning: this causes ArgumentError in the next release >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node. 在這里會停頓幾分鍾，通知並等待被刪除節點退出（exit），被刪除節點在將數據寫到RDB文件中后退出，所以停頓時長和寫RDB文件時長有關，數據量越大時間就越長。5~6G的RAID1的SATA盤數據大概需要45秒左右。 |
被刪除節點日志：
| 15577:S 06 Sep 20:06:37.774 - Accepted 10.49.126.98:14669 15577:S 06 Sep 20:06:38.741 # User requested shutdown... 15577:S 06 Sep 20:06:38.741 * Calling fsync() on the AOF file. 15577:S 06 Sep 20:06:38.742 * Saving the final RDB snapshot before exiting. 15577:S 06 Sep 20:07:19.683 * DB saved on disk 15577:S 06 Sep 20:07:19.683 * Removing the pid file. 15577:S 06 Sep 20:07:19.683 # Redis is now ready to exit, bye bye... |
成功后不用再調用“CLUSTER FORGET”，否則報錯：
| $ redis-cli -c CLUSTER FORGET aa6754a093ea4047f92cc0ea77f1859553bc5c57 (error) ERR Unknown node aa6754a093ea4047f92cc0ea77f1859553bc5c57 |
如果待刪除節點已經不能連接，則調用CLUSTER FORGET剔除（可能需要在所有機器上執行一次FORGET）：
| CLUSTER FORGET <node-id> |
注意如果是刪除一個master節點，則需要先將它管理的slots的遷走，然后才可以刪除它。
如果是master或slave機器不能連接，比如硬件故障導致無法啟動，這個時候做不了del-node，只需要直接做CLUSTER 即可，在FORGET后，節點狀態變成handshake。
！！！請注意，需要在所有node上執行一次“CLUSTER FORGET”，否則可能遇到被剔除node的總是處於handshake狀態。
如果有部分node沒有執行到FORGET，導致有部分node還處於fail狀態，則在一些node將看到待剔除節點仍然處於handshake狀態，並且nodeid在不斷變化，所以需要在所有node上執行“CLUSTER FORGET”。
如果一個節點處於“:0 master,fail,noaddr”狀態，執行“del-node”會報錯：
[ERR] No such node ID 80560d0d97a0b3fa975203350516437b58251745
這種情況下，只需要執行“CLUSTER FORGET”將其剔除即可（注意，需要在所有節點上執行一次，不然未執行的節點上可能仍然看得到“:0 master,fail,noaddr”）：
| # redis-cli -c -p 1383 cluster nodes 80560d0d97a0b3fa975203350516437b58251745 :0 master,fail,noaddr - 1528947205054 1528947203553 0 disconnected fa7bbbf7d48389409ce05d303272078c3a6fd44f 127.0.0.1:1379 slave 689f7c1ae71ea294c4ad7c5d1b32ae4e78e27915 0 1535871825187 138 connected c1a9d1d23438241803ec97fbd765737df80f402a 127.0.0.1:1381 slave f03b1008988acbb0f69d96252decda9adf747be9 0 1535871826189 143 connected 50003ccd5885771196e717e27011140e7d6c94e0 127.0.0.1:1385 slave f03b1008988acbb0f69d96252decda9adf747be9 0 1535871825688 143 connected f6080015129eada3261925cc1b466f1824263358 127.0.0.1:1380 slave 4e932f2a3d80de29798660c5ea62e473e63a6630 0 1535871825388 145 connected 4e932f2a3d80de29798660c5ea62e473e63a6630 127.0.0.1:1383 myself,master - 0 0 145 connected 5458-10922 689f7c1ae71ea294c4ad7c5d1b32ae4e78e27915 127.0.0.1:1382 master - 0 1535871826490 138 connected 0-1986 1988-5457 f03b1008988acbb0f69d96252decda9adf747be9 127.0.0.1:1384 master - 0 1535871825187 143 connected 1987 10923-16383 |
14. master機器硬件故障
這種情況下，master機器可能無法啟動，導致其上的master無法連接，master將一直處於“master,fail”狀態，如果是slave則處於“slave,fail”狀態。
如果是master，則會它的slave變成了master，因此只需要添加一個新的從節點作為原slave（已變成master）的slave節點。完成后，通過CLUSTER FORGET將故障的master或slave從集群中剔除即可。
！！！請注意，需要在所有node上執行一次“CLUSTER FORGET”，否則可能遇到被剔除node的總是處於handshake狀態。
15. 檢查節點狀態
redis-trib.rb check 127.0.0.1:6380
如發現如下這樣的錯誤：
[WARNING] Node 192.168.0.11:6380 has slots in migrating state (5461).
[WARNING] The following slots are open: 5461
可以使用redis命令取消slots遷移（5461為slot的ID）：
cluster setslot 5461 stable
需要注意，須登錄到192.168.0.11:6380上執行redis的setslot子命令。
16. 變更主從關系
使用命令cluster replicate，參數為master節點ID，注意不是IP和端口，在被遷移的slave上執行該命令。
17. slots相關命令
| CLUSTER ADDSLOTS slot1 [slot2] ... [slotN] CLUSTER DELSLOTS slot1 [slot2] ... [slotN] CLUSTER SETSLOT slot NODE node CLUSTER SETSLOT slot MIGRATING node CLUSTER SETSLOT slot IMPORTING node |
17.1. 遷移slosts
官方參考：https://redis.io/commands/cluster-setslot。
示例：將值為8的slot從源節點A遷移到目標節點B，有如下兩種方法：
| 在目標節點B上執行：CLUSTER SETSLOT 8 IMPORTING src-A-node-id 或 在源節點A上執行：CLUSTER SETSLOT 8 MIGRATING dst-B-node-id |
上述操作只是將slot標記為遷移狀態，完成遷移還需要執行（在目標node上執行）：
| CLUSTER SETSLOT <slot> NODE <dst-node-id> |
其中node-id為目標的Node ID，取消遷移使用“CLUSTER SETSLOT <slot> STABLE”。
操作示例：
| # 將值為11677的slot遷到192.168.31.3:6379 $ redis-cli -c -h 192.168.31.3 -p 6379 CLUSTER SETSLOT 11677 IMPORTING 216e0069af11eca91465394b2ad7bf1c27f5f7fe OK $ redis-cli -c -h 192.168.31.3 -p 6379 CLUSTER SETSLOT 11677 NODE 4e149c72aff2b6651370ead476dd70c8cf9e3e3c OK |
17.2. redis-trib.rb rebalance
當有增減節點時，可以使用命令：
redis-trib.rb rebalance 192.168.0.31:6379 --auto-weights
做一次均衡，簡單點可以只指定兩個參數：“192.168.0.31:6379”為集群中已知的任何一個節點，參數“-auto-weights”表示自動權重。
18. 人工主備切換
在需要的slaves節點上執行命令：CLUSTER FAILOVER。如果人工發起failover，則其它master會收到“Failover auth granted to 4291f18b5e9729e832ed15ceb6324ce5dfc2ffbe for epoch 31”，每次epoch值增一。
| 23038:M 06 Sep 20:31:24.815 # Failover auth granted to 4291f18b5e9729e832ed15ceb6324ce5dfc2ffbe for epoch 31
當出現下面兩條日志時，表示failover完成： 23038:M 06 Sep 20:32:44.019 * FAIL message received from ea28f68438e5bb79c26a9cb2135241f11d7a50ba about 5e6ffacb2c5d5761e39aba5270fbf48f296cb5ee 23038:M 06 Sep 20:32:58.487 * Clear FAIL state for node 5e6ffacb2c5d5761e39aba5270fbf48f296cb5ee: slave is reachable again. |
原master收到failover后的日志：
| 35475:M 06 Sep 20:35:43.396 - DB 0: 16870482 keys (7931571 volatile) in 50331648 slots HT. 35475:M 06 Sep 20:35:43.396 - 1954 clients connected (1 slaves), 5756515544 bytes in use 35475:M 06 Sep 20:35:48.083 # Manual failover requested by slave 58a40dbe01e1563773724803854406df04c62724. 35475:M 06 Sep 20:35:48.261 # Failover auth granted to 58a40dbe01e1563773724803854406df04c62724 for epoch 32 35475:M 06 Sep 20:35:48.261 - Client closed connection
10.51.147.216:7388為failover前的slave， 10.51.147.216:7388的ID為58a40dbe01e1563773724803854406df04c62724 35475:M 06 Sep 20:35:48.261 # Connection with slave 10.51.147.216:7388 lost. 35475:M 06 Sep 20:35:48.278 # Configuration change detected. Reconfiguring myself as a replica of 58a40dbe01e1563773724803854406df04c62724 35475:S 06 Sep 20:35:48.280 - Client closed connection 35475:S 06 Sep 20:35:48.408 - DB 0: 16870296 keys (7931385 volatile) in 50331648 slots HT. 35475:S 06 Sep 20:35:48.408 - 1953 clients connected (0 slaves), 5722753736 bytes in use 35475:S 06 Sep 20:35:48.408 * Connecting to MASTER 10.51.147.216:7388 35475:S 06 Sep 20:35:48.408 * MASTER <-> SLAVE sync started 35475:S 06 Sep 20:35:48.408 * Non blocking connect for SYNC fired the event. 35475:S 06 Sep 20:35:48.408 * Master replied to PING, replication can continue... 35475:S 06 Sep 20:35:48.408 * Partial resynchronization not possible (no cached master) 35475:S 06 Sep 20:35:48.459 * Full resync from master: 36beb63d32b3809039518bf4f3e4e10de227f3ee:16454238619 35475:S 06 Sep 20:35:48.493 - Client closed connection 35475:S 06 Sep 20:35:48.880 - Client closed connection |
19. 查看集群信息
對應的redis命令為：cluster info，示例：
| 127.0.0.1:6381> cluster info cluster_state:ok 所有slots正常則顯示為OK，否則為error cluster_slots_assigned:16384 多少slots被分配了，即多少被master管理了，16384為全部slots cluster_slots_ok:16384 有多少slots是正常的 cluster_slots_pfail:0 有多少slots可能處於異常狀態，處於這個狀態並不表示有問題，仍能繼續提供服務 cluster_slots_fail:0 有多少slots處於異常狀態，需要修復才能服務 cluster_known_nodes:10 集群中的節點數 cluster_size:3 集群中master個數 cluster_current_epoch:11 本地的當前時間變量，用於故障切換時生成獨一無二的增量版本號 cluster_my_epoch:0 cluster_stats_messages_sent:4049 通過集群消息總線發送的消息總數 cluster_stats_messages_received:4051 通過過集通過群消息總線收到的消息總數 |
20. 禁止指定命令
KEYS命令很耗時，FLUSHDB和FLUSHALL命令可能導致誤刪除數據，所以線上環境最好禁止使用，可以在Redis配置文件增加如下配置：
| rename-command KEYS "" rename-command FLUSHDB "" rename-command FLUSHALL "" |
21. 各版本配置文件
https://raw.githubusercontent.com/antirez/redis/3.0/redis.conf
https://raw.githubusercontent.com/antirez/redis/3.2.9/redis.conf
https://raw.githubusercontent.com/antirez/redis/4.0/redis.conf
https://raw.githubusercontent.com/antirez/redis/4.0.1/redis.conf
https://raw.githubusercontent.com/antirez/redis/4.0.3/redis.conf
https://raw.githubusercontent.com/antirez/redis/4.0.5/redis.conf
https://raw.githubusercontent.com/antirez/redis/4.0.9/redis.conf
https://raw.githubusercontent.com/antirez/redis/4.0.11/redis.conf
22. 大壓力下Redis參數調整要點
| 參數 |
建議最小值 |
說明 |
| repl-ping-slave-period |
10 |
每10秒ping一次 |
| repl-timeout |
60 |
60秒超時，也就是ping十次 |
| cluster-node-timeout |
15000 |
|
| repl-backlog-size |
1GB |
Master對slave的隊列大小 |
| appendfsync |
no |
讓系統自動刷 |
| save |
|
大壓力下，調大參數值，以減少寫RDB帶來的壓力： "900 20 300 200 60 200000" |
| appendonly |
|
對於隊列，建議單獨建立集群，並且設置該值為no |
為何大壓力下要這樣調整？
最重要的原因之一Redis的主從復制，兩者復制共享同一線程，雖然是異步復制的，但因為是單線程，所以也十分有限。如果主從間的網絡延遲不是在0.05左右，比如達到0.6，甚至1.2等，那么情況是非常糟糕的，因此同一Redis集群一定要部署在同一機房內。
這些參數的具體值，要視具體的壓力而定，而且和消息的大小相關，比如一條200~500KB的流水數據可能比較大，主從復制的壓力也會相應增大，而10字節左右的消息，則壓力要小一些。大壓力環境中開啟appendfsync是十分不可取的，容易導致整個集群不可用，在不可用之前的典型表現是QPS毛刺明顯。
這么做的目的是讓Redis集群盡可能的避免master正常時觸發主從切換，特別是容納的數據量很大時，和大壓力結合在一起，集群會雪崩。
當Redis日志中，出現大量如下信息，即可能意味着相關的參數需要調整了：
| 22135:M 06 Sep 14:17:05.388 * FAIL message received from 1d07e208db56cfd7395950ca66e03589278b8e12 about e438a338e9d9834a6745c12931950da87e360ca2 22135:M 06 Sep 14:17:07.551 * FAIL message received from ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 about d6eb06e9d118c120d3961a659972a1d0191a8652 22135:M 06 Sep 14:17:08.438 # Failover auth granted to f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f for epoch 285 （We can vote for this slave）
有投票資格的node： 1）為master 2）至少有一個slot 3）投票node的epoch不能小於node自己當前的epoch（reqEpoch < curEpoch） 4）node沒有投票過該epoch（already voted for epoch） 5）投票node不能為master（it is a master node） 6）投票node必須有一個master（I don't know its master） 7）投票node的master處於fail狀態（its master is up）
22135:M 06 Sep 14:17:19.844 # Failover auth denied to 534b93af6ba45a7033dbf38c8f47cd688514125a: already voted for epoch 285
如果一個node又聯系上了，則它當是一個slave，或者無slots的master時，直接清除FAIL標志；但如果是一個master，則當“(now - node->fail_time) > (server.cluster_node_timeout * CLUSTER_FAIL_UNDO_TIME_MULT)”時，也清除FAIL標志，定義在cluster.h中（cluster.h:#define CLUSTER_FAIL_UNDO_TIME_MULT 2 /* Undo fail if master is back. */） 22135:M 06 Sep 14:17:29.243 * Clear FAIL state for node d6eb06e9d118c120d3961a659972a1d0191a8652: master without slots is reachable again.
如果消息類型為fail。 22135:M 06 Sep 14:17:31.995 * FAIL message received from f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f about 1ba437fa1683a8caafd38ff977e5fbabdaf84fd6 22135:M 06 Sep 14:17:32.496 * FAIL message received from 1d07e208db56cfd7395950ca66e03589278b8e12 about d7942cfe636b25219c6d56aa72828fcfde2ee261 22135:M 06 Sep 14:17:32.968 # Failover auth granted to 938d9ae2de278938beda1d39185608b02d3b31ec for epoch 286 22135:M 06 Sep 14:17:33.177 # Failover auth granted to d9dadf3342006e2c92def3071ca0a76390be62b0 for epoch 287 22135:M 06 Sep 14:17:36.336 * Clear FAIL state for node 1ba437fa1683a8caafd38ff977e5fbabdaf84fd6: master without slots is reachable again. 22135:M 06 Sep 14:17:36.855 * Clear FAIL state for node d7942cfe636b25219c6d56aa72828fcfde2ee261: master without slots is reachable again. 22135:M 06 Sep 14:17:38.419 * Clear FAIL state for node e438a338e9d9834a6745c12931950da87e360ca2: is reachable again and nobody is serving its slots after some time. 22135:M 06 Sep 14:17:54.954 * FAIL message received from ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 about 7990d146cece7dc83eaf08b3e12cbebb2223f5f8 22135:M 06 Sep 14:17:56.697 * FAIL message received from 1d07e208db56cfd7395950ca66e03589278b8e12 about fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5 22135:M 06 Sep 14:17:57.705 # Failover auth granted to e1c202d89ffe1c61b682e28071627635974c84a7 for epoch 288 22135:M 06 Sep 14:17:57.890 * Clear FAIL state for node 7990d146cece7dc83eaf08b3e12cbebb2223f5f8: slave is reachable again. 22135:M 06 Sep 14:17:57.892 * Clear FAIL state for node fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5: master without slots is reachable again. |
23. 問題排查
1) 如果最后一條日志為“16367:M 08 Jun 14:48:15.560 # Server started, Redis version 3.2.0”，節點狀態始終終於fail狀態，則可能是aof文件損壞了，這時可以使用工具edis-check-aof --fix進行修改，如：
../../bin/redis-check-aof --fix appendonly-6380.aof
0x a1492b9b: Expected prefix '
AOF analyzed: size=2705928192, ok_up_to=2705927067, diff=1125
This will shrink the AOF from 2705928192 bytes, with 1125 bytes, to 2705927067 bytes
Continue? [y/N]: y
2) in `call': ERR Slot 16011 is already busy (Redis::CommandError)
將所有節點上的配置項cluster-config-file指定的文件刪除，然后重新啟；或者在所有節點上執行下FLUSHALL命令。
另外，如果使用主機名而不是IP，也可能遇到這個錯誤，如：“redis-trib.rb create --replicas 1 redis1:6379 redis2:6379 redis3:6379 redis4:6379 redis5:6379 redis6:6379”，可能也會得到錯誤“ERR Slot 16011 is already busy (Redis::CommandError)”。
3) for lack of backlog (Slave request was: 51875158284)
默認值：
# redis-cli config get repl-timeout
A) "repl-timeout"
B) "10"
# redis-cli config get client-output-buffer-limit
A) "client-output-buffer-limit"
B) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"
增大：
redis-cli config set "client-output-buffer-limit" "normal 0 0 0 slave 2684354560 671088640 60 pubsub 33554432 8388608 60"
4) 復制中斷場景
A) master的slave緩沖區達到限制的硬或軟限制大小，與參數client-output-buffer-limit相關；
B) 復制時間超過repl-timeout指定的值，與參數repl-timeout相關。
slave反復循環從master復制，如果調整以上參數仍然解決不了，可以嘗試刪除slave上的aof和rdb文件，然后再重啟進程復制，這個時候可能能正常完成復制。
5) 日志文件出現：Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
考慮優化以下配置項：
no-appendfsync-on-rewrite值設為yes
repl-backlog-size和client-output-buffer-limit調大一點
6) 日志文件出現：MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
考慮設置stop-writes-on-bgsave-error值為“no”。
7) Failover auth granted to
當日志大量反反復復出現下列內容時，很可能表示master和slave間同步和通訊不順暢，導致無效的failover和狀態變更，這個時候需要調大相關參數值，容忍更長的延遲，因此也特別注意集群內所有節點間的網絡延遲要盡可能的小，最好達到0.02ms左右的水平，調大參數的代價是主備切換變遲鈍。
Slave日志：
| 31019:S 06 Sep 11:07:24.169 * Connecting to MASTER 10.5.14.8:6379 31019:S 06 Sep 11:07:24.169 * MASTER <-> SLAVE sync started 31019:S 06 Sep 11:07:24.169 # Start of election delayed for 854 milliseconds (rank #0, offset 5127277817). 31019:S 06 Sep 11:07:24.169 * Non blocking connect for SYNC fired the event. 31019:S 06 Sep 11:07:25.069 # Starting a failover election for epoch 266. 31019:S 06 Sep 11:07:29.190 * Clear FAIL state for node ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467: is reachable again and nobody is serving its slots after some time. 31019:S 06 Sep 11:07:29.191 * Master replied to PING, replication can continue... 31019:S 06 Sep 11:07:29.191 * Clear FAIL state for node f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f: is reachable again and nobody is serving its slots after some time. 31019:S 06 Sep 11:07:29.192 * Trying a partial resynchronization (request ea2261c827fbc54135a95f707046581a55dff133:5127277818). 31019:S 06 Sep 11:07:29.192 * Successful partial resynchronization with master. 31019:S 06 Sep 11:07:29.192 * MASTER <-> SLAVE sync: Master accepted a Partial Resynchronization. 31019:S 06 Sep 11:07:29.811 * Clear FAIL state for node e438a338e9d9834a6745c12931950da87e360ca2: is reachable again and nobody is serving its slots after some time. 31019:S 06 Sep 11:07:37.680 * FAIL message received from 5b41f7860cc800e65932e92d1d97c6c188138e56 about 3114cec541c5bcd36d712cd6c9f4c5055510e386 31019:S 06 Sep 11:07:43.710 * Clear FAIL state for node 3114cec541c5bcd36d712cd6c9f4c5055510e386: slave is reachable again. 31019:S 06 Sep 11:07:48.119 * FAIL message received from 7d61af127c17d9c19dbf9af0ac8f7307f1c96c4b about e1c202d89ffe1c61b682e28071627635974c84a7 31019:S 06 Sep 11:07:49.410 * FAIL message received from 5b41f7860cc800e65932e92d1d97c6c188138e56 about d9dadf3342006e2c92def3071ca0a76390be62b0 31019:S 06 Sep 11:07:53.352 * Clear FAIL state for node d9dadf3342006e2c92def3071ca0a76390be62b0: slave is reachable again. 31019:S 06 Sep 11:07:57.147 * Clear FAIL state for node e1c202d89ffe1c61b682e28071627635974c84a7: slave is reachable again. 31019:S 06 Sep 11:08:36.516 * FAIL message received from ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 about 938d9ae2de278938beda1d39185608b02d3b31ec 31019:S 06 Sep 11:08:41.900 * Clear FAIL state for node 938d9ae2de278938beda1d39185608b02d3b31ec: slave is reachable again. 31019:S 06 Sep 11:08:46.380 * FAIL message received from d7942cfe636b25219c6d56aa72828fcfde2ee261 about fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5 31019:S 06 Sep 11:08:46.531 * Marking node 7990d146cece7dc83eaf08b3e12cbebb2223f5f8 as failing (quorum reached). 31019:S 06 Sep 11:09:01.882 * Clear FAIL state for node 7990d146cece7dc83eaf08b3e12cbebb2223f5f8: master without slots is reachable again. 31019:S 06 Sep 11:09:01.883 * Clear FAIL state for node fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5: master without slots is reachable again. 31019:S 06 Sep 11:09:06.538 * FAIL message received from e438a338e9d9834a6745c12931950da87e360ca2 about d7942cfe636b25219c6d56aa72828fcfde2ee261 31019:S 06 Sep 11:09:06.538 * FAIL message received from e438a338e9d9834a6745c12931950da87e360ca2 about 1ba437fa1683a8caafd38ff977e5fbabdaf84fd6 31019:S 06 Sep 11:09:12.555 * Clear FAIL state for node 1ba437fa1683a8caafd38ff977e5fbabdaf84fd6: is reachable again and nobody is serving its slots after some time. 31019:S 06 Sep 11:09:12.555 * Clear FAIL state for node d7942cfe636b25219c6d56aa72828fcfde2ee261: master without slots is reachable again. 31019:S 06 Sep 11:09:15.565 * Marking node 534b93af6ba45a7033dbf38c8f47cd688514125a as failing (quorum reached). 31019:S 06 Sep 11:09:16.599 * FAIL message received from 0a92bd7472c9af3e52f9185eac1bd1bbf36146e6 about e1c202d89ffe1c61b682e28071627635974c84a7 31019:S 06 Sep 11:09:22.262 * Clear FAIL state for node 534b93af6ba45a7033dbf38c8f47cd688514125a: slave is reachable again. 31019:S 06 Sep 11:09:27.906 * Clear FAIL state for node e1c202d89ffe1c61b682e28071627635974c84a7: is reachable again and nobody is serving its slots after some time. 31019:S 06 Sep 11:09:50.744 * FAIL message received from ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 about e1c202d89ffe1c61b682e28071627635974c84a7 31019:S 06 Sep 11:09:55.141 * FAIL message received from 5b41f7860cc800e65932e92d1d97c6c188138e56 about d9dadf3342006e2c92def3071ca0a76390be62b0 31019:S 06 Sep 11:09:55.362 * FAIL message received from 7d61af127c17d9c19dbf9af0ac8f7307f1c96c4b about 938d9ae2de278938beda1d39185608b02d3b31ec 31019:S 06 Sep 11:09:55.557 * FAIL message received from ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 about 1d07e208db56cfd7395950ca66e03589278b8e12 31019:S 06 Sep 11:09:55.578 * FAIL message received from ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 about 144347d5a51acf047887fe81f22e8f7705c911ec 31019:S 06 Sep 11:09:56.521 * Marking node 534b93af6ba45a7033dbf38c8f47cd688514125a as failing (quorum reached). 31019:S 06 Sep 11:09:57.996 * Clear FAIL state for node 1d07e208db56cfd7395950ca66e03589278b8e12: slave is reachable again. 31019:S 06 Sep 11:09:58.329 * FAIL message received from 5b41f7860cc800e65932e92d1d97c6c188138e56 about 0a92bd7472c9af3e52f9185eac1bd1bbf36146e6 31019:S 06 Sep 11:10:09.239 * Clear FAIL state for node 144347d5a51acf047887fe81f22e8f7705c911ec: slave is reachable again. 31019:S 06 Sep 11:10:09.812 * Clear FAIL state for node d9dadf3342006e2c92def3071ca0a76390be62b0: is reachable again and nobody is serving its slots after some time. 31019:S 06 Sep 11:10:13.549 * Clear FAIL state for node 534b93af6ba45a7033dbf38c8f47cd688514125a: slave is reachable again. 31019:S 06 Sep 11:10:13.590 * FAIL message received from 716f2e2dd9792eaf4ee486794c9797fa6e1c9650 about 1ba437fa1683a8caafd38ff977e5fbabdaf84fd6 31019:S 06 Sep 11:10:13.591 * FAIL message received from f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f about d7942cfe636b25219c6d56aa72828fcfde2ee261 31019:S 06 Sep 11:10:14.316 * Clear FAIL state for node e1c202d89ffe1c61b682e28071627635974c84a7: is reachable again and nobody is serving its slots after some time. 31019:S 06 Sep 11:10:15.108 * Clear FAIL state for node d7942cfe636b25219c6d56aa72828fcfde2ee261: slave is reachable again. 31019:S 06 Sep 11:10:17.588 * Clear FAIL state for node 938d9ae2de278938beda1d39185608b02d3b31ec: slave is reachable again. 31019:S 06 Sep 11:10:32.622 * Clear FAIL state for node 0a92bd7472c9af3e52f9185eac1bd1bbf36146e6: slave is reachable again. 31019:S 06 Sep 11:10:32.623 * FAIL message received from 5b41f7860cc800e65932e92d1d97c6c188138e56 about 3114cec541c5bcd36d712cd6c9f4c5055510e386 31019:S 06 Sep 11:10:32.623 * Clear FAIL state for node 3114cec541c5bcd36d712cd6c9f4c5055510e386: slave is reachable again. |
Master日志：
| 31014:M 06 Sep 14:08:54.083 * Background saving terminated with success 31014:M 06 Sep 14:09:55.093 * 10000 changes in 60 seconds. Saving... 31014:M 06 Sep 14:09:55.185 * Background saving started by pid 41395 31014:M 06 Sep 14:11:00.269 # Disconnecting timedout slave: 10.15.40.9:6018 31014:M 06 Sep 14:11:00.269 # Connection with slave 10.15.40.9:6018 lost. 41395:C 06 Sep 14:11:01.141 * DB saved on disk 41395:C 06 Sep 14:11:01.259 * RDB: 5 MB of memory used by copy-on-write 31014:M 06 Sep 14:11:01.472 * Background saving terminated with success 31014:M 06 Sep 14:11:11.525 * FAIL message received from 1d07e208db56cfd7395950ca66e03589278b8e12 about 534b93af6ba45a7033dbf38c8f47cd688514125a 31014:M 06 Sep 14:11:23.039 * FAIL message received from 1ba437fa1683a8caafd38ff977e5fbabdaf84fd6 about d78845370c98b3ce4cfc02e8d3e233a9a1d84a83 31014:M 06 Sep 14:11:23.541 * Clear FAIL state for node 534b93af6ba45a7033dbf38c8f47cd688514125a: slave is reachable again. 31014:M 06 Sep 14:11:23.813 * Slave 10.15.40.9:6018 asks for synchronization 31014:M 06 Sep 14:11:23.813 * Partial resynchronization request from 10.15.40.9:6018 accepted. Sending 46668 bytes of backlog starting from offset 5502672944. 31014:M 06 Sep 14:11:23.888 # Failover auth granted to 7d61af127c17d9c19dbf9af0ac8f7307f1c96c4b for epoch 283 31014:M 06 Sep 14:11:32.464 * FAIL message received from d6eb06e9d118c120d3961a659972a1d0191a8652 about 3114cec541c5bcd36d712cd6c9f4c5055510e386 31014:M 06 Sep 14:11:47.616 * Clear FAIL state for node d78845370c98b3ce4cfc02e8d3e233a9a1d84a83: master without slots is reachable again. 31014:M 06 Sep 14:11:55.515 * FAIL message received from d6eb06e9d118c120d3961a659972a1d0191a8652 about ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 31014:M 06 Sep 14:11:57.135 # Failover auth granted to ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 for epoch 284 31014:M 06 Sep 14:12:01.766 * Clear FAIL state for node ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467: slave is reachable again. 31014:M 06 Sep 14:12:08.753 * Clear FAIL state for node 3114cec541c5bcd36d712cd6c9f4c5055510e386: master without slots is reachable again. 31014:M 06 Sep 14:16:02.070 * 10 changes in 300 seconds. Saving... 31014:M 06 Sep 14:16:02.163 * Background saving started by pid 13832 31014:M 06 Sep 14:17:18.443 * FAIL message received from ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 about d6eb06e9d118c120d3961a659972a1d0191a8652 31014:M 06 Sep 14:17:18.443 # Failover auth granted to f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f for epoch 285 31014:M 06 Sep 14:17:29.272 # Connection with slave client id #40662 lost. 31014:M 06 Sep 14:17:29.273 # Failover auth denied to 534b93af6ba45a7033dbf38c8f47cd688514125a: already voted for epoch 285 31014:M 06 Sep 14:17:29.278 * Slave 10.15.40.9:6018 asks for synchronization 31014:M 06 Sep 14:17:29.278 * Partial resynchronization request from 10.15.40.9:6018 accepted. Sending 117106 bytes of backlog starting from offset 5502756264. 13832:C 06 Sep 14:17:29.850 * DB saved on disk 13832:C 06 Sep 14:17:29.970 * RDB: 7 MB of memory used by copy-on-write 31014:M 06 Sep 14:17:38.449 * FAIL message received from f7d6b2c72fa3b801e7dcfe0219e73383d143dd0f about 1ba437fa1683a8caafd38ff977e5fbabdaf84fd6 31014:M 06 Sep 14:17:38.449 * FAIL message received from 1d07e208db56cfd7395950ca66e03589278b8e12 about d7942cfe636b25219c6d56aa72828fcfde2ee261 31014:M 06 Sep 14:17:38.449 # Failover auth denied to 938d9ae2de278938beda1d39185608b02d3b31ec: reqEpoch (286) < curEpoch(287) 31014:M 06 Sep 14:17:38.449 # Failover auth granted to d9dadf3342006e2c92def3071ca0a76390be62b0 for epoch 287 31014:M 06 Sep 14:17:38.449 * Background saving terminated with success 31014:M 06 Sep 14:17:38.450 * Clear FAIL state for node d7942cfe636b25219c6d56aa72828fcfde2ee261: master without slots is reachable again. 31014:M 06 Sep 14:17:38.450 * Clear FAIL state for node 1ba437fa1683a8caafd38ff977e5fbabdaf84fd6: master without slots is reachable again. 31014:M 06 Sep 14:17:38.452 * Clear FAIL state for node d6eb06e9d118c120d3961a659972a1d0191a8652: slave is reachable again. 31014:M 06 Sep 14:17:54.985 * FAIL message received from ae8f6e7e0ab16b04414c8f3d08b58c0aa268b467 about 7990d146cece7dc83eaf08b3e12cbebb2223f5f8 31014:M 06 Sep 14:17:56.729 * FAIL message received from 1d07e208db56cfd7395950ca66e03589278b8e12 about fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5 31014:M 06 Sep 14:17:57.737 # Failover auth granted to e1c202d89ffe1c61b682e28071627635974c84a7 for epoch 288 31014:M 06 Sep 14:17:57.922 * Clear FAIL state for node fbe774cdbd2acd24f9f5ea90d61c607bdf800eb5: master without slots is reachable again. 31014:M 06 Sep 14:17:57.923 * Clear FAIL state for node 7990d146cece7dc83eaf08b3e12cbebb2223f5f8: slave is reachable again. |