Redis集群模式之分布式集群模式

前言

Redis集群模式主要有2種:

• 主從集群
• 分布式集群。

前者主要是為了高可用或是讀寫分離，后者為了更好的存儲數據，負載均衡。 
本文主要講解主從集群。本章主要講解后一半部分，Redis集群。

與本文相關的代碼與配置文件都已經上傳至github上: 
地址: https://github.com/SeanYanxml/bigdata

---

原理

Redis為了實現負載均衡，提供集群模式。以三個節點為例，集群模式相當於將1-15000片分片，分為1-5000、5000-10000、10000-15000。每個節點分一段數據片。這樣的話，當一個節點宕機后，這個節點沒有備份的話，此段分片將不再可以使用。所以，官方推薦，集群內的每個節點都應該配備一個從節點，作為冷備。部署原理圖如下所示(暫略)。

---

部署

由於沒有那么多的機器，所以我們一般單機部署6個節點(3主3從)，也就是官網推薦的模式。

主要步驟如下:

• 安裝ruby，因為分配集群的代碼時ruby腳本；
• 分配集群
• 查看集群結果

---

部署代碼

# mkdir cluster-test # cd cluster-test # mkdir 7000 7001 7002 7003 7004 7005 # 在文件夾內 分別放置redis.conf文件 文件內容見下 # 啟動6個節點 # nohup ../../src/redis-server redis.conf > start.log 2>&1 & ＃ 使用ruby腳本分配集群資源 ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

# redis.conf
port 7000 cluster-enabled yes cluster-config-file nodes.conf cluster-node-timeout 5000 appendonly yes

### 啟動redis進程7000 7001 7002 7003 7004 7005 touch start-all.sh ### 內容如下 cd 7000 nohup ../../src/redis-server redis.conf > start.log 2>&1 & cd ../7001 nohup ../../src/redis-server redis.conf > start.log 2>&1 & cd ../7002 nohup ../../src/redis-server redis.conf > start.log 2>&1 & cd ../7003 nohup ../../src/redis-server redis.conf > start.log 2>&1 & cd ../7004 nohup ../../src/redis-server redis.conf > start.log 2>&1 & cd ../7005 nohup ../../src/redis-server redis.conf > start.log 2>&1 & chmod +x start-all.sh ./start-all.sh ps -ef|grep redis 查看進程 [cpic@cpic-redis2-77 src]$ ps -ef|grep redis cpic 19876 1 0 3月21 ? 05:14:19 ./redis-server *:6379 cpic 19886 1 0 3月21 ? 06:17:36 ./redis-sentinel *:26379 [sentinel] cpic 25971 1 0 15:07 pts/0 00:00:00 ../../src/redis-server *:7000 [cluster] cpic 25972 1 0 15:07 pts/0 00:00:00 ../../src/redis-server *:7001 [cluster] cpic 25973 1 0 15:07 pts/0 00:00:00 ../../src/redis-server *:7002 [cluster] cpic 25974 1 0 15:07 pts/0 00:00:00 ../../src/redis-server *:7003 [cluster] cpic 25975 1 0 15:07 pts/0 00:00:00 ../../src/redis-server *:7004 [cluster] cpic 25976 1 0 15:07 pts/0 00:00:00 ../../src/redis-server *:7005 [cluster] cpic 26273 24815 0 15:15 pts/0 00:00:00 grep --color=auto redis

 

成功結果

＃ create 腳本初始化后腳本 [root@cpic-redis-76 src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 Adding replica 127.0.0.1:7003 to 127.0.0.1:7000 Adding replica 127.0.0.1:7004 to 127.0.0.1:7001 Adding replica 127.0.0.1:7005 to 127.0.0.1:7002 M: 562665aca5f7db25bbd81e7b971c2f4c9aa65f96 127.0.0.1:7000 slots:0-5460 (5461 slots) master M: 6f93b84f7da7d6ea1b77bce1709432de472bbbe3 127.0.0.1:7001 slots:5461-10922 (5462 slots) master M: 6522a9ddec5b40cade141793351df44fe40a529f 127.0.0.1:7002 slots:10923-16383 (5461 slots) master S: f122aea6aa02be27f77454c0fcde9c65b891cd47 127.0.0.1:7003 replicates 562665aca5f7db25bbd81e7b971c2f4c9aa65f96 S: e5b5bbd378d8b80250dca6fb5db80e755f88c53c 127.0.0.1:7004 replicates 6f93b84f7da7d6ea1b77bce1709432de472bbbe3 S: 93ac017741ad4e65a4de3cb17ae295e6e5505b9a 127.0.0.1:7005 replicates 6522a9ddec5b40cade141793351df44fe40a529f Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 127.0.0.1:7000) M: 562665aca5f7db25bbd81e7b971c2f4c9aa65f96 127.0.0.1:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 6f93b84f7da7d6ea1b77bce1709432de472bbbe3 127.0.0.1:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: e5b5bbd378d8b80250dca6fb5db80e755f88c53c 127.0.0.1:7004 slots: (0 slots) slave replicates 6f93b84f7da7d6ea1b77bce1709432de472bbbe3 M: 6522a9ddec5b40cade141793351df44fe40a529f 127.0.0.1:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 93ac017741ad4e65a4de3cb17ae295e6e5505b9a 127.0.0.1:7005 slots: (0 slots) slave replicates 6522a9ddec5b40cade141793351df44fe40a529f S: f122aea6aa02be27f77454c0fcde9c65b891cd47 127.0.0.1:7003 slots: (0 slots) slave replicates 562665aca5f7db25bbd81e7b971c2f4c9aa65f96 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.

# 自動生成的node.conf文件 [root@cpic-redis-76 7005]# cat nodes.conf 1c7cbff1069612078d63329e5841c7430757a2cd 127.0.0.1:7003@17003 slave f5cc2de688640de572d16d72c2f174de027609a6 0 1516672833000 4 connected dd09a06827e9bfeab3876f4464cd467f6d1279f0 127.0.0.1:7002@17002 master - 0 1516672833086 3 connected 10923-16383 5863f9ee11d3ee3beef466d76f48f221b48aa6b5 127.0.0.1:7004@17004 slave 3ed6531ec985e50e87ab8436d49f4c32ce82bdef 0 1516672832000 5 connected 3ed6531ec985e50e87ab8436d49f4c32ce82bdef 127.0.0.1:7001@17001 master - 0 1516672832000 2 connected 5461-10922 9013209020c2fe2322df8c420b1c251fe9c84955 127.0.0.1:7005@17005 myself,slave dd09a06827e9bfeab3876f4464cd467f6d1279f0 0 1516672831000 6 connected f5cc2de688640de572d16d72c2f174de027609a6 127.0.0.1:7000@17000 master - 0 1516672832885 1 connected 0-5460 vars currentEpoch 6 lastVoteEpoch 0

# 啟動的log日志
[root@cpic-redis-76 7005]# cat start.log nohup: 忽略輸入 5627:C 22 Jan 21:02:58.212 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 5627:C 22 Jan 21:02:58.212 # Redis version=4.0.6, bits=64, commit=00000000, modified=0, pid=5627, just started 5627:C 22 Jan 21:02:58.212 # Configuration loaded 5627:M 22 Jan 21:02:58.214 * Increased maximum number of open files to 10032 (it was originally set to 1024). 5627:M 22 Jan 21:02:58.215 * Node configuration loaded, I'm 9013209020c2fe2322df8c420b1c251fe9c84955 5627:M 22 Jan 21:02:58.216 * Running mode=cluster, port=7005. 5627:M 22 Jan 21:02:58.216 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 5627:M 22 Jan 21:02:58.216 # Server initialized 5627:M 22 Jan 21:02:58.216 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 5627:M 22 Jan 21:02:58.216 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 5627:M 22 Jan 21:02:58.216 * Ready to accept connections 5627:M 23 Jan 10:00:27.212 # configEpoch set to 6 via CLUSTER SET-CONFIG-EPOCH 5627:M 23 Jan 10:00:27.286 # IP address for this node updated to 127.0.0.1 5627:S 23 Jan 10:00:31.238 * Before turning into a slave, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer. 5627:S 23 Jan 10:00:31.239 # Cluster state changed: ok 5627:S 23 Jan 10:00:31.981 * Connecting to MASTER 127.0.0.1:7002 5627:S 23 Jan 10:00:31.981 * MASTER <-> SLAVE sync started 5627:S 23 Jan 10:00:31.981 * Non blocking connect for SYNC fired the event. 5627:S 23 Jan 10:00:31.981 * Master replied to PING, replication can continue... 5627:S 23 Jan 10:00:31.982 * Trying a partial resynchronization (request 033127a1144abc3683a83a6b7257c2b56d2bd0b8:1). 5627:S 23 Jan 10:00:31.983 * Full resync from master: e5e51569d1f49762952e6cc05e1f13ddff68719e:0 5627:S 23 Jan 10:00:31.983 * Discarding previously cached master state. 5627:S 23 Jan 10:00:32.084 * MASTER <-> SLAVE sync: receiving 175 bytes from master 5627:S 23 Jan 10:00:32.084 * MASTER <-> SLAVE sync: Flushing old data 5627:S 23 Jan 10:00:32.084 * MASTER <-> SLAVE sync: Loading DB in memory 5627:S 23 Jan 10:00:32.084 * MASTER <-> SLAVE sync: Finished with success 5627:S 23 Jan 10:00:32.085 * Background append only file rewriting started by pid 10164 5627:S 23 Jan 10:00:32.108 * AOF rewrite child asks to stop sending diffs. 10164:C 23 Jan 10:00:32.108 * Parent agreed to stop sending diffs. Finalizing AOF... 10164:C 23 Jan 10:00:32.108 * Concatenating 0.00 MB of AOF diff received from parent. 10164:C 23 Jan 10:00:32.108 * SYNC append only file rewrite performed 10164:C 23 Jan 10:00:32.109 * AOF rewrite: 0 MB of memory used by copy-on-write 5627:S 23 Jan 10:00:32.181 * Background AOF rewrite terminated with success 5627:S 23 Jan 10:00:32.181 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB) 5627:S 23 Jan 10:00:32.181 * Background AOF rewrite finished successfully

---

BUG

• BUG1

[root@cpic-redis-76 src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 >>> Creating cluster [ERR] Node 127.0.0.1:7005 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0. 解決辦法: 刪掉原有數據dump.rdb/appendonly.aof/nodes.conf文件。

• BUG2

ruby安裝一些列困難。

詳見Redis Cluster 運維環境安裝記錄。

---

Reference

見Redis Cluster 運維環境安裝記錄