變更需求為:
1、調整主從關系,所有節點都調整到10.129.51.30機器上 2、停掉10.128.51.14上的所有redis,14機器關機 14機器下線遷移至新機房,這段時間將不能提供服務.
當前集群情況
1、一共是2台12個實例,6主6從,每台6個實例。 2、30上有3台master,沒有分配槽位,需變更為從節點。
3、10.128.51.14 放在A機房,10.129.51.30放在B機房。
4、目前經過上一次的調整,把0-16383槽位從10.128.51.14都遷移到了10.129.51.30的新偽集群中。
//查看集群主從節點信息。進行分析
[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7736 //輸出結果我進行了小范圍的調整
10.129.51.30:7736> cluster nodes
//6台從,每2台從同步一個master,黃色代表對應。
4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733@17733 slave d1c7d99e13a2d7317baf883dffa906470a606641 0 1646269992000 62 connected 07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641 0 1646269990000 62 connected fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731@17731 slave d46f032ea50763de8353fd530535412df6ffdc00 0 1646269993529 60 connected ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00 0 1646269992327 60 connected 41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732@17732 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 0 1646269990324 63 connected b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 0 1646269994330 63 connected
//閑置 27f7e8dede5b68581486ce8cefdd656032baed70 10.129.51.30:7734@17734 master - 0 1646269994826 4 connected 32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735@17735 master - 0 1646269989315 0 connected bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736@17736 myself,master - 0 1646269991000 6 connected //帶槽位的master d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 master - 0 1646269995327 60 connected 0-5460 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646269993324 63 connected 5461-10922 d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master - 0 1646269994000 62 connected 10923-16383
我們繼續分析,我把主要信息調整了一下,刪除掉了防礙分析的信息。只保留相互之間關系(黃色做為示例,標記了相互關系)
10.129.51.30:7736> cluster nodes 1、10.129.51.30:7731 主 0-5460, master標記 d46f032ea50763de8353fd530535412df6ffdc00 的兩個從節點 fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731 slave d46f032ea50763de8353fd530535412df6ffdc00 ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734 slave d46f032ea50763de8353fd530535412df6ffdc00 2、10.129.51.30:7732 主 5461-10922, master標記 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 的兩個從節點 41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 3、10.129.51.30:7733 主 10923-16383, master標記 d1c7d99e13a2d7317baf883dffa906470a606641 的兩個從節點 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733 slave d1c7d99e13a2d7317baf883dffa906470a606641 07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736 slave d1c7d99e13a2d7317baf883dffa906470a606641 //以上6台主機全部要關機 //master,這幾台不變 d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731 master 0-5460 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732 master 5461-10922 d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733 master 10923-16383 //需要等待調整為具有槽位主節點變成slave 27f7e8dede5b68581486ce8cefdd656032baed70 10.129.51.30:7734@17734 master - 32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735@17735 master - bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736@17736 myself,master
根據第二次整理,我再次整理出相互關系表格,方便我下面實際操作。
主節點ID 主節點地址 從節點ID 從節點地址 //主槽位 0-5460 d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731 fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731 d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731 ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734 //主槽位 5461-10922 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732 41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732 b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735 //主槽位 10923-16383 d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733 d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733 07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736
看到以上的信息,基本就已經很明確,redis集群中相互之間的關系,以及遷移動機和實際要達成的目標。
要達成目標為:
1、10.128.51.14:7731和10.128.51.14:7734 刪除 2、10.129.51.30:7734從集群中移除 (原來不帶槽位的master) 3、把10.129.51.30:7734節點,重新配置后,設置為當前集群中的10.129.51.30:7731主節點的從節點。
實際遷移、調整操作
一、設置新的主從關系(7734->7731)
1、刪除從節點,在任意一台服務器上執行下面命令:
//操作1.刪除從節點10.128.51.14:7734 [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7731 ff34a5fadf42078332445bcf41a2b59416fd92da >>> Removing node ff34a5fadf42078332445bcf41a2b59416fd92da from cluster 10.129.51.30:7731 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node. //我們到另一台14機器上查下,已經被殺掉,也沒有在cluster nodes里面出現. //操作2.刪除從節點 10.128.51.14:7731 [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7731 fb28e95eb37b8d827c0f48800c4d08e64c6fc335 >>> Removing node fb28e95eb37b8d827c0f48800c4d08e64c6fc335 from cluster 10.129.51.30:7731 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node. //我們看到10.128.51.14:7731從redis集群中被刪除,並不會殺死該實例 解釋:
a、刪掉 10.129.51.30:7731 的兩個從節點 10.128.51.14:7731 和 10.128.51.14:7734 b、解釋 10.129.51.30:7731 為主節點地址, fb28e95eb37b8d827c0f48800c4d08e64c6fc335 為 10.128.51.14:7731 從節點標記 c、解釋 10.129.51.30:7731 為主節點地址, ff34a5fadf42078332445bcf41a2b59416fd92da 為 10.128.51.14:7734 從節點標記 從節點斷開主從關系后並不會拋棄原有數據,只是無法再獲取主節點上的數據變化: 2、把10.129.51.30:7734從集群中移除 (准備把該節點作為10.129.51.30:7731的從節點) 語法: /redis-trib.rb del-node ip:端口號 節點id [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7734 d46f032ea50763de8353fd530535412df6ffdc00
[root@ht20 redis]# ps -ef | grep redis
root 40843 126564 0 12:34 pts/0 00:00:00 grep --color=auto redis
root 57063 1 0 Mar01 ? 00:03:53 /data/redis_fpmai1/redis/redis-server 10.129.51.30:7731 [cluster]
root 58394 1 0 Mar01 ? 00:03:25 /data/redis_fpmai2/redis/redis-server 10.129.51.30:7732 [cluster]
root 58881 1 0 Mar01 ? 00:07:27 /data/redis_fpmai3/redis/redis-server 10.129.51.30:7733 [cluster]
root 58893 1 0 Mar01 ? 00:03:03 /data/redis_fpmai4/redis/redis-server 10.129.51.30:7734 [cluster]
root 58939 1 0 Mar01 ? 00:02:54 /data/redis_fpmai6/redis/redis-server 10.129.51.30:7736 [cluster]
root 100693 1 0 Mar01 ? 00:02:45 /data/redis_fpmai5/redis/redis-server 10.129.51.30:7735 [cluster]
//我們看到10.128.51.14:7734 master從redis集群中被刪除,並不會殺死該實例,不過還是要有幾步處理。
//a、先停掉 10.129.51.30:7734 ,刪除data里面的內容,否則添加主從關系會提示不成功node not empty。
[root@ht20 redis]# kill -9 58893 //7734進程id
//b、因為以前已經加入過集群,刪除其中的信息
[root@ht20 redis]#rm -rf /data/redis_fpmai4/redis/data/*.*
//c、重新啟動7734實例
[root@ht20 redis]# /data/redis_fpmai4/redis/redis-server /data/redis_fpmai4/redis/redis.conf
3、前面的准備工作做完之后,就可以把10.129.51.30:7731 mater上添加一個slave從節點(需要重新建立主從關系)
語法 ./redis-trib.rb add-node --slave --master-id 主節點id 從節點地址 主節點地址
[root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7734 10.129.51.30:7731
>>> Adding node 10.129.51.30:7734 to cluster 10.129.51.30:7731
>>> Performing Cluster Check (using node 10.129.51.30:7731)
M: d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731
slots:0-5460 (5461 slots) master
0 additional replica(s)
M: d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733
slots:10923-16383 (5461 slots) master
2 additional replica(s)
S: 07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736
slots: (0 slots) slave
replicates d1c7d99e13a2d7317baf883dffa906470a606641
S: 41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732
slots: (0 slots) slave
replicates 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
M: 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732
slots:5461-10922 (5462 slots) master
2 additional replica(s)
M: 32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735
slots: (0 slots) master
0 additional replica(s)
M: bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736
slots: (0 slots) master
0 additional replica(s)
S: 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733
slots: (0 slots) slave
replicates d1c7d99e13a2d7317baf883dffa906470a606641
S: b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735
slots: (0 slots) slave
replicates 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.129.51.30:7734 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 10.129.51.30:7731.
[OK] New node added correctly.
//檢查加入情況,14的2個節點已經刪除,30的7734已變成slave
解釋:
1、d46f032ea50763de8353fd530535412df6ffdc00 master標識
2、10.129.51.30:7734 要加入的新的機器 (之前加入過,所以需要停掉該實例,刪除data目錄下的內容,再重新啟動該實例)
3、10.129.51.30:7731 是主節點,即有槽位的機器(當然也可以是其他任意主節點,有槽位的),
//檢查下集群中是不是已經達到我們的目的(紅色這一行就是我們的目標)。
[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7731
10.129.51.30:7731> cluster nodes
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master - 10923-16383
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 myself,master - 0-5460
07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641
27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr //這一條就是設置主從時候,失誤導致的
41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732@17732 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 5461-10922
32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735@17735 master - connected
bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736@17736 master - connected
4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733@17733 slave d1c7d99e13a2d7317baf883dffa906470a606641
b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
所以看到紅色的關鍵語句就4條。
回顧下主從的關系
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1、10.129.51.30:7731 主 0-5460, master標記 d46f032ea50763de8353fd530535412df6ffdc00 的兩個從節點 fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731 slave d46f032ea50763de8353fd530535412df6ffdc00 ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734 slave d46f032ea50763de8353fd530535412df6ffdc00 2、10.129.51.30:7732 主 5461-10922, master標記 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 的兩個從節點 41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 3、10.129.51.30:7733 主 10923-16383, master標記 d1c7d99e13a2d7317baf883dffa906470a606641 的兩個從節點 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733 slave d1c7d99e13a2d7317baf883dffa906470a606641 07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736 slave d1c7d99e13a2d7317baf883dffa906470a606641 //master d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 master - 0 1646269995327 60 connected 0-5460 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646269993324 63 connected 5461-10922 d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master - 0 1646269994000 62 connected 10923-16383 //等待調整為slave 27f7e8dede5b68581486ce8cefdd656032baed70 10.129.51.30:7734@17734 master - 0 1646269994826 4 connected 32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735@17735 master - 0 1646269989315 0 connected bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736@17736 myself,master - 0 1646269991000 6 connected 主節點ID 主節點地址 從節點ID 從節點地址 //主槽位 0-5460 d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731 fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731 d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731 ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734 //主槽位 5461-10922 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732 41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732 b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735 //主槽位 10923-16383 d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733 d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733 07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
二、設置新的主從關系(7735->7732)
二、 設置新的主從關系(7735->7732) 1、從redis集群中刪掉兩個從節點(10.128.51.14:7732 和10.128.51.14:7735) 在任意一台服務器上執行下面命令: //刪除10.129.51.30:7732的從節點10.128.51.14:7735和10.128.51.14:7732 [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7732 41767372c36cc268872e86d96660a32bf540624f [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7732 b9a55f739a65e6277e40cdbb806b5443c8aad66e 2、移除7735節點 [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7735 32ee19af1f6a534c4014b9c41d387666f049354b [root@ht20 redis]# kill -9 58894 [root@ht20 redis]#rm -rf /data/redis_fpmai5/redis/data/*.* [root@ht20 redis]# /data/redis_fpmai5/redis/redis-server /data/redis_fpmai5/redis/redis.conf 3、添加一個新的主從關系(重新建立主從關系) [root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7735 10.129.51.30:7732
三、 設置新的主從關系(7736->7733)
三、 設置新的主從關系(7736->7733) 1、從redis集群中刪掉兩個從節點(10.128.51.14:7733 和10.128.51.14:7736) 在任意一台服務器上執行下面命令: //刪除10.129.51.30:7732的從節點10.128.51.14:7733 和10.128.51.14:7736 [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7733 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7733 07c99eae8176eb0e985c5a9914f42900d18fb39b 2、移除7736節點 [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7736 bc5051aef1a756c274907d23198fbb932c384151 [root@ht20 redis]# kill -9 58894 [root@ht20 redis]#rm -rf /data/redis_fpmai5/redis/data/*.* [root@ht20 redis]# /data/redis_fpmai5/redis/redis-server /data/redis_fpmai5/redis/redis.conf 3、添加一個新的主從關系(重新建立主從關系) [root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7736 10.129.51.30:7733 注意:del-node只能刪除沒有分配slot的節點
執行相關過程就單獨放這里吧。 這樣可以和上面的操作分開,上面的二和三中的操作,看着基本很簡潔了,
//刪除7732節點
[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7732 41767372c36cc268872e86d96660a32bf540624f >>> Removing node 41767372c36cc268872e86d96660a32bf540624f from cluster 10.129.51.30:7732 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node.
//刪除7732從節點 [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7732 b9a55f739a65e6277e40cdbb806b5443c8aad66e >>> Removing node b9a55f739a65e6277e40cdbb806b5443c8aad66e from cluster 10.129.51.30:7732 >>> Sending CLUSTER FORGET messages to the cluster... >>> SHUTDOWN the node.
//刪除集群中的節點(之前是標記為了master,但是槽位是空的)
[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7735 32ee19af1f6a534c4014b9c41d387666f049354b
>>> Removing node 32ee19af1f6a534c4014b9c41d387666f049354b from cluster 10.129.51.30:7735
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
//刪除集群中的節點(之前標記也是master,槽位為空)
[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7736 bc5051aef1a756c274907d23198fbb932c384151
>>> Removing node bc5051aef1a756c274907d23198fbb932c384151 from cluster 10.129.51.30:7736
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
//為確保遷移過程中正確,檢查一下當前集群情況
[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7731
10.129.51.30:7731> cluster nodes
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733 master 10923-16383
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734 slave d46f032ea50763de8353fd530535412df6ffdc00
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731 myself,master 0-5460
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732 master 5461-10922
//刪除掉redis_fpmai5和redis_fpmai6的data目錄下的文件,重新啟動
[root@ht20 redis]# /data/redis_fpmai5/redis/redis-server /data/redis_fpmai5/redis/redis.conf
[root@ht20 redis]# /data/redis_fpmai6/redis/redis-server /data/redis_fpmai6/redis/redis.conf
[root@ht20 redis]# ps -ef | grep redis
root 42097 1 0 12:40 ? 00:00:07 /data/redis_fpmai4/redis/redis-server 10.129.51.30:7734 [cluster]
root 57063 1 0 Mar01 ? 00:04:03 /data/redis_fpmai1/redis/redis-server 10.129.51.30:7731 [cluster]
root 58394 1 0 Mar01 ? 00:03:35 /data/redis_fpmai2/redis/redis-server 10.129.51.30:7732 [cluster]
root 58881 1 0 Mar01 ? 00:07:40 /data/redis_fpmai3/redis/redis-server 10.129.51.30:7733 [cluster]
root 67736 1 0 14:35 ? 00:00:00 /data/redis_fpmai5/redis/redis-server 10.129.51.30:7735 [cluster]
root 67774 1 0 14:35 ? 00:00:00 /data/redis_fpmai6/redis/redis-server 10.129.51.30:7736 [cluster]
//執行中出現這樣錯誤,我已經刪除掉了data目錄下的內容了,可能操作那塊漏掉了,剛才操作有一步嚴謹。重新來一遍就不會出下面問題。
[root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7735 10.129.51.30:7732
>>> Adding node 10.129.51.30:7735 to cluster 10.129.51.30:7732
>>> Performing Cluster Check (using node 10.129.51.30:7732)
M: 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732
slots:5461-10922 (5462 slots) master
0 additional replica(s)
M: d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733
slots:10923-16383 (5461 slots) master
0 additional replica(s)
S: 421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734
slots: (0 slots) slave
replicates d46f032ea50763de8353fd530535412df6ffdc00
M: d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[ERR] Node 10.129.51.30:7735 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
//我還是先殺掉7735,刪除對應data目錄下的文件,再重新啟動7735實例,再次執行后正常
[root@ht20 redis]# /data/redis_fpmai5/redis/redis-server /data/redis_fpmai5/redis/redis.conf
[root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7735 10.129.51.30:7732
>>> Adding node 10.129.51.30:7735 to cluster 10.129.51.30:7732
>>> Performing Cluster Check (using node 10.129.51.30:7732)
M: 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732
slots:5461-10922 (5462 slots) master
0 additional replica(s)
M: d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733
slots:10923-16383 (5461 slots) master
0 additional replica(s)
S: 421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734
slots: (0 slots) slave
replicates d46f032ea50763de8353fd530535412df6ffdc00
M: d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.129.51.30:7735 to make it join the cluster.
Waiting for the cluster to join..
>>> Configure node as replica of 10.129.51.30:7732.
[OK] New node added correctly.
[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7731
10.129.51.30:7731> cluster nodes
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master - 0 1646289642962 62 connected 10923-16383
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00 0 1646289644000 60 connected
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 myself,master - 0 1646289642000 60 connected 0-5460
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646289644965 63 connected 5461-10922
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 0 1646289643963 63 connected
執行到這里,我讓開發人員測試了下目前redis的情況,暫時都一切正常。 但是沒有讓k8s pod重新部署.
//最后執行一下
[root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7736 10.129.51.30:7733
>>> Adding node 10.129.51.30:7736 to cluster 10.129.51.30:7733
>>> Performing Cluster Check (using node 10.129.51.30:7733)
M: d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733
slots:10923-16383 (5461 slots) master
0 additional replica(s)
S: f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735
slots: (0 slots) slave
replicates 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
M: 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734
slots: (0 slots) slave
replicates d46f032ea50763de8353fd530535412df6ffdc00
M: d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.129.51.30:7736 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 10.129.51.30:7733.
[OK] New node added correctly.
[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7731
10.129.51.30:7731> cluster nodes
//下面的關系,稍微整理了一下,如下:
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 myself,master - 0 1646290141000 60 connected 0-5460
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00 0 1646290140700 60 connected
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646290140000 63 connected 5461-10922
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 0 1646290141702 63 connected
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master - 0 1646290138698 62 connected 10923-16383
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641 0 1646290142703 62 connected
遷移到此就基本結束了。 這個階段也沒有出現任何問題。
為了穩妥,讓開發人員進行重新的部署。來看下pod日志打印情況和業務使用情況
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master 10923-16383
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 myself,master - 0 1646290592000 60 connected 0-5460
27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr - 1646282374602 1646282369596 4 disconnected
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646290592000 63 connected 5461-10922
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
日志打印正常,使用都正常。