redis遷移第三篇(cluster forget)

 

1、刪除錯誤節點,帶有 fail,noaddr , 這種需要用 cluster forget

redis集群遷移之后，由於之前的誤操作，導致pod日志里面出現這樣的錯誤，出現一會好一會不好的情況，就是由於這個錯誤節點導致
[uts-node-0] [12:22:48:102] [ERROR] - org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:113) - Unable to decode data. channel: [id: 0x2770e76d, L:/172.17.246.215:45564 - R:/10.129.51.30:7733] message: $879
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 0 1646366975000 63 connected
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646366976611 63 connected 5461-10922
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641 0 1646366975000 62 connected
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 myself,master - 0 1646366969000 62 connected 10923-16383
27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr - 1646366063257 1646366063257 4 disconnected
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00 0 1646366976000 60 connected
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 master - 0 1646366977611 60 connected 0-5460

紅色的一條就是我之前在遷移中失誤操作造成的

  pod啟動后還會報錯信息為：

java.lang.IllegalArgumentException: Illegal character in scheme name at index 0: []r[]e[]d[]i[]s[]:[]/[]/[]:[]0[]

紅色的一條節點信息，是由於之前redis失誤命令為：

  [root@ht20 redis]#./redis-trib.rb del-node 10.129.51.30:7735 10.129.51.30
  
  正確命令應該為： 后面是 node_id
  [root@ht20 redis]#./redis-trib.rb del-node 10.129.51.30:7735 10.129.51.30 32ee19af1f6a534c4014b9c41d387666f049354b

  2、cluster forget命令操作,中間影響信息的部分處理掉了。

[root@ht20 data]# vi /data/redis_fpmai1/redis/data/nodes.conf   //查看master1的集群配置文件
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 slave   
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave  
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 myself,master -  5461-10922
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master -  10923-16383
27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr - 1646368054015 1646368054015 4 disconnected
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 master -  0-5460
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave 
vars currentEpoch 64 lastVoteEpoch 64 [root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7732 cluster forget 27f7e8dede5b68581486ce8cefdd656032baed70
OK
[root@ht20 data]# vi /data/redis_fpmai2/redis/data/nodes.conf   //查看master2的集群配置文件
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave  
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 master  0-5460
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave  
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 slave 
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 myself,master  5461-10922
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master  10923-16383
vars currentEpoch 64 lastVoteEpoch 64 ###就會發現沒有這一條了 27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr - 1646368054015 1646368054015 4 disconnected

1、注意cluster forget 只針對當前的主和當前的從起作用。

2、我依次處理掉其他master ，一共3個master

//對三個master都要進行處理,否則會有殘留信息。

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7733 cluster forget 27f7e8dede5b68581486ce8cefdd656032baed70
OK

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7732 cluster forget 27f7e8dede5b68581486ce8cefdd656032baed70 

OK

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7733 cluster forget 27f7e8dede5b68581486ce8cefdd656032baed70
OK

3、程序執行正常，但是會出現 No reachable node in cluster（這個意思很怪異，如果你剛接觸redis，會蒙的，怎么會是不能到達節點，集群中的）
  經過多方查找這個錯誤，最終確定是由於在程序中jredis沒有改配置中心的配置文件的ip信息，開發人員只修改了redisson即連接池的信息，
但是沒有修改jredis的ip端口信息導致jredis連接不上redis集群，報錯為： No reachable node in cluster 詳細說下排查過程
 我們配置中心采用config-->eureka-->應用-->redis集群

[root@master-web-38 src]# kubectl exec -ti us-node-0 -- /bin/bash  //進入master ping 外網也沒有問題

[root@uts-node-0:/# kubectl cp us-node-0:/app.jar /usr/local/src    //把程序包app.jar下載之后,改成app.zip，查看里面內容
里面lib下的java jar包進行破解，利用工具為 http://java-decompiler.github.io/#jd-gui-download 
破解lib包里面的redission-redisson-3.6.5.jar和spring-data-redis-2.0.6.RELEASE.jar后跟蹤到提示信息。 

說明如下
1、程序端采用了redission連接池和spring支持的jredis客戶端
2、redisson-3.6.5.jar //java客戶端連接池,檢查沒有問題。
3、spring-data-redis-2.0.6.RELEASE.jar  //客戶端的jredis支持。
 



最終跟蹤pod日志報錯，發現提示set錯誤，跟蹤連接池一直到jredis顯示報錯

public Jedis getConnection() {
List<JedisPool> pools = this.cache.getShuffledNodesPool();
for (JedisPool pool : pools) {
Jedis jedis = null;
try {
jedis = pool.getResource();
if (jedis == null)
continue;
String result = jedis.ping();
if (result.equalsIgnoreCase("pong"))
return jedis;
jedis.close();
} catch (JedisException ex) {
if (jedis != null)
jedis.close();
}
}
 throw new JedisNoReachableClusterNodeException("No reachable node in cluster"); 
 //查到這里基本問題就確定了，明顯是連接jredis連接不上，從而確定問題來自 jredis，訪問集群問題.
}

//這里jredis用了集群的ping命令

登錄集群服務器客戶端，執行ping,演示下jredis的機制。

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7733
10.129.51.30:7733> ping
PONG

附日志錯誤

org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception is

 redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
     
    at redis.clients.jedis.BinaryJedisCluster.set(BinaryJedisCluster.java:77)
    at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.set(JedisClusterStringCommands.java:113)
    ... 80 more
org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception 

is redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnection(JedisSlotBasedConnectionHandler.java:57)
 
    at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.set(JedisClusterStringCommands.java:113)
    ... 80 more
org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception is 
redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67)
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)