Redis Cluster是一種服務器sharding分片技術,關於Redis的集群方案應該怎么做,請參考我的另一篇博客http://www.cnblogs.com/xckk/p/6134655.html
本文主要介紹Redis Cluster集群的工作原理,詳細講解了Redis Cluster集群如何搭建與配置。
一、redis安裝
redis官網下載后是源碼包,需要make安裝。
1、解壓redis-3.2.5.tar.gz
2、cd redis-3.2.5
3、make
4、cd src/
5、make install
安裝完成,src包下面新增redis-server、redis-cli等這些可執行腳本,表示安裝完成。
二、啟動實例集群配置
搭建Redis Cluster,建立6個redis實例,3主3從,端口號分別為7000-7005
創建以6個端口命名的子目錄,之后我們將在6個子目錄上都開啟Redis實例
cd redis-3.2.5 mkdir cluster-test cd cluster-test mkdir 7000 7001 7002 7003 7004 7005
Redis實例配置
在7000到7005每一個目錄中都創建redis.conf文件,配置項如下:
cd redis-3.2.5 mkdir cluster-test cd cluster-test mkdir 7000 7001 7002 7003 7004 7005
各配置描述如下:
配置項redis.conf |
示例 |
備注 |
port | 7000 | 端口號 |
cluster-enabled | yes | yes,表示支持Redis Cluster |
cluster-config-file | nodes-7000.conf | Redis Cluster記錄的啟動信息文件,文件由cluster自動生成,不需要用戶編輯 |
cluster-node-timeout | 5000 | 毫秒單位,失效時間 |
appendonly | yes | 支持appendonly持久化方式 |
protected-mode | no | 默認是打開的,此處關閉。因為jedis訪問時,如果是保護模式,影響訪問 |
bind | 192.168.121.130 | 機器IP,此處最好寫IP,不要寫localhost或127.0.0.1,可能影響jedis客戶端訪問 |
Redis實例啟動
將第一步安裝生成的redis-server拷貝到cluster-test目錄,分別后台啟動6個redis實例
nohup ./redis-server 7000/redis.conf & nohup ./redis-server 7001/redis.conf & nohup ./redis-server 7002/redis.conf & nohup ./redis-server 7003/redis.conf & nohup ./redis-server 7004/redis.conf & nohup ./redis-server 7005/redis.conf &
啟動完后,cluster-test目錄下生成了nodes-7000.conf等啟動文件信息。
查看進程啟動成功:[root@centos2 cluster-test]# ps -ef|grep redis
三、創建集群
6個redis實例已經創建,它們目前是相互獨立的,沒有關聯。接下來是創建Redis Cluster集群,將6個實例組成一個集群。
redis-trib命令創建集群
src目錄下make生成的redis-trib命令,可用於Redis Cluster集群創建的一些腳本啟動,它是一個ruby程序。因此我們需要配置ruby環境
安裝ruby
yum install ruby
yum install ruby-irb
具體ruby的安裝可參考:http://jingyan.baidu.com/article/b7001fe173fe9a0e7382dd57.html
安裝Gem
Gem是一個管理Ruby庫和程序的標准包,它通過RubyGem(如 http://rubygems.org/ )源來查找、安裝、升級和卸載軟件包,非常的便捷。
配置了ruby環境后,還需要安裝redis整個包才能夠運行redis-trib。安裝過程中,需要先安裝gem庫,不然會出現gem command not found錯誤。
[root@centos2 src]# gem install redis -bash: gem: command not found
安裝gem過程中,可能出現以下問題
yum install rubygems ERROR: Could not find a valid gem 'redis' (>= 0) in any repository
問題原因:無法連接gem服務器,原因是該gem服務器被牆了。按如下步驟手動下載安裝。
wget https://rubygems.global.ssl.fastly.net/gems/redis-3.2.1.gem gem install -l ./redis-3.2.1.gem
安裝完后,src包下就會生成redis-trib.rb文件
啟動命令
./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005
啟動后,各Redis實例之間分別會新增一個端口用於通信,通信端口是Redis實例端口+1W,分別為17000,17001,17002,17003,17004,17005
命令的解釋如下:
1、給定的命令將會被ruby程序翻譯為create,這表示我們想要創建一個新的集群。
2、選項--replicas 1意思是為每一個master節點創建一個slave節點。
3、其他參數表示redis實例的地址及端口,以空格為間隔。
啟動日志分析
出現[OK] All 16384 slots covered.表示啟動成功。從啟動日志可以看出,master是7000,7001,7002,對應的slave是7003,7004,7005
[root@centos1 src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 Adding replica 127.0.0.1:7003 to 127.0.0.1:7000 Adding replica 127.0.0.1:7004 to 127.0.0.1:7001 Adding replica 127.0.0.1:7005 to 127.0.0.1:7002 M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1:7000 slots:0-5460 (5461 slots) master M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1:7001 slots:5461-10922 (5462 slots) master M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1:7002 slots:10923-16383 (5461 slots) master S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1:7003 replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45 S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1:7004 replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1:7005 replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 127.0.0.1:7000) M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1:7005 slots: (0 slots) slave replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251 S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1:7004 slots: (0 slots) slave replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1:7003 slots: (0 slots) slave replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45 M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
集群啟動狀態檢測
我們可以通過./redis-trib.rb check 192.168.121.130:7000來檢測Redis Cluster的啟動狀態。
[root@centos2 src]# ./redis-trib.rb check 192.168.121.130:7000 >>> Performing Cluster Check (using node 192.168.121.130:7000) M: dca47b274799927fd15aa4e6312b94752418f0c0 192.168.121.130:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: fe1082655e832bd2afb7d4bfabfec3ce2354868b 192.168.121.130:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 7fe93723c48b7ea33bbf21f12e97d18bdd0221ea 192.168.121.130:7003 slots: (0 slots) slave replicates dca47b274799927fd15aa4e6312b94752418f0c0 M: 59c926f7bb1e24e293d379507742c52b6dd08cc2 192.168.121.130:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: a554424a9969af2b55f4e11f9eedf1bc3bce9000 192.168.121.130:7004 slots: (0 slots) slave replicates 59c926f7bb1e24e293d379507742c52b6dd08cc2 S: 065fb213e9db64627c27af93500420351b80ac34 192.168.121.130:7005 slots: (0 slots) slave replicates fe1082655e832bd2afb7d4bfabfec3ce2354868b [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
登錄集群
通過命令./redis-cli -c -h 192.168.121.129 -p 7000用來登錄
重定向
從以下命令可以看出,我們登錄的是192.168.121.130機器的7000端口Redis實例,set foo 123,foo經過hash后落到了12182槽,12182槽存儲在192.168.121.130機器的7002端口實例,因此日志顯示重定向到7002端口的實例。
[root@centos2 src]# ./redis-cli -c -h 192.168.121.130 -p 7000 192.168.121.130:7000> set foo 123 -> Redirected to slot [12182] located at 192.168.121.130:7002 OK 192.168.121.130:7002> get foo "123" 192.168.121.130:7002>
redis cluster的主從同步采用異步復制(提高性能),因此會存在部分寫內容丟失的情況
四、創建集群-通過create-cluster腳本
通過redis-trib.rb命令創建集群涉及命令較多,稍顯復雜。好在redis提供了create-cluster腳本utils/create-cluster/create-cluster來創建集群,方便很多。
create-cluster命令介紹
從create-cluster源碼看,其實現原理也是利用了redis-trib.rb命令。該腳本默認創建3個master節點,3個salve節點。
1、create-cluster start
創建6個redis實例,默認端口號是30001-30006,如果想改端口號,在create-cluster里面更改PORT值即可。
如果想要端口號是7000-7005,設置PORT=6999即可
2、create-cluster create
創建集群,3個master,3個slave
3、create-cluster stop
關閉Redis Cluster集群和Redis實例
4、create-cluster clean
清除所在Redis實例的data, logs, config文件
使用方法
cd /home/redis-3.2/utils/create-cluster --進入目錄
./create-cluster clean ---先清除Redis實例的文件
./create-cluster start ---啟動Redis實例
./create-cluster create ---創建Redis Cluster集群
啟動完后,使用redis-cli命令登錄,即可正常訪問。
[root@centos1 create-cluster]# redis-cli -c -p 7000 127.0.0.1:7000> set foo hello -> Redirected to slot [12182] located at 127.0.0.1:7002 OK 127.0.0.1:7002> set hello world -> Redirected to slot [866] located at 127.0.0.1:7000 OK 127.0.0.1:7000> get foo -> Redirected to slot [12182] located at 127.0.0.1:7002 "hello" 127.0.0.1:7002> get hello -> Redirected to slot [866] located at 127.0.0.1:7000 "world" 127.0.0.1:7000>
五、Jedis測試程序
Jedis是Redis官方首選的 Java 客戶端開發包。我們通過Junit進行單元測試
maven添加jedis和junit依賴包
<dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> <!-- https://mvnrepository.com/artifact/redis.clients/jedis --> <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> <version>2.8.1</version> </dependency> </dependencies>
java代碼
package com.hk.test.helloWorld; import java.util.HashSet; import java.util.List; import java.util.Set; import org.apache.commons.pool2.impl.GenericObjectPoolConfig; import org.junit.Before; import org.junit.Test; import redis.clients.jedis.HostAndPort; import redis.clients.jedis.JedisCluster; public class TestJedis { JedisCluster jc = null; @Before public void before() { Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>(); // Jedis Cluster will attempt to discover cluster nodes automatically jedisClusterNodes.add(new HostAndPort("centos2", 7000)); GenericObjectPoolConfig config; jc = new JedisCluster(jedisClusterNodes); } @Test public void testFoo() { for (int i = 0; i <= 3000; i++) { String key = "foo" + i; String value = String.valueOf(i); jc.set(key, value); } } @Test public void test_incr() { String key = "page_view"; jc.del(key); jc.incr(key); String result = jc.get(key); System.out.println(result); } @Test public void test_setAndGetStringVal() { String key = "foo"; String value = "bar"; jc.set(key, value); String result = jc.get(key); System.out.println(result); } @Test public void test_setAndGetStringVal_and_set_expire() throws InterruptedException { String key = "hello"; String value = "world"; int seconds = 3; jc.setex(key, seconds, value); String result = jc.get(key); System.out.println(result); Thread.sleep(seconds * 1000); result = jc.get(key); System.out.println(result); } @Test public void test_setAndGetHashVal() { String key = "website"; String field = "google"; String value = "google.com"; jc.del(key); jc.hset(key, field, value); String result = jc.hget(key, field); System.out.println(result); } @Test public void test_setAndGetListVal() { String key = "mylist"; jc.del(key); String[] vals = { "a", "b", "c" }; jc.rpush(key, vals); List<String> result = jc.lrange(key, 0, -1); System.out.println(result); } @Test public void test_setAndGetSetVal() { String key = "language"; jc.del(key); String[] members = { "java", "ruby", "python" }; jc.sadd(key, members); Set<String> result = jc.smembers(key); System.out.println(result); } }
六、常見問題
一、Could not connect to Redis at 127.0.0.1:30001 Connection refused
原因:create-cluster腳本默認設定的端口從30001開始,由於我設置的redis端口是7000-7005,當時啟動cluster時用的redis-trib,因此端口號不同,
./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005
解決方法:
1、create-cluster里面將PORT由30001改為6999。
2、關閉Redis實例進程
可以通過netstat -apn|grep redis或者ps -ef|grep redis查找進程號,kill -9 {進程號}進行關閉。
3、create-cluster stop
4、create-cluster clean 清除data,log,conf文件
5、create-cluster start
6、create-cluster create
二、redis.clients.jedis.exceptions.JedisDataException: DENIED Redis is running in protected mode because protected mode is enabled
原因:Redis實例啟動時默認設置了保護模式
解決方法:需要在啟動時增加配置項 protected-mode no
1、如果是redis-trib.rb啟動,啟動配置文件內增加 protected-mode no
port 7000 cluster-enabled yes cluster-config-file nodes-7000.conf cluster-node-timeout 5000 appendonly yes protected-mode no bind 192.168.121.129
2、如果是create-cluster啟動,在create-cluster腳本里start分支中增加protected-mode no
if [ "$1" == "start" ] then while [ $((PORT < ENDPORT)) != "0" ]; do PORT=$((PORT+1)) echo "Starting $PORT" ../../src/redis-server --port $PORT --cluster-enabled yes --cluster-config-file nodes-${PORT}.conf --cluster-node-timeout $TIMEOUT --appendonly yes --bind 192.168.121.130 --protected-mode no --appendfilename appendonly-${PORT}.aof --dbfilename dump-${PORT}.rdb --logfile ${PORT}.log --daemonize yes done exit 0 fi
三、Jedis訪問時,redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException: Too many Cluster redirections?
若用JedisCluster操作集群報以上錯誤,就是有Redis節點通訊被拒絕,最好是將redis.conf文件中的bind屬性設置成redis節點所在服務器的物理ip地址,而不是localhost
1、更改node節點配置文件綁定地址
2、重啟redis節點
3、重新create cluster
./redis-trib.rb create --replicas 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003192.168.121.130:7004 192.168.121.130:7005
四、[ERR] Node 172.168.63.202:7001 is not empty. Either the nodealready knows other nodes (check with CLUSTER NODES) or contains some
解決方法:(一般1,2步即可解決)
1)、將需要新增的節點下aof、rdb等本地備份文件刪除;
2)、同時將新Node的集群配置文件刪除,即:刪除你redis.conf里面cluster-config-file所在的文件;
3)、再次添加新節點如果還是報錯,則登錄新Node,./redis-cli–h x –p對數據庫進行清除:
172.168.63.201:7001> flushdb #清空當前數據庫
4)、刪除后再執行第2,3步,再./create-cluster start即可
五、ERR Invalid node address specified
./redis-trib.rb create --replicas 1 centos2:7000 centos2:7001 centos2:7002 centos2:7003 centos2:7004 centos2:7005
報" ERR Invalid node address specified"
由於Redis-trib.rb 對域名或主機名支持不好,故在創建集群的時候要使用ip:port的方式
./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005
六、ERR Slot 4648 is already busy (Redis::CommandError)
這是由於之前創建集群沒有成功,需要
1、將nodes.conf和dir里面的文件全部刪除(注意不要刪除了redis.conf)
2、將redis實例關閉后重新啟動
3、再執行/redis-trib.rb create 。。。腳本
Sorry, the cluster configuration file nodes.conf is already used by a different Redis Cluster node. Please make sure that different nodes use different cluster configuration files.
七、[ERR] Node 127.0.0.1:7000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
解決方法:先stop和clean,再start和create
./create-cluster stop
./create-cluster clean
./create-cluster start
./create-cluster create
秀才坤坤出品
轉載請注明來源http://www.cnblogs.com/xckk/p/6144447.html