Redis 搭建一主二從三哨兵高可用集群

1、單個redis服務搭建請參考：redis服務搭建

2、在/usr/local下創建目錄redis-cluster,並在redis-cluster下創建 6379、6380、6381目錄以及data、temp目錄

# cd /usr/local
# mkdir redis-cluster

--其他文件創建類似，此處不一一寫出

3、復制安裝后的redis的配置文件（我的在/etc/目錄下）的redis.conf 和 sentinel.conf文件到 6379、6380、6381目錄中.

# cp /etc/redis.conf /usr/local/redis-cluster/6379
# cp /etc/redis-sentinel.conf /usr/local/redis-cluster/6379

--6380和6379類似，此處不一一寫出

4、主從配置，修改 redis.conf文件 

主redis 6379目錄:

protected-mode yes
port 6379
daemonize yes
pidfile /var/run/redis_6379.pid
logfile /var/log/redis/redis_6379.log
dir /usr/local/redis-cluster/6379/data
slave-read-only yes
requirepass foo(設置訪問登錄密碼)

#從服務設置了密碼需要加上 
masterauth foo 

從slave1 6380:

protected-mode yes
port 6380
daemonize yes
pidfile /var/run/redis_6380.pid
logfile /var/log/redis/redis_6380.log
dir /usr/local/redis-cluster/6380/data 
slaveof 127.0.0.1 6379 
#若主服務設置了密碼需要加上 
masterauth foo 
#從服務密碼設置 
requirepass foo

從slave2 6381:

protected-mode yes
port 6381
daemonize yes
pidfile /var/run/redis_6381.pid
logfile /var/log/redis/redis_6381.log
dir /usr/local/redis-cluster/6381/data 
slaveof 127.0.0.1 6379 
#若主服務設置了密碼需要加上，在設置哨兵時主從之間連接需要
masterauth foo 
#從服務密碼設置 
requirepass foo

master既可以讀,也可以寫,而 從服務器是只可以讀,不可寫的.

5、哨兵配置.

主redis 6379 sentinel.conf

protected-mode no
port 26379
dir "/usr/local/redis-cluster/6379/temp"
sentinel monitor redis1 127.0.0.1 6379 2
sentinel down-after-milliseconds redis1 10000
sentinel failover-timeout redis1 60000

其中redis1可自定義

從slave1 6380 centinel.conf

protected-mode no
port 26380
dir "/usr/local/redis-cluster/6380/temp"
sentinel monitor redis1 127.0.0.1 6379 2
sentinel down-after-milliseconds redis1 10000
sentinel failover-timeout redis1 60000

從slave2 7003 sentinel.conf

protected-mode no
port 26381
dir "/home/redis/redis-cluster/6381/temp"
sentinel monitor redis1 127.0.0.1 6381 2
sentinel down-after-milliseconds redis1 10000
sentinel failover-timeout redis1 60000

哨兵配置完成.

6、啟動

啟動redis
分別到6379、6380、6381的目錄下執行啟動命令:

redis-server ./redis.conf 

啟動哨兵

redis-server ./sentinel.conf 

7、查看主從信息:

127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=127.0.0.1,port=6380,state=online,offset=420,lag=1
slave1:ip=127.0.0.1,port=6381,state=online,offset=420,lag=1
master_repl_offset:434
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:433

 

127.0.0.1:6380> info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:168
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

 

127.0.0.1:6381> info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:9
master_sync_in_progress:0
slave_repl_offset:406
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

 

8、驗證

127.0.0.1:6379> set test 123
OK

 

127.0.0.1:6380> get test
"123"
127.0.0.1:6380> set hh 6380
(error) READONLY You can't write against a read only slave.

 

127.0.0.1:6381> get test
"123"
127.0.0.1:6381> set xx 6381
(error) READONLY You can't write against a read only slave.

 

此時,我們可以把主服務或者從服務停掉進行測試來看看效果

停掉主服務6379:

[root@VM_0_15_centos 6379]# ps -ef |grep redis
root     10690 30720  0 18:24 pts/4    00:00:00 tailf -n 200 /var/log/redis/sentinel.log
root     15650     1  0 19:01 ?        00:00:01 redis-sentinel *:26379 [sentinel]
root     16404     1  0 19:06 ?        00:00:00 redis-sentinel *:26381 [sentinel]
root     16565     1  0 19:07 ?        00:00:00 redis-sentinel *:26380 [sentinel]
root     17248 28933  0 19:12 pts/0    00:00:00 grep --color=auto redis
root     31060     1  0 16:55 ?        00:00:06 redis-server *:6379
root     31488     1  0 16:58 ?        00:00:06 redis-server *:6380
root     31563 29043  0 16:58 pts/1    00:00:00 redis-cli -c -p 6380 -a foo
root     31952     1  0 17:01 ?        00:00:06 redis-server *:6381
root     32057 29125  0 17:02 pts/2    00:00:00 redis-cli -c -p 6381 -a foo
[root@VM_0_15_centos 6379]# kill 31060

sentinel日志：

15650:X 18 Dec 19:13:44.834 # +sdown master mymaster 127.0.0.1 6379
16565:X 18 Dec 19:13:44.852 # +sdown master mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:44.861 # +sdown master mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:44.920 # +odown master mymaster 127.0.0.1 6379 #quorum 3/2
16404:X 18 Dec 19:13:44.920 # +new-epoch 1
16404:X 18 Dec 19:13:44.920 # +try-failover master mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:44.927 # +vote-for-leader 6d5a34396cd5912cbfe1134a70cd3e14338ebf24 1
15650:X 18 Dec 19:13:44.934 # +new-epoch 1
16565:X 18 Dec 19:13:44.934 # +new-epoch 1
16565:X 18 Dec 19:13:44.940 # +vote-for-leader 6d5a34396cd5912cbfe1134a70cd3e14338ebf24 1
15650:X 18 Dec 19:13:44.940 # +vote-for-leader 6d5a34396cd5912cbfe1134a70cd3e14338ebf24 1
16404:X 18 Dec 19:13:44.940 # 69c00be66f0461192b2db901ece6282e00b6462c voted for 6d5a34396cd5912cbfe1134a70cd3e14338ebf24 1
16404:X 18 Dec 19:13:44.940 # f51e8307952eba4264cc9089adf3c716e658609f voted for 6d5a34396cd5912cbfe1134a70cd3e14338ebf24 1
16565:X 18 Dec 19:13:44.942 # +odown master mymaster 127.0.0.1 6379 #quorum 2/2
16565:X 18 Dec 19:13:44.942 # Next failover delay: I will not start a failover before Tue Dec 18 19:14:05 2018
16404:X 18 Dec 19:13:45.003 # +elected-leader master mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:45.003 # +failover-state-select-slave master mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:45.075 # +selected-slave slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:45.075 * +failover-state-send-slaveof-noone slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:45.166 * +failover-state-wait-promotion slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6379
15650:X 18 Dec 19:13:45.900 # +odown master mymaster 127.0.0.1 6379 #quorum 3/2
15650:X 18 Dec 19:13:45.900 # Next failover delay: I will not start a failover before Tue Dec 18 19:14:05 2018
16404:X 18 Dec 19:13:45.960 # +promoted-slave slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:45.960 # +failover-state-reconf-slaves master mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:46.011 * +slave-reconf-sent slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379
16565:X 18 Dec 19:13:46.012 # +config-update-from sentinel 6d5a34396cd5912cbfe1134a70cd3e14338ebf24 127.0.0.1 26381 @ mymaster 127.0.0.1 6379
16565:X 18 Dec 19:13:46.012 # +switch-master mymaster 127.0.0.1 6379 127.0.0.1 6381
16565:X 18 Dec 19:13:46.012 * +slave slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6381
16565:X 18 Dec 19:13:46.012 * +slave slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6381
15650:X 18 Dec 19:13:46.013 # +config-update-from sentinel 6d5a34396cd5912cbfe1134a70cd3e14338ebf24 127.0.0.1 26381 @ mymaster 127.0.0.1 6379
15650:X 18 Dec 19:13:46.013 # +switch-master mymaster 127.0.0.1 6379 127.0.0.1 6381
15650:X 18 Dec 19:13:46.013 * +slave slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6381
15650:X 18 Dec 19:13:46.013 * +slave slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6381
16404:X 18 Dec 19:13:47.002 * +slave-reconf-inprog slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:47.002 * +slave-reconf-done slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:47.067 # -odown master mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:47.067 # +failover-end master mymaster 127.0.0.1 6379
16404:X 18 Dec 19:13:47.067 # +switch-master mymaster 127.0.0.1 6379 127.0.0.1 6381
16404:X 18 Dec 19:13:47.067 * +slave slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6381
16404:X 18 Dec 19:13:47.067 * +slave slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6381
16565:X 18 Dec 19:14:16.015 # +sdown slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6381
15650:X 18 Dec 19:14:16.017 # +sdown slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6381
16404:X 18 Dec 19:14:17.154 # +sdown slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6381

查看replication:

127.0.0.1:6380> info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6381
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:72129
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

127.0.0.1:6381> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=6380,state=online,offset=78422,lag=0
master_repl_offset:78422
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:78421

此時從服務6381成為主服務

重啟6379服務：

[root@VM_0_15_centos 6379]# redis-server ./redis.conf 

sentinel日志：

16404:X 18 Dec 19:22:33.181 * +convert-to-slave slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6381

查看replication:

127.0.0.1:6379> info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6381
master_link_status:up
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:1
master_link_down_since_seconds:1545132401
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

 再次停掉從服務器6379可以測試從服務器宕機時的效果

9、java集成

引入依賴：

 <!-- https://mvnrepository.com/artifact/redis.clients/jedis -->
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.9.0</version>
        </dependency>

具體測試見：java集成測試參考

10、注意事項

1. 只有Sentinel 集群中大多數服務器認定master主觀下線時master才會被認定為客觀下線，才可以進行故障遷移，也就是說，即使不管我們在sentinel monitor中設置的數是多少，就算是滿足了該值，只要達不到大多數，就不會發生故障遷移。
2. 官方建議sentinel至少部署三台，且分布在不同機器。這里主要考慮到sentinel的可用性，假如我們只部署了兩台sentinel，且quorum設置為1，也可以實現自動故障遷移，但假如其中一台sentinel掛了，就永遠不會觸發自動故障遷移，因為永遠達不到大多數sentinel認定master主觀下線了。
3. sentinel monitor配置中的master IP盡量不要寫127.0.0.1或localhost，因為客戶端，如jedis獲取master是根據這個獲取的，若這樣配置，jedis獲取的ip則是127.0.0.1，這樣就可能導致程序連接不上master
4. 當sentinel 啟動后會自動的修改sentinel.conf文件，如已發現的master的slave信息，和集群中其它sentinel 的信息等,這樣即使重啟sentinel也能保持原來的狀態。注意，當集群服務器調整時，如更換sentinel的機器，或者新配置一個sentinel，請不要直接復制原來運行過得sentinel配置文件，因為其里面自動生成了以上說的那些信息，應該復制一個新的配置文件或者把自動生成的信息給刪掉。
5. 當發生故障遷移的時候，master的變更記錄與slave更換master的修改會自動同步到redis的配置文件，這樣即使重啟redis也能保持變更后的狀態。