1.主從復制高可用的問題
主從復制高可用的作用
1.為master提供備份，當master宕機時，slave有完整的備份數據
2.對master實現分流，實現讀寫分離
但是主從架構有一個問題
1.如果master宕機，故障轉移需要手動完成或者由別的工具來完成，從slave中選擇一個slave做為新的master
寫能力和存儲能力受限
只能在一個節點是寫入數據
所有數據只能保存在一個節點上
上圖模擬了主從復制架構中一主兩從情況下，master宕機，則slave從master同步數據也斷開，此時client向master寫入數據會失敗，讀寫分離時讀取數據正常，但不能更新數據
master出現故障之后，手動進行故障轉移步驟
1.選擇一個slave，執行slave no one命令使之成為一個master
2.對其余的slave執行slaveof new master命令，將這些slave指定為新的master的slave
3.client會對新的master進行寫入數據，從slave中讀取原來的數據
上面的操作過程需要手動完成，或者編寫腳本，由腳本來執行這個過程，但是這個過程是有問題的：
怎么判斷一個Redis節點是有問題的，怎么通知client對新master進行寫入操作
怎么保證一個完整的事務實現過程
上面的過程就可以使用Redis Sentinel來實現
2.Redis Sentinel架構說明
Redis Sentinel的功能：對Redis節點進行監控，故障判斷，故障轉移，故障通知
對於Redis的主從架構，Redis Sentinel會運行幾個sentinel進程
sentinel進程不操作數據，而是對Redis故障進行判斷和轉移
同時多個sentinel運行，即使一個sentinel進程運行異常，還有別的sentinel繼續運行，可以保證對故障節點判斷的准確性，同時保證Redis的高可用
對於redis-cli來說，Redis cli不會再記錄Redis的IP和端口，而是從sentinel獲取Redis信息，然后進行連接Redis節點，進行數據寫入和讀取操作
多個Redis Sentinel對所有的master和slave進行監控，會實時記錄master和slave的地址信息
Redis Sentinel故障轉移步驟：
1.當某個master發生故障，多個sentinel會監控到這個異常，這些sentinel會按照一定規則從多個slave中選中一個做為新的master,並通知別的slave從新的master中同步數據
2.當某個slave轉換為新的master，sentinel會記錄新的master的地址信息和slave的地址信息，通知Redis cli
3.Redis cli接收到新的master和slave的信息，就會向新的master寫入數據，從slave中讀取數據
4.等到原來的master重啟之后，會變成新的master的slave,並從新的master同步數據
在上面的步驟里，sentinel實現了Redis的故障自動發現，自動轉移和自動通知
說明：一套Redis sentinel集合可以通知master-name做為標識同時監控多套主從架構
3.Redis Sentinel安裝配置
3.1 環境說明
實驗在兩台虛擬機上完成，IP地址分別為：192.168.81.100和192.168.81.101
在兩台虛擬機上運行4個redis-server,其中
192.168.81.100的6379端口為master節點
192.168.81.100的6380端口為slave節點
192.168.81.101的6379端口和6380端口為slave節點
在192.168.81.101的26379，26380，26381端口開啟三個sentinel進行監控
3.2 在192.168.81.100虛擬機上配置主從節點
[root@localhost ~]# cd /etc/ # 進入/etc目錄
[root@localhost ~]# systemctl stop redis # 關閉系統中運行的redis
[root@localhost etc]# cp redis.conf redis_6379.conf # 復制redis配置文件，以端口區分，方便后面進行配置
[root@localhost etc]# cp redis.conf redis_6380.conf # 復制redis配置文件，以端口區分，方便后面進行配置
[root@localhost etc]# vi redis_6379.conf # 編輯redis-server配置文件，修改下面幾行
bind 0.0.0.0 # 修改bing選項，才能從系統外連接redis
protected-mode yes # 開啟保存模式
port 6379 # 指定redis運行的端口
daemonize yes # 以守護進程啟動redis
pidfile "/var/run/redis_6379.pid" # 指定redis運行時pid保存路徑
logfile "/var/log/redis/redis_6379.log" # 指定redis運行時日志保存路徑
dir /var/lib/redis_6379 # 指定redis運行時數據文件保存路徑
[root@localhost etc]# vi redis_6380.conf # 修改redis-server,修改下面幾行
bind 0.0.0.0
port 6380 # 指定redis運行的端口
daemonize yes
pidfile "/var/run/redis_6380.pid"
logfile "/var/log/redis/redis_6380.log"
dir /var/lib/redis_6380
slaveof 192.168.81.100 6379 # 指定redis-server為192.168.81.100:6379的slave
[root@localhost etc]# redis-server /etc/redis_6379.conf # 指定配置文件運行redis-server
[root@localhost etc]# redis-server /etc/redis_6380.conf # 指定配置文件運行redis-server
[root@localhost etc]# ps aux | grep redis-server # 查看redis-server是否運行
root 2548 0.3 1.7 155192 17720 ? Ssl 23:14 0:00 redis-server 0.0.0.0:6379
root 2562 1.3 1.7 155192 17596 ? Ssl 23:15 0:00 redis-server 0.0.0.0:6380
root 2567 0.0 0.0 112648 960 pts/3 R+ 23:15 0:00 grep --color=auto redis-server
[root@localhost etc]# redis-cli -p 6380 info replication # 進入6380端口運行redis客戶端，並執行'info replication'命令
# Replication
role:slave # 角色為slave
master_host:192.168.81.100 # master為192.168.81.100
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:1919
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
3.3 在192.168.81.101虛擬機上配置從節點
[root@mysql ~]# cd /etc/ # 操作同192.168.81.100相同
[root@mysql ~]# systemctl stop redis
[root@mysql etc]# cp redis.conf redis_6379.conf
[root@mysql etc]# cp redis.conf redis_6380.conf
[root@mysql etc]# vi redis_6379.conf
bind 0.0.0.0
protected-mode yes
port 6379
daemonize yes
pidfile "/var/run/redis_6379.pid"
logfile "/var/log/redis/redis_6379.log"
dir /var/lib/redis_6379
slaveof 192.168.81.100 6379 # 指定redis-server為192.168.81.100:6379的slave
[root@mysql etc]# vi redis_6380.conf
bind 0.0.0.0
port 6380
daemonize yes
pidfile "/var/run/redis_6380.pid"
logfile "/var/log/redis/redis_6380.log"
dir /var/lib/redis_6380
slaveof 192.168.81.100 6379 # 指定redis-server為192.168.81.100:6379的slave
[root@mysql etc]# redis-server /etc/redis_6379.conf # 指定配置文件運行redis-server
[root@mysql etc]# redis-server /etc/redis_6380.conf # 指定配置文件運行redis-server
[root@mysql ~]# ps aux | grep redis-server # 查看redis-server是否運行
root 2178 0.2 0.8 155204 17728 ? Ssl 15:10 0:02 redis-server 0.0.0.0:6379
root 2184 0.2 0.8 155204 17724 ? Ssl 15:10 0:02 redis-server 0.0.0.0:6380
root 2411 0.0 0.0 112664 972 pts/2 R+ 15:29 0:00 grep --color=auto redis-server
[root@mysql ~]# redis-cli -p 6379 info replication
# Replication
role:slave # 角色為slave
master_host:192.168.81.100 # master為192.168.81.100
master_port:6379
master_link_status:up
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:1961
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
[root@mysql ~]# redis-cli -p 6380 info replication
# Replication
role:slave # 角色為slave
master_host:192.168.81.100 # master為192.168.81.100
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:1975
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
3.4 在192.168.81.101虛擬機上配置並運行sentinel
[root@mysql etc]# cp redis-sentinel.conf sentinel_26379.conf # 復制sentinel配置文件，方便區分
[root@mysql etc]# cp redis-sentinel.conf sentinel_26380.conf # 復制sentinel配置文件，方便區分
[root@mysql etc]# cp redis-sentinel.conf sentinel_26381.conf # 復制sentinel配置文件，方便區分
[root@mysql etc]# vi sentinel_26379.conf # 修改sentinel配置文件,修改下面幾行
daemonize yes # 以守護進程方式啟動
port 26379 # 指定端口
protected-mode no # 關閉保護模式
sentinel monitor mymaster 192.168.81.100 6379 2 # 設置sentinel監控信息
logfile /var/log/redis/sentinel_26379.log # 設置日志文件保存路徑
[root@mysql etc]# vi sentinel_26380.conf
daemonize yes
port 26380
protected-mode no
sentinel monitor mymaster 192.168.81.100 6379 2
logfile /var/log/redis/sentinel_26380.log
[root@mysql etc]# vi sentinel_26381.conf
protected-mode no
port 26381
daemonize yes
sentinel monitor mymaster 192.168.81.100 6379 2
logfile /var/log/redis/sentinel_26381.log
[root@mysql etc]# redis-sentinel /etc/sentinel_26379.conf # 指定配置文件，啟動Redis Sentinel
[root@mysql etc]# redis-sentinel /etc/sentinel_26380.conf # 指定配置文件，啟動Redis Sentinel
[root@mysql etc]# redis-sentinel /etc/sentinel_26381.conf # 指定配置文件，啟動Redis Sentinel
[root@mysql etc]# ps aux | grep sentinel # 查看Redis Sentinel是否運行
root 2709 0.9 0.2 142916 5464 ? Ssl 15:49 0:00 redis-sentinel *:26379 [sentinel]
root 2713 1.1 0.2 142916 5472 ? Ssl 15:49 0:00 redis-sentinel *:26380 [sentinel]
root 2717 2.0 0.2 142916 5476 ? Rsl 15:49 0:00 redis-sentinel *:26381 [sentinel]
root 2721 0.0 0.0 112664 964 pts/2 R+ 15:49 0:00 grep --color=auto sentinel
[root@mysql ~]# redis-cli -p 26379
127.0.0.1:26379> ping # 執行ping操作
PONG
127.0.0.1:26379> info sentinel # 查看所有sentinel的信息
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.81.100:6379,slaves=3,sentinels=3 # 被監控的Redis主從架構命名為mymaster，被監控Redis節點的master為192.168.81.100L6379,有三個slave,同時有3個sentinel運行
127.0.0.1:26379> exit
[root@mysql ~]# grep -v '^#' /etc/sentinel_26379.conf | grep -v '^$' #查看sentinel_26379配置文件，去除注釋和空行，Redis Sentinel向配置文件中添加了幾行內容
port 26379 # sentinel運行的端口
dir "/tmp"
sentinel myid 9611958fc3e8b7c2be43385e44be88f87d725a77
sentinel monitor mymaster 192.168.81.100 6379 2 # sentinel監控的Redis節點名為mymaster,master地址為192.168.81.100:6379,quorem設置為2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
logfile "/var/log/redis/sentinel_26379.log"
daemonize yes
sentinel known-slave mymaster 192.168.81.101 6379 # sentinel探測到的slave
sentinel known-slave mymaster 192.168.81.100 6380 # sentinel探測到的slave
sentinel known-slave mymaster 192.168.81.101 6380 # sentinel探測到的slave
sentinel known-sentinel mymaster 192.168.81.101 26380 17ca0cb82becb58bd24e5a87ee3b6e8e9a49caf1 # Redis Sentinel深測到的別的運行的sentinel
sentinel known-sentinel mymaster 127.0.0.1 26381 fb9342f3007e2abff165f5c33de1d48cf089f062 # Redis Sentinel深測到的別的運行的sentinel
sentinel current-epoch 0
[root@mysql ~]# grep -v '^#' /etc/sentinel_26380.conf | grep -v '^$'
port 26380
dir "/tmp"
sentinel myid 17ca0cb82becb58bd24e5a87ee3b6e8e9a49caf1
sentinel monitor mymaster 192.168.81.100 6379 2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
logfile "/var/log/redis/sentinel_26380.log"
daemonize yes
sentinel known-slave mymaster 192.168.81.101 6379
sentinel known-slave mymaster 192.168.81.101 6380
sentinel known-slave mymaster 192.168.81.100 6380
sentinel known-sentinel mymaster 127.0.0.1 26381 fb9342f3007e2abff165f5c33de1d48cf089f062
sentinel known-sentinel mymaster 192.168.81.101 26379 9611958fc3e8b7c2be43385e44be88f87d725a77
sentinel current-epoch 0
[root@mysql ~]# grep -v '^#' /etc/sentinel_26381.conf | grep -v '^$'
port 26381
daemonize yes
dir "/tmp"
sentinel myid fb9342f3007e2abff165f5c33de1d48cf089f062
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
logfile "/var/log/redis/sentinel_26381.log"
sentinel known-sentinel mymaster 127.0.0.1 26380 17ca0cb82becb58bd24e5a87ee3b6e8e9a49caf1
sentinel known-sentinel mymaster 192.168.81.101 26379 9611958fc3e8b7c2be43385e44be88f87d725a77
sentinel current-epoch 0
至此，3個sentinel已經正常運行了
3.5 python客戶端持續通過sentinel向Redis寫入數據，讀取數據
import random
import time
from redis.sentinel import Sentinel
sentinel = Sentinel([
('192.168.81.101',26379),
('192.168.81.101',26380),
('192.168.81.101',26381),
],socket_timeout=0.1) # 傳入Redis Sentinel集合
while True:
try:
master = sentinel.discover_master('mymaster')
print('current master IP:',master) # 打印當前master的IP地址和端口
val = random.randint(0,10000) # 獲取10000以內隨機整數
key = 'k%d' % val
m = sentinel.master_for('mymaster', socket_timeout=0.5)
m.set(key,'val%d' % val) # 通過sentinel向master節點寫入數據
v = m.get(key) # 通過sentinel讀取數據
print('{0} value is {1}'.format(key,v))
time.sleep(1)
except Exception as e:
print("get no val:",e)
運行上面的代碼：
current master IP: ('192.168.81.100', 6379)
k6081 value is b'val6081'
current master IP: ('192.168.81.100', 6379)
k1778 value is b'val1778'
current master IP: ('192.168.81.100', 6379)
k4927 value is b'val4927'
current master IP: ('192.168.81.100', 6379)
k4074 value is b'val4074'
current master IP: ('192.168.81.100', 6379)
k1138 value is b'val1138'
current master IP: ('192.168.81.100', 6379)
k862 value is b'val862'
current master IP: ('192.168.81.100', 6379)
k4854 value is b'val4854'
current master IP: ('192.168.81.100', 6379)
k9233 value is b'val9233'
current master IP: ('192.168.81.100', 6379)
k6844 value is b'val6844'
current master IP: ('192.168.81.100', 6379)
k8089 value is b'val8089'
3.6 在192.168.81.100虛擬機上模擬master故障
[root@localhost etc]# redis-cli -p 6379 # 連接6379端口連接redis-server
127.0.0.1:6379> info server # 執行命令查看當前redis-server的進程ID
# Server
redis_version:3.2.10
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:c8b45a0ec7dc67c6
redis_mode:standalone
os:Linux 3.10.0-514.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:2548 # 192.168.81.100:6379這個master節點的進程ID為2548
run_id:8ae71ba92660697d52bfb74b99fb15ee82a7cf84
tcp_port:6379
uptime_in_seconds:6552
uptime_in_days:0
hz:10
lru_clock:12896355
executable:/etc/redis-server
config_file:/etc/redis_6379.conf
127.0.0.1:6379> exit
[root@localhost etc]# kill -9 2548 # 在系統中kill掉2548這個進程
[root@localhost etc]# ps aux | grep redis-server # 查看redis-server運行的進程，2548已經被終止
root 2562 0.2 1.7 155192 17756 ? Ssl Oct15 0:19 redis-server 0.0.0.0:6380
root 2651 0.0 0.0 112648 960 pts/3 S+ 01:19 0:00 grep --color=auto redis-server
3.7 查看python代碼執行結果
current master IP: ('192.168.81.100', 6379)
1539597332.9189174 k6187 value is b'val6187'
current master IP: ('192.168.81.100', 6379)
1539597333.9236474 k1462 value is b'val1462'
current master IP: ('192.168.81.100', 6379)
get no val: Timeout connecting to server
current master IP: ('192.168.81.100', 6379)
get no val: Timeout connecting to server
current master IP: ('192.168.81.100', 6379)
get no val: Timeout connecting to server
...中間報錯信息省略
get no val: No master found for 'mymaster'
get no val: No master found for 'mymaster'
get no val: DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
get no val: No master found for 'mymaster'
...中間報錯信息省略
get no val: DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
current master IP: ('192.168.81.101', 6379)
1539597365.971147 k3568 value is b'val3568'
current master IP: ('192.168.81.101', 6379)
1539597366.974567 k7745 value is b'val7745'
current master IP: ('192.168.81.101', 6379)
1539597367.9783657 k6281 value is b'val6281'
current master IP: ('192.168.81.101', 6379)
可以看到初始master是192.168.81.100:6379，在系統命令提示符中kill掉6379端口的redis-server后，python程序從1539597333.9236474秒后一直拋出異常，直到1539597365.971147秒，python程序通過sentinel獲取到新的master為192.168.81.101:6379。
1539597365.971147減去1539597333.9236474恰好就是sentinel配置文件中sentinel down-after-milliseconds mymaster選項設定的30000毫秒，也就是30秒
3.8 查看Redis Sentinel的日志
在192.168.81.101虛擬機上查看/var/log/redis/sentinel_26379.log日志，下面的部分日志就是sentinel在切換master的記錄
4067:X 15 Oct 17:55:40.303 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:42.119 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:42.334 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:44.208 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:44.349 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:46.266 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:46.365 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:48.419 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:48.481 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:50.429 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:50.506 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:52.548 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:52.571 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:54.592 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:54.598 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:56.655 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:56.662 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:58.734 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:55:58.886 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:00.798 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:00.972 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:02.884 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:03.042 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:04.917 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:05.099 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:07.045 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:07.112 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:09.145 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:09.324 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:11.218 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:11.345 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:12.211 # +sdown master mymaster 192.168.81.100 6379
4067:X 15 Oct 17:56:12.345 # +new-epoch 8
4067:X 15 Oct 17:56:12.346 # +vote-for-leader fb9342f3007e2abff165f5c33de1d48cf089f062 8
4067:X 15 Oct 17:56:13.267 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:13.269 * +sentinel-address-switch master mymaster 192.168.81.100 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:13.289 # +odown master mymaster 192.168.81.100 6379 #quorum 2/2
4067:X 15 Oct 17:56:13.289 # Next failover delay: I will not start a failover before Mon Oct 15 18:02:13 2018
4067:X 15 Oct 17:56:13.440 # +config-update-from sentinel fb9342f3007e2abff165f5c33de1d48cf089f062 192.168.81.101 26381 @ mymaster 192.168.81.100 6379
4067:X 15 Oct 17:56:13.441 # +switch-master mymaster 192.168.81.100 6379 192.168.81.101 6379
4067:X 15 Oct 17:56:13.441 * +slave slave 192.168.81.101:6380 192.168.81.101 6380 @ mymaster 192.168.81.101 6379
4067:X 15 Oct 17:56:13.441 * +slave slave 192.168.81.100:6380 192.168.81.100 6380 @ mymaster 192.168.81.101 6379
4067:X 15 Oct 17:56:13.441 * +slave slave 192.168.81.100:6379 192.168.81.100 6379 @ mymaster 192.168.81.101 6379
4067:X 15 Oct 17:56:15.586 * +sentinel-address-switch master mymaster 192.168.81.101 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:15.604 * +sentinel-address-switch master mymaster 192.168.81.101 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:15.689 * +sentinel-address-switch master mymaster 192.168.81.101 6379 ip 127.0.0.1 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
4067:X 15 Oct 17:56:15.693 * +sentinel-address-switch master mymaster 192.168.81.101 6379 ip 192.168.81.101 port 26381 for fb9342f3007e2abff165f5c33de1d48cf089f062
從上面的日志可以看出，在17:56:13.441時間點sentinel把master從182.168.81.100:6379切換到192.168.81.101:6379
4.Redis Sentinel客戶端連接(python版)
Redis Sentinel的高可用指的是服務端的高可用，對於Redis服務端的master宕機，sentinel可以對故障實現自動發現，自動轉移，自動通知。這個過程客戶端是感知不到的
Redis高可用即依賴於服務端的高可用，又依賴於客戶端的高可用
通過分析Redis Sentinel的請求響應流程，可以知道客戶端實現高可用步驟：
1.客戶端遍歷sentinel節點集合，獲取一個可用的sentinel節點，同時獲取masterName
2.在可用的sentinel節點上執行sentinel的API，獲取master的地址和端口
3.在sentinel內部，sentinel會按照一定頻率在master或者slave上執行info命令，獲取對應節點的信息
4.客戶端獲取到master的地址和端口，會執行role命令或者role replication命令，對master進行驗證
5.當master出現故障，sentinel按照算法從slave中選出一個做為新的master，同時把其余的slave做為新的master的slave
6.sentinel維護一個頻道，sentinel在這個頻道中發布消息，指出新master的IP和端口
7.客戶端訂閱這個頻道，接收消息后就知道新master的IP和端口，向新master連接進行數據處理
原理圖如下
python客戶端接入Redis Sentinel需要兩個參數：sentinel地址集合，masterName
需要注意的是Redis節點的配置文件中的
protected-mode必須設置為yes，否則連接會失敗
from redis.sentinel import Sentinel
sentinel = Sentinel([
('192.168.81.101',26379),
('192.168.81.101',26380),
('192.168.81.101',26381),
],socket_timeout=0.1)
master = sentinel.discover_master('mymaster')
print(master) # ('192.168.81.100', 6379)
slave = sentinel.discover_slaves('mymaster')
print(slave) # [('192.168.81.100', 6380), ('192.168.81.101', 6379), ('192.168.81.101', 6380)]
5.Redis Sentinel實現原理
5.1 Redis Sentinel內部的三個定時任務
Redis Sentinel內部有三個定時任務來對redid節點進行故障判斷和轉移
- 1.每10秒每個sentinel對master和slave執行
info命令，以發現slave節點和確認主從關系
sentinel在master節點執行info replication命令，從命令執行結果中解析出slave節點
- 2.每2秒每個sentinel通過master節點的channel交換信息(發布訂閱)
master節點上有一個發布訂閱的channel頻道：__sentinel__:hello，用於所有sentinel之間進行信息交換
一個sentinel發布消息，消息包含當前sentinel節點的信息，對其他sentinel節點的判斷以及當前sentinel對master節點和slave節點的一些判斷
其他sentinel都可以接收到這條消息
新加入sentinel節點時，sentinel節點之間可以相互感知，以達到信息交互的功能
- 3.每1秒每個sentinel對其他sentinel節點和Redis節點執行ping操作
每個sentinel都可以知道其他sentinel節點，當監控的master發生故障時，方便進行判斷和新master的挑選，這個定時任務是master進行故障判定的依據
5.2 主觀下線和客觀下線
主觀下線：每個sentinel節點對Redis節點失敗的'偏見'
在redis-sentinel配置文件中，有下面這種配置
sentinel monitor <master-name> <ip> <redis-port> <quorum>
sentinel down-after-milliseconds <master-name> <timeout>
一個sentinel集合可以同時監控多個master,slave的節點
sentinel對多個master，slave節點進行區分的標識就是master-name,ip和port是master節點的IP地址和端口,quorum是master客觀下線之后sentinel進行判斷的節點數
sentinel對master進行主觀下線判斷的時間，單們為毫秒
每個sentinel每秒對master和slave執行ping操作，當sentinel對master或slave在timeout定義的毫秒時間內沒有回復，則sentinel會認為這個節點已經被主觀下線了
在前面的例子中對sentinel的配置是
sentinel monitor mymaster 192.168.81.100 6379 2
sentinel down-after-milliseconds mymaster 30000
解釋：
sentinel集合監控名為mymaster的master,slave節點
被監控的master節點的IP地址是192.168.81.100，端口為6379，
sentinel會在`__sentinel__:hello`頻道中交流對master節點的看法，如果sentinel節點都對master節點ping失敗'達成共識'，sentinel個數超過quorum的個數，sentinel集合則會認為master節點客觀下線
當兩個sentinel對master節點執行ping操作，在30000毫秒(30秒)時間內沒有得到回復，則認為節點已經被主觀下線
quorum建議設置為：(sentinel節點數 / 2) + 1，可以根據應用場景進行設定
5.3 sentinel領導者選舉
要點：
只需要一個sentinel節點就可以完成故障轉移
通過`sentinel is-master-down-by-addr`命令來完成sentinel交換對master節點的失敗判定和新master的選舉
完成sentinel領導者選舉步驟：
1.每個做主觀下線的sentinel節點向其他sentinel節點發送命令，要求將自己設置為領導者
2.收到命令的sentinel節點如果沒有同意同意其他sentinel節點發送的命令，那么將同意該請求，否則拒絕
3.如果該sentinel節點發現自己的票數已經超過sentinel集合半數且超過quorum，將成為領導者
4.如果此過程中有多個sentinel節點成為領導者，那么將等待一段時間重新進行選舉
5.4 故障轉移(由sentinel領導者節點完成)
故障轉移步驟：
1.從slave節點中選出一個合適的節點作為新的master節點
2.對選出的slave節點執行`slaveof no one`命令，使成為新的master節點
3.向剩余的slave節點發送命令，讓slave節點成為新master節點的slave節點，然后從新master節點同步數據
數據同步規則和parallel-syncs參數有關
如一個一主三從架構中，master故障，sentinel領導者從3個slave中選出一個作為新的master節點，剩余的兩個slave節點會成為新master節點的slave，從新master節點同步同步數據
master節點只需要生成一次RDB文件
如果parallel-syncs參數設置為1，則剩余兩個slave節點會按順序從新master節點拷貝數據，一個slave切點拷貝完成，另外一個slave才會從新master節點拷貝數據
如果parallel-syncs參數設置為2，則兩個slave節點會同時從master節點進行數據拷貝，這無疑會加入新master的開銷
4.sentinel領導者會把原來的master節點設置為slave節點，並保持對其'關注'，當原來的master節點恢復后，sentinel會使其去復制新master節點的數據
5.5 slave節點的選擇
slave節點選擇規則
1.選擇slave-priority(slave節點優先級)最高的slave節點，如果存在則返回，不存在則繼續
2.選擇復制偏移量(offset)最大的slave節點,offset最大說明對master的數據復制的最完整，如果存在則返回，不存在則繼續
3.選擇run_id最小的slave節點，run_id最小說明slave節點啟動最早
6.總結：
Redis Sentinel是Redis的高可用實現方案：故障發現，故障自動轉移，配置中心，客戶端通知
Redis Sentinel是Redis 2.8版本開始才正式生產可用，之前版本不可用於生產
盡可以在不同物理機上部署Redis Sentinel所有節點，但是最好一個局域網內
Redis Sentinel中sentinel節點個數應該大於等於3，且最好為奇數，可以保證判斷的公平
Redis Sentinel中的數據節點與普通數據節點沒有區別
客戶端初始化時連接的是Sentinel節點集合，不是具體的Redis節點，但是Sentinel只是配置中心不是代理
Redis Sentinel通過三個定時任務實現了Sentinel節點對於master，slave，其余sentinel節點的監控
Redis Sentinel在對節點做失敗判定時分為主觀下線和客觀下線
看懂Redis Sentinel故障轉移日志對於Redis Sentinel以及問題排查非常有幫助
Redis Sentinel實現讀寫分離高可用可以依賴Redis Sentinel節點的消息通知，獲取Redis數據節點的狀態變化