Redis主從自動failover

Redis主從架構持久化存在一個問題，即前次測試的結論，持久化需要配置在主實例上才能跨越實例保證數據不丟失，這樣以來主實例在持久化數據到硬 盤的過程中，勢必會造成磁盤的I/O等待，經過實際測試，這個持久化寫硬盤的過程給應用程序帶來的影響無法忍受；因而在大多數場景下，會考慮把持久化配置 在從實例上，當主實例宕機后，通過手動或者自動的方式將從實例提升為主實例，繼續提供服務！當主實例恢復后，先從原從實例上同步數據，同步完成后再恢復到 原始的主從狀態！要實現這種的要求，需要有keepalive的配合，一方面keepalive提供了VIP，可以避免修改應用程序連接，同時redis 實例的配置文件監聽部分也需要修改為全網監聽；另一方面keepalive定時調度腳本來監控主從實例的狀態，根據具體情況進行切換！本文將重點介紹下使 用keepalive實現redis主從自動failover！

環境介紹
操作系統版本均為：rhel5.4 64bit
redis版本：2.6.4
redis實例端口均為：6379
redis實例密碼均為：123
VIP：192.168.1.120
主實例為server11(192.168.1.112)
從實例為server12(192.168.1.113，開啟快照持久化)

一：安裝keepalive軟件,server11安裝完成后直接scp至server12上即可

    [root@server11 ~]# wget http://keepalived.org/software/keepalived-1.1.19.tar.gz  
    [root@server11 ~]# tar -zxvf ../tarbag/keepalived-1.1.19.tar.gz  
    [root@server11 ~]# cd keepalived-1.1.19/  
    [root@server11 ~]# ./configure --prefix=/usr/local/keepalived && make && make install

二：配置主節點server11配置文件

    [root@server11 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf  
    ! Configuration File for keepalived  
     
    global_defs {  
     router_id LVS_DEVEL  
    }  
     
    vrrp_script Monitor_redis {  
     script "/usr/local/scripts/redis_monitor.sh"  
     interval 2   
     weight 2    
    }  
     
    vrrp_instance VI_1{   
     state MASTER  
     interface eth0  
     virtual_router_id 51  
     mcast_src_ip 192.168.1.112  
     priority  100  
     advert_int 1  
     authentication {  
     auth_type PASS  
     auth_pass password_123  
    }  
     track_script {  
     Monitor_redis  
    }  
     virtual_ipaddress {  
     192.168.1.120  
     }  
     notify_fault  /usr/local/scripts/redis_fault.sh    
     notify_stop   /usr/local/scripts/redis_stop.sh    
     
    } 

三：配置從節點server12配置文件

    [root@server12 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf  
    ! Configuration File for keepalived  
     
    global_defs {  
     router_id LVS_DEVEL  
    }  
     
    vrrp_script Monitor_redis {  
     script "/usr/local/scripts/redis_monitor.sh"  
     interval 2   
     weight 2    
    }  
     
    vrrp_instance VI_1{   
     state BACKUP   
     interface eth0  
     virtual_router_id 51  
     mcast_src_ip 192.168.1.113  
     priority  99  
     advert_int 1  
     authentication {  
     auth_type PASS  
     auth_pass password_123  
    }  
     track_script {  
     Monitor_redis  
    }  
     virtual_ipaddress {  
     192.168.1.120  
     }  
     notify_master /usr/local/scripts/redis_master.sh    
     notify_backup /usr/local/scripts/redis_backup.sh    
     notify_fault  /usr/local/scripts/redis_fault.sh    
     notify_stop   /usr/local/scripts/redis_stop.sh    
     
    } 

四：准備相關的腳本，主從實例上都需要存在這些腳本，同時注意腳本需要由可執行權限

    [root@server11 ~]# cat /usr/local/scripts/redis_monitor.sh   
    #!/bin/bash    
    ALIVE=$(/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123 PING)  
     
    if [ "$ALIVE" == "PONG" ]; then   
        echo $ALIVE    
        exit 0    
        else   
        echo $ALIVE    
        killall -9 keepalived  
        service network restart  
        exit 1    
    fi   
     
    [root@server11 ~]# sh /usr/local/scripts/redis_monitor.sh   
    PONG  
     
    [root@server11 ~]# cat /usr/local/scripts/redis_master.sh    
    #!/bin/bash    
    REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123"   
    LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"   
     
    echo "[master]" >> $LOGFILE    
    date >> $LOGFILE    
    echo "Being master...." >> $LOGFILE 2>&1    
    echo "Run SLAVEOF cmd ..." >> $LOGFILE    
    $REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE  2>&1    
    sleep 10   
    echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE    
    $REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1   
     
     
    [root@server11 ~]# cat /usr/local/scripts/redis_backup.sh   
    #!/bin/bash    
    REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123"   
    LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"   
     
    echo "[backup]" >> $LOGFILE    
    date >> $LOGFILE    
    echo "Being slave...." >> $LOGFILE 2>&1    
    sleep 15    
    echo "Run SLAVEOF cmd ..." >> $LOGFILE    
    $REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE  2>&1   
     
    [root@server11 ~]# cat /usr/local/scripts/redis_stop.sh   
    #!/bin/bash  
    LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"   
    echo "[stop]" >> $LOGFILE  
    date >> $LOGFILE   
    [root@server11 ~]# cat /usr/local/scripts/redis_fault.sh 
    #!/bin/bash  
    LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log" 
    echo "[fault]" >> $LOGFILE  
    date >> $LOGFILE 

五：主從實例分別啟動keepalive進程，測試VIP是否正常(這里就要修改redis配置文件的監聽地址為0.0.0.0)

[root@server11 ~]# /usr/local/keepalived/sbin/keepalived -D -f  /usr/local/keepalived/etc/keepalived/keepalived.conf  
[root@server11 ~]# tail -f /var/log/messages  
Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Configuration is using : 5499 Bytes  
Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Using LinkWatch kernel netlink reflector...  
Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]  
Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP_Script(Monitor_redis) succeeded  
Dec 12 09:25:50 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Transition to MASTER STATE  
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Entering MASTER STATE  
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) setting protocol VIPs.  
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120  
Dec 12 09:25:51 server11 avahi-daemon[4519]: Registering new address record for 192.168.1.120 on eth0.  
Dec 12 09:25:51 server11 Keepalived_healthcheckers[7710]: Netlink reflector reports IP 192.168.1.120 added  
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: Netlink reflector reports IP 192.168.1.120 added  
Dec 12 09:25:56 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120  
 
[root@server11 ~]# ip a |grep 192  
    inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0  
    inet 192.168.1.120/32 scope global eth0  
 
[root@server12 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf  
[root@server12 ~]# tail -f /var/log/messages  
Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Configuration is using : 5595 Bytes  
Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering BACKUP STATE  
Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Using LinkWatch kernel netlink reflector...  
Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]  
Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Script(Monitor_redis) succeeded  
 
[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'  
# Replication  
role:master  
connected_slaves:1  
slave0:192.168.1.113,6379,online

六：主實例寫入測試數據，該腳本原則上會寫入25條測試數據，不過由於未優化redis默認並發數，會導致一些寫入請求失敗，最終功寫入231839條測 試數據，占內存總大小為25M左右，寫入過程中可以觀察主從實例的持久化文件變化情況，主實例的持久化文件維持在30k，從實例的則不斷的擴展！

    [root@server11 ~]# cat test.sh   
    #!/bin/bash    
    REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 -n 1 SET"   
    ID=1    
    while(($ID<50001))    
    do   
    INSTANCE_NAME="i-2-$ID-VM"   
    UUID=`cat /proc/sys/kernel/random/uuid`    
    PRIVATE_IP_ADDRESS=10.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`.`echo   
     
    "$RANDOM % 255 + 1" | bc`\    
    CREATED=`date "+%Y-%m-%d %H:%M:%S"`    
    $REDISCLI vm_instance:$ID:instance_name "$INSTANCE_NAME"   
    $REDISCLI vm_instance:$ID:uuid "$UUID"   
    $REDISCLI vm_instance:$ID:private_ip_address "$PRIVATE_IP_ADDRESS"   
    $REDISCLI vm_instance:$ID:created "$CREATED"   
    $REDISCLI vm_instance:$INSTANCE_NAME:id "$ID"   
    ID=$(($ID+1))    
    done   
     
    [root@server11 ~]# sh test.sh   
    [root@server11 redis2]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep   
     
    'used_memory_peak_human|db1:keys'  
    used_memory_peak_human:24.98M  
    db1:keys=231839,expires=0 

七：模擬主實例故障，觀察日志輸出，驗證從實例是否能成功接管VIP，同時將實例變成讀寫模式

    [root@server11 ~]# killall -9  redis-server  
    [root@server11 ~]# ip a |grep 192  
        inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0  
     
    [root@server11 ~]# ps -ef |grep redis  
    root     15886  6458  0 09:49 pts/0    00:00:00 grep redis  
    [root@server11 ~]# ps -ef |grep keep  
    root     16029  6458  0 09:49 pts/0    00:00:00 grep keep  
     
     
    [root@server12 ~]# tail -f /usr/local/redis2/var/keepalived-redis-state.log   
    [master]  
    Wed Dec 12 09:48:52 CST 2012  
    Being master....  
    Run SLAVEOF cmd ...  
    OK Already connected to specified master  
    Run SLAVEOF NO ONE cmd ...  
    OK  
     
    [root@server12 ~]# tail -f /var/log/messages  
    Dec 12 09:48:51 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Transition to MASTER STATE  
    Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering MASTER STATE  
    Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) setting protocol VIPs.  
    Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120  
    Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: Netlink reflector reports IP 192.168.1.120 added  
    Dec 12 09:48:52 server12 avahi-daemon[2921]: Registering new address record for 192.168.1.120 on eth0.  
    Dec 12 09:48:52 server12 Keepalived_healthcheckers[3106]: Netlink reflector reports IP 192.168.1.120 added  
    Dec 12 09:48:57 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120  
     
    [root@server12 ~]# ip a |grep 192  
        inet 192.168.1.113/24 brd 192.168.1.255 scope global eth0  
        inet 192.168.1.120/32 scope global eth0  
     
    [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'  
    # Replication  
    role:master  
    connected_slaves:0 

    [root@server12 ~]# sh test.sh  
    [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep   
     
    'used_memory_peak_human|db1:keys'  
    used_memory_peak_human:26.78M  
    db1:keys=249925,expires=0 

九:主實例角色的恢復過程，使用shell腳本自動恢復

    [root@server11 ~]# ssh-keygen   
    [root@server11 ~]# cd .ssh/  
    [root@server11 .ssh]# ssh-copy-id -i id_rsa.pub root@192.168.1.113  
    [root@server11 ~]# cat /usr/local/scripts/recover_mastart.sh   
    #!/bin/sh  
    ALIVE=$(/usr/local/redis2/bin/redis-cli -h 192.168.1.113 -p 6379 -a 123 PING)  
    MDB=/usr/local/redis2/master_dump.rdb  
    SDB=/usr/local/redis2/slave_dump.rdb  
     
    if [ "$ALIVE" == "PONG" ]; then   
        echo $ALIVE    
        scp root@192.168.1.113:$SDB  $MDB  
        else   
        echo $ALIVE   
        exit 1    
    fi   
     
    /usr/local/redis2/bin/redis-server /usr/local/redis2/etc/redis.conf  
    /usr/local/keepalived/sbin/keepalived -D -f    
     
    /usr/local/keepalived/etc/keepalived/keepalived.conf  
     
    [root@server11 ~]# chmod +x  /usr/local/scripts/recover_mastart.sh   
    [root@server11 ~]# sh /usr/local/scripts/recover_mastart.sh  

十：驗證數據完整性和主從角色恢復情況

    [root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep 'used_memory_peak_human|db1:keys'  
    used_memory_peak_human:26.78M  
    db1:keys=249925,expires=0 
     
    [root@server11 ~]#  /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'  
    # Replication  
    role:master  
    connected_slaves:1  
    slave0:192.168.1.113,6379,online  
     
    [root@server12 ~]#  /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 info |grep -A 3 'Replication'  
    # Replication  
    role:slave  
    master_host:192.168.1.112  
    master_port:6379  
     
    [root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep 'used_memory_peak_human|db1:keys'  
    used_memory_peak_human:26.78M  
    db1:keys=249925,expires=0 
     
    主實例keepalive日志：  
    [root@server11 ~]# tail -f /var/log/messages  
    Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]  
    Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Script(Monitor_redis) succeeded  
    Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Transition to MASTER STATE  
    Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Received higher prio advert  
    Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Entering BACKUP STATE  
    Dec 12 10:08:15 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) forcing a new MASTER election  
    Dec 12 10:08:16 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Transition to MASTER STATE  
    Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Entering MASTER STATE  
    Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) setting protocol VIPs.  
    Dec 12 10:08:17 server11 Keepalived_healthcheckers[20230]: Netlink reflector reports IP 192.168.1.120 added  
    Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120  
    Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: Netlink reflector reports IP 192.168.1.120 added  
    Dec 12 10:08:17 server11 avahi-daemon[4519]: Registering new address record for 192.168.1.120 on eth0.  
     
    [root@server11 ~]# ip a |grep 192  
        inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0  
        inet 192.168.1.120/32 scope global eth0  
     
    從實例keepalive日志：  
    [root@server12 ~]# tail -f /var/log/messages  
    Dec 12 09:56:01 server12 last message repeated 4 times  
    Dec 12 10:08:13 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Received lower prio advert, forcing new election  
    Dec 12 10:08:13 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120  
    Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Received higher prio advert  
    Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering BACKUP STATE  
    Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) removing protocol VIPs.  
    Dec 12 10:08:15 server12 Keepalived_healthcheckers[3106]: Netlink reflector reports IP 192.168.1.120 removed  
    Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: Netlink reflector reports IP 192.168.1.120 removed  
    Dec 12 10:08:15 server12 avahi-daemon[2921]: Withdrawing address record for 192.168.1.120 on eth0.  
     
    從實例角色轉換日志：  
    [root@server12 ~]# tail -f /usr/local/redis2/var/keepalived-redis-state.log   
    [backup]  
    Wed Dec 12 10:08:15 CST 2012  
    Being slave....  
    Run SLAVEOF cmd ...  
    OK