1. 前言
負載均衡LB，高可用HA，這一小結主要講雙機熱備方案保證高可用。這里選擇Keepalived作為雙機熱備方案，下面就對具體的配置進行了解。
2. 下載Keepalived
wget http://www.keepalived.org/software/keepalived-1.4.0.tar.gz
文檔 http://www.keepalived.org/doc
參考 https://www.cnblogs.com/abclife/p/7909818.html
https://www.cnblogs.com/kevingrace/p/6138185.html
系統 Debian 8
1 ./configure --prefix=/opt/keepalive #這一步，可能要額外安裝一些依賴
1 make 2 make install 3 mkdir /etc/keepalived 4 cp ./etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf 5 cp ./sbin/keepalived /usr/sbin/ 6 vim /etc/init.d/keepalived 7 chmod a+x /etc/init.d/keepalived 8 service keepalived start
/etc/init.d/keepalived 加入如下內容
1 #!/bin/sh 2 # 3 # keepalived High Availability monitor built upon LVS and VRRP 4 # 5 # chkconfig: - 86 14 6 # description: Robust keepalive facility to the Linux Virtual Server project \ 7 # with multilayer TCP/IP stack checks. 8 9 ### BEGIN INIT INFO 10 # Provides: keepalived 11 # Required-Start: $local_fs $network $named $syslog 12 # Required-Stop: $local_fs $network $named $syslog 13 # Should-Start: smtpdaemon httpd 14 # Should-Stop: smtpdaemon httpd 15 # Default-Start: 16 # Default-Stop: 0 1 2 3 4 5 6 17 # Short-Description: High Availability monitor built upon LVS and VRRP 18 # Description: Robust keepalive facility to the Linux Virtual Server 19 # project with multilayer TCP/IP stack checks. 20 ### END INIT INFO 21 22 # Source function library. 23 . /etc/rc.d/init.d/functions 24 25 exec="/usr/sbin/keepalived" 26 prog="keepalived" 27 config="/etc/keepalived/keepalived.conf" 28 29 [ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog 30 31 lockfile=/var/lock/subsys/keepalived 32 33 start() { 34 [ -x $exec ] || exit 5 35 [ -e $config ] || exit 6 36 echo -n $"Starting $prog: " 37 daemon $exec $KEEPALIVED_OPTIONS 38 retval=$? 39 echo 40 [ $retval -eq 0 ] && touch $lockfile 41 return $retval 42 } 43 44 stop() { 45 echo -n $"Stopping $prog: " 46 killproc $prog 47 retval=$? 48 echo 49 [ $retval -eq 0 ] && rm -f $lockfile 50 return $retval 51 } 52 53 restart() { 54 stop 55 start 56 } 57 58 reload() { 59 echo -n $"Reloading $prog: " 60 killproc $prog -1 61 retval=$? 62 echo 63 return $retval 64 } 65 66 force_reload() { 67 restart 68 } 69 70 rh_status() { 71 status $prog 72 } 73 74 rh_status_q() { 75 rh_status &>/dev/null 76 } 77 78 79 case "$1" in 80 start) 81 rh_status_q && exit 0 82 $1 83 ;; 84 stop) 85 rh_status_q || exit 0 86 $1 87 ;; 88 restart) 89 $1 90 ;; 91 reload) 92 rh_status_q || exit 7 93 $1 94 ;; 95 force-reload) 96 force_reload 97 ;; 98 status) 99 rh_status 100 ;; 101 condrestart|try-restart) 102 rh_status_q || exit 0 103 restart 104 ;; 105 *) 106 echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}" 107 exit 2 108 esac 109 exit $?
3. 雙機熱備（主從模式）
修改配置文件 keepalived.conf
vim /etc/keeplived/keeplived.conf
1 global_defs { 2 notification_email { #指定Keepalived在發生事情的時候，發送郵件通知，每行一個地址 3 yyyyy@qq.com 4 } 5 notification_email_from yyyyy@qq.com #指定發件人 6 smtp_server 192.168.8.208 #發送email的smtp地址 7 smtp_connect_timeout 30 #超時時間 8 router_id nginx_dev #運行Keepalived的機器標識號，可以相同也可以不同 9 } 10 11 vrrp_instance nginx_dev { 12 state MASTER 13 interface eth1 14 #mcast_src_ip 172.16.23.203 15 virtual_router_id 100 16 priority 200 17 advert_int 5 18 authentication { 19 auth_type PASS 20 auth_pass 123456 21 } 22 virtual_ipaddress { 23 172.16.23.222 24 } 25 }
然后 service keepalived restart
然后就可以通過 172.16.23.222 訪問當前主機了。如果訪問不了的要判斷是否所處於同個網絡中，即可以通過路由訪問到的。這個Keepalived就是通過發廣播包這里的通知內網機器實現的。
同理在另外一台電腦上配置上面信息 把state MASTER 改為 state BACKUP
4. 雙機熱備（主主模式）
主從模式的話，由於一般情況下是不會出現宕機，所以往往會有一台機器浪費，這樣是對機器的浪費，所以現在雙機熱備主主模式是比較推薦的。所謂的主主模式，就是建立兩個實例，互為主從而已。
172.16.23.203 配置如下
1 global_defs { 2 notification_email { #指定Keepalived在發生事情的時候，發送郵件通知，每行一個地址 3 xxx@aa.com 4 } 5 notification_email_from xxx@aa.com #指定發件人 6 smtp_server 192.168.8.208 #發送email的smtp地址 7 smtp_connect_timeout 30 #超時時間 8 router_id nginx_dev_1 #運行Keepalived的機器標識號，可以相同也可以不同 9 router_id nginx_dev_2 10 } 11 12 vrrp_instance nginx_dev_1 { 13 state MASTER 14 interface eth1 15 mcast_src_ip 172.16.23.203 16 virtual_router_id 100 17 priority 100 18 advert_int 5 19 authentication { 20 auth_type PASS 21 auth_pass 123456 22 } 23 virtual_ipaddress { 24 172.16.23.222 25 } 26 } 27 28 vrrp_instance nginx_dev_2 { 29 state BACKUP 30 interface eth1 31 mcast_src_ip 172.16.23.203 32 virtual_router_id 101 33 priority 200 34 advert_int 5 35 authentication { 36 auth_type PASS 37 auth_pass 123456 38 } 39 virtual_ipaddress { 40 172.16.23.223 41 } 42 }
172.16.23.204 配置如下
1 global_defs { 2 notification_email { #指定Keepalived在發生事情的時候，發送郵件通知，每行一個地址 3 xxxx@aa.com 4 } 5 notification_email_from xxxx@aa.com #指定發件人 6 smtp_server 192.168.8.208 #發送email的smtp地址 7 smtp_connect_timeout 30 #超時時間 8 router_id nginx_dev_1 #運行Keepalived的機器標識號，可以相同也可以不同 9 router_id nginx_dev_2 10 } 11 12 vrrp_instance nginx_dev_2 { 13 state MASTER 14 interface eth1 15 mcast_src_ip 172.16.23.204 16 virtual_router_id 101 17 priority 100 18 advert_int 5 19 authentication { 20 auth_type PASS 21 auth_pass 123456 22 } 23 virtual_ipaddress { 24 172.16.23.223 25 } 26 } 27 28 vrrp_instance nginx_dev_1 { 29 state BACKUP 30 interface eth1 31 mcast_src_ip 172.16.23.204 32 virtual_router_id 100 33 priority 200 34 advert_int 5 35 authentication { 36 auth_type PASS 37 auth_pass 123456 38 } 39 virtual_ipaddress { 40 172.16.23.222 41 } 42 }
就是這兩份配置，基本是一樣的，互為主備，里面一個比較重要修改的是virtual_router_id
上面實現的效果是訪問172.16.23.222 時先轉到172.16.20.203主機上，訪問172.16.23.223時轉到172.16.20.204主機上。當其中一台主機宕機時，就會自動切換，切換到好的主機上，這個過程就幾秒的時間。
這里是需要兩個IP地址，需要客戶端進行負載選擇，這一步可以通過DNS進行分發處理。
1 root@debian-t6:/usr/local/nginx/html# ip addr 2 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 3 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 4 inet 127.0.0.1/8 scope host lo 5 valid_lft forever preferred_lft forever 6 2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default 7 link/ether fc:aa:14:9d:d9:5a brd ff:ff:ff:ff:ff:ff 8 inet 172.16.23.204/24 brd 172.16.23.255 scope global eth1 9 valid_lft forever preferred_lft forever 10 inet 172.16.23.223/32 scope global eth1 11 valid_lft forever preferred_lft forever 12 inet 172.16.23.222/32 scope global eth1 13 valid_lft forever preferred_lft forever
日志在 /var/log/message
1 Jan 3 21:02:33 debian-t6 Keepalived_vrrp[31784]: VRRP_Instance(nginx_dev_1) Sending/queueing gratuitous ARPs on eth1 for 172.16.23.222 2 Jan 3 21:02:33 debian-t6 Keepalived_vrrp[31784]: Sending gratuitous ARP on eth1 for 172.16.23.222 3 Jan 3 21:02:33 debian-t6 Keepalived_vrrp[31784]: Sending gratuitous ARP on eth1 for 172.16.23.222 4 Jan 3 21:02:33 debian-t6 Keepalived_vrrp[31784]: Sending gratuitous ARP on eth1 for 172.16.23.222 5 Jan 3 21:02:33 debian-t6 Keepalived_vrrp[31784]: Sending gratuitous ARP on eth1 for 172.16.23.222 6 Jan 3 21:03:02 debian-t6 Keepalived_vrrp[31784]: VRRP_Instance(nginx_dev_1) sent 0 priority 7 Jan 3 21:03:02 debian-t6 Keepalived_healthcheckers[31783]: Stopped 8 Jan 3 21:03:02 debian-t6 Keepalived_vrrp[31784]: VRRP_Instance(nginx_dev_1) removing protocol VIPs. 9 Jan 3 21:03:04 debian-t6 Keepalived_vrrp[31784]: Stopped 10 Jan 3 21:03:16 debian-t6 Keepalived_healthcheckers[31845]: Opening file '/etc/keepalived/keepalived.conf'. 11 Jan 3 21:03:16 debian-t6 Keepalived_vrrp[31846]: Registering Kernel netlink reflector 12 Jan 3 21:03:16 debian-t6 Keepalived_vrrp[31846]: Registering Kernel netlink command channel 13 Jan 3 21:03:16 debian-t6 Keepalived_vrrp[31846]: Registering gratuitous ARP shared channel 14 Jan 3 21:03:16 debian-t6 Keepalived_vrrp[31846]: Opening file '/etc/keepalived/keepalived.conf'. 15 Jan 3 21:03:16 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_2) removing protocol VIPs. 16 Jan 3 21:03:16 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_1) removing protocol VIPs. 17 Jan 3 21:03:16 debian-t6 Keepalived_vrrp[31846]: Using LinkWatch kernel netlink reflector... 18 Jan 3 21:03:16 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_1) Entering BACKUP STATE 19 Jan 3 21:03:16 debian-t6 Keepalived_vrrp[31846]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] 20 Jan 3 21:03:18 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_1) forcing a new MASTER election 21 Jan 3 21:03:20 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_2) Transition to MASTER STATE 22 Jan 3 21:03:20 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_2) Received advert with higher priority 200, ours 100 23 Jan 3 21:03:20 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_2) Entering BACKUP STATE 24 Jan 3 21:03:23 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_2) Transition to MASTER STATE 25 Jan 3 21:03:23 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_1) Transition to MASTER STATE 26 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_2) Entering MASTER STATE 27 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_2) setting protocol VIPs. 28 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.223 29 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_2) Sending/queueing gratuitous ARPs on eth1 for 172.16.23.223 30 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.223 31 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.223 32 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.223 33 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.223 34 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_1) Entering MASTER STATE 35 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_1) setting protocol VIPs. 36 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.222 37 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: VRRP_Instance(nginx_dev_1) Sending/queueing gratuitous ARPs on eth1 for 172.16.23.222 38 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.222 39 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.222 40 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.222 41 Jan 3 21:03:28 debian-t6 Keepalived_vrrp[31846]: Sending gratuitous ARP on eth1 for 172.16.23.222
5. KeepAlived其他配置
下面講解一些其他配置和高級應用
（1）email通知，這里就不做了，建議用第三方獨立監控服務，如Nagios、Zabbix進行監控
（2）router_id 用戶標識本節點名稱，通常為hostname
（3）vrrp_instance 實例定義
（4）state 實例狀態，只有MASTER 和 BACKUP兩種狀態，並且需要全部大寫。搶占模式下其中MASTER為工作狀態，BACKUP為備用狀態。當MASTER所在服務器失效時，BACKUP所在服務器會自動把它的狀態由BACKUP切換到MASTER狀態。當失效的MASTER所在的服務恢復時，BACKUP從MASTER恢復到BACKUP狀態
（5）interface 對望提供服務網卡接口，即VIP綁定的網卡接口。一些服務器都有兩個以上的網卡接口，在選擇接口時要正確選擇
（6）mcast_src_ip 本機IP地址
（7）virtual_router_id 虛擬路由的ID號，每個節點必須設置一樣。相同的VRID為一個組，將決定多播的MAC地址，不同的實例節點必須不一樣
（8）priority 節點優先級，取值0~254， MASTER 要比BACKUP高
（9）advert_int MASTER 與 BACKUP 節點間同步檢查的時間間隔，單位秒
（10）authentication 驗證類型和驗證密碼。主要用PASS 密碼模式 {auth_type PASS auth_pass 123456} 同一個vrrp實例MASTER與BACKUP使用相同的密碼才能正常通信
（11）nopreempt 禁止搶占服務。默認情況，當MASTER服務掛掉之后，BACKUP自動升級為MASTER並接替其任務，當MASTER服務恢復后，升級為MASTER的BACKUP服務又自動降為BACKUP，把工作權交給原MASTER。當配置了nopreempt，MASTER從掛到到恢復，不再將服務搶占過來
（12）virtual_ipaddress 虛擬IP地址池，可以有多個IP，每個IP占一行，不需要指定子網掩碼。注意這里的IP必須與我們設定的VIP保持一致 VRRP HA虛擬地址
（13）notify_master 表示當切換到master狀態時，要執行的腳本
（14）notify_backup 表示當切換到backup狀態時，要執行的腳本
（15）notify_fault 表示切換出現故障是要執行的腳本(這里也可以發送郵件什么的)
（16）track_script 執行監控的服務
（17）vrrp_script VRRP 腳本檢測
6. KeepAlived 高級應用
172.16.23.203
/etc/keepalived/keepalived.conf
1 global_defs { 2 router_id nginx_dev_1 3 } 4 5 #用於監控Nginx、Redis等應用是否在運行 6 vrrp_script chk_nginx_port { 7 script "/etc/keepalived/check.sh" #通過腳本檢測，根據返回值進行判斷 8 interval 2 #腳本執行間隔 每2秒執行一次 9 weight -5 #檢測失敗，優先級變更 10 fall 2 #連續檢測2次失敗才算失敗 11 rise 1 #檢測一次成功算成功，但不修改優先級 12 } 13 14 vrrp_instance nginx_dev_1 { 15 state MASTER 16 interface eth1 17 mcast_src_ip 172.16.23.203 18 virtual_router_id 100 19 priority 100 20 advert_int 5 21 authentication { 22 auth_type PASS 23 auth_pass 123456 24 } 25 virtual_ipaddress { #VRRP HA 虛擬地址，可以寫多個 26 172.16.23.222 27 } 28 29 notify_master "/etc/keepalived/run.sh master1" 30 notify_backup "/etc/keepalived/run.sh backup1" 31 notify_fault "/etc/keepalived/run.sh fault1" 32 33 track_script { 34 chk_nginx_port #執行對於腳本 35 } 36 } 37 38 vrrp_instance nginx_dev_2 { 39 state BACKUP 40 interface eth1 41 mcast_src_ip 172.16.23.203 42 virtual_router_id 101 43 priority 200 44 advert_int 5 45 authentication { 46 auth_type PASS 47 auth_pass 123456 48 } 49 virtual_ipaddress { 50 172.16.23.223 #VRRP HA虛擬地址 51 } 52 53 notify_master "/etc/keepalived/run.sh master2" 54 notify_backup "/etc/keepalived/run.sh backup2" 55 notify_fault "/etc/keepalived/run.sh fault2" 56 }
/etc/keepalived/check.sh
1 #!/bin/bash 2 counter=$(ps -C nginx --no-heading | wc -l) 3 if [ "${counter}" = "0" ]; then 4 service nginx start 5 #sleep 2 # 這個在執行過程中有問題 6 counter=$(ps -C nginx --no-heading | wc -l) 7 if [ "${counter}" = "0" ]; then 8 service keepalived stop 9 fi 10 fi
/etc/keepalived/run.sh
1 #!/bin/sh 2 echo $(date +%H:%M:%S) $1 >> /etc/keepalived/time.txt
172.16.23.204
/etc/keepalived/keepalived.conf
1 global_defs { 2 router_id nginx_dev_2 #運行Keepalived的機器標識號，可以相同也可以不同 3 } 4 5 #用於監控Nginx、Redis等應用是否在運行 6 vrrp_script chk_nginx_port { 7 script "/etc/keepalived/check.sh" 8 interval 2 9 weight -5 10 fall 2 11 rise 1 12 } 13 14 vrrp_instance nginx_dev_2 { 15 state MASTER 16 interface eth1 17 mcast_src_ip 172.16.23.204 18 virtual_router_id 101 19 priority 100 20 advert_int 5 21 authentication { 22 auth_type PASS 23 auth_pass 123456 24 } 25 virtual_ipaddress { 26 172.16.23.223 27 } 28 29 notify_master "/etc/keepalived/run.sh master1" 30 notify_backup "/etc/keepalived/run.sh backup1" 31 notify_fault "/etc/keepalived/run.sh fault1" 32 33 track_script { 34 chk_nginx_port 35 } 36 } 37 38 vrrp_instance nginx_dev_1 { 39 state BACKUP 40 interface eth1 41 mcast_src_ip 172.16.23.204 42 virtual_router_id 100 43 priority 200 44 advert_int 5 45 authentication { 46 auth_type PASS 47 auth_pass 123456 48 } 49 virtual_ipaddress { 50 172.16.23.222 51 } 52 53 notify_master "/etc/keepalived/run.sh master2" 54 notify_backup "/etc/keepalived/run.sh backup2" 55 notify_fault "/etc/keepalived/run.sh fault2" 56 57 track_script { 58 chk_nginx_port 59 } 60 }
/etc/keepalived/check.sh
1 #!/bin/bash 2 counter=$(ps -C nginx --no-heading | wc -l) 3 if [ "${counter}" = "0" ]; then 4 service nginx start 5 counter=$(ps -C nginx --no-heading | wc -l) 6 if [ "${counter}" = "0" ]; then 7 service keepalived stop 8 fi 9 fi
/etc/keepalived/run.sh
1 #!/bin/sh 2 echo $(date +%H:%M:%S) $1 >> /etc/keepalived/time.txt
上面的那些sh文件要 chmod a+x *.sh ，在keepalived.conf 文件中最好是使用絕對路徑
測試過程:
先啟動203 的KeepAlived和Nginx
然后啟動204的KeepAlived和Nginx
用瀏覽器訪問 172.16.23.223 / 172.16.23.222 都是沒有問題，基本被負載到兩台主機上去了
場景一
然后204機器 service nginx stop && service nginx status 模擬異常退出並查看狀態
過幾秒后再查看204 service nginx status 發現nginx 自動重啟了
場景二
修改check.sh 文件 把service nginx start 這一行注釋掉
然后204 機器 service nginx stop && service nginx status 模擬異常退出並查看狀態
過幾秒后再查看204 service nginx status / service keepalived status 發現nginx關閉、對於的204上的KeepAlived也被關閉
這個時候，203服務器 tail -f time.txt 會出現一條master切換信息 表示VIP進行切換
瀏覽器訪問 172.16.23.223/172.16.23.222 訪問正常，不過都是指向同一個機器
取消check.sh 文件的注釋
啟動204的 KeepAlived service keepalived start 由於取消注釋，對於的nginx也自動啟動了
這時，203服務器 tail -f time.txt 會出現一條backup切換信息，表示203主機切換為Backup狀態了。
這時瀏覽器訪問以上兩個IP，同樣被負載的兩台主機上