keepalived介紹
Keepalived是一個基於vrrp協議的高可用方案，vrrp協議的軟件實現，原生設計的目的為了高可用ipvs服務。
1. vrrp協議
VRRP是一種容錯協議，它通過把幾台路由設備聯合組成一台虛擬的路由設備，並通過一定的機制來保證當主機的下一跳設備出現故障時，可以及時將業務切換到其它設備，從而保持通訊的連續性和可靠性，
- vrrp協議中常見術語：
- 虛擬路由器：Virtual Router
- 虛擬路由器標識：VRID(0-255)
- 物理路由器：master（主設備），backup（備用設備），priority（優先級）
- VIP：Virtual IP
- VMAC：Virtual MAC（00-00-5e-00-01-VIRD）
- 虛擬路由器工作機制如下：
- 根據優先級的大小挑選Master設備，比較優先級的大小，優先級高者當選為Master。
- 當兩台優先級相同的路由器同時競爭Master時，比較接口IP地址大小。接口地址大者當選為Master。
備份路由器隨時監聽Master的狀態。 - 當主路由器正常工作時，它會每隔一段時間（Advertisement_Interval）發送一個VRRP組播報文，以通知組內的備份路由器，主路由器處於正常工作狀態。
- 當組內的備份路由器一段時間（Master_Down_Interval）內沒有接收到來自主路由器的報文，則將自己轉為主路由器。
2. HA Cluster配置
2.1 HA Cluster的配置前提
- 各節點時間必須同步（ntp，chrony）。
- 確保iptables及selinux不會成為阻礙。
- 各節點之間可通過主機名互相通信（對KeepAlived並非必須），簡易使用/etc/hosts文件實現。
- 各節點之間的root用戶可以基於密鑰認證的ssh服務完成相互通信（非必須）；
- 配置keepalived的網卡必須支持並開啟多播(multicast)功能。（ip link set dev enoxxxxx multicast on|off）
2.2 HA Cluster的虛擬路由器配置
環境：兩台linux主機，配置虛擬路由器組，使用 10.1.
- 同步時間
-
安裝ntp軟件包
[root@ _8_ ~]# yum -y install ntp -
編輯ntp配置文件，使本機作為ntp時間服務器，注釋掉多行以server開頭的行，添加server 127.127.0.1
-
重啟ntp服務
[root@ _9_ ~]# service ntpd restart -
另一台主機2同步時間
[root@ _9_ ~]# ntpdate 10.1.6.11 1 Nov 18:38:03 ntpdate[46881]: adjust time server 10.1.6.11 offset -0.000035 sec
- keepalived 單主模型(10.1.7.19)
-
安裝keepalived
[root@ _14_ ~]# yum -y install keepalived -
主機1修改keepalived配置文件，注釋掉Virtual server的內容（這里暫不配置）
[root@ _15_ ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost #管理員郵箱 } notification_email_from keepalived@localhost #發送者keepalived smtp_server 127.0.0.1 #郵箱服務器 smtp_connect_timeout 30 #郵件發送超時時間 router_id node1 #當前路由器物理標識符 vrrp_mcast_group4 224.0.200.158 #多播地址（默認開啟），應該與其他組成虛擬路由器的主機一致 } vrrp_instance VI_1 { #配置vrp示例，VI_1，隨意定義，需唯一 state MASTER #定義當前節點在此虛擬路由器上的初始狀態；只能有一個是MASTER，其余都為BACKUP interface eth0 #綁定為當前虛擬路由器使用的物理接口 virtual_router_id 16 #當前虛擬路由器的唯一標識（0-255） priority 100 #當前主機在此虛擬路由器中的優先級 advert_int 1 #vrrp通告的時間間隔 authentication { auth_type PASS #認證類型，PASS為簡單認證，AH為復雜認證，推薦使用PASS auth_pass RrpIoZU7 #認證字符 } virtual_ipaddress { 10.1.7.19/16 dev eth0 #配置的接口虛擬ip } } 注：網卡多播功能開啟與關閉：ip link set multicast on|off -
使用scp拷貝給另一台主機2，修改對應參數 。
[root@ _15_ ~]# scp /etc/keepalived/keepalived.conf root@10.1.7.11:/etc/keepalived/keepalived.conf 修改 state MASTER 為 state BACKUP priority 100 為 priority 98 備用節點優先級要比主節點低
- 測試
-
啟動主節點，能看到啟動為MASTER，添加了10.1.7.19的地址
[root@ _1_ ~]# service keepalived start Starting keepalived: [root@ _1_ ~]# tail /var/log/message Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Opening file '/etc/keepalived/keepalived.conf'. Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Configuration is using : 7552 Bytes Nov 1 20:13:44 localhost kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP) Nov 1 20:13:44 localhost kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes) Nov 1 20:13:44 localhost kernel: IPVS: ipvs loaded. Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Using LinkWatch kernel netlink reflector... Nov 1 20:13:44 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 20:13:45 localhost Keepalived_healthcheckers[36312]: Netlink reflector reports IP 10.1.7.19 added Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:13:47 localhost ntpd[2238]: Listen normally on 8 eth0 10.1.7.19 UDP 123 Nov 1 20:13:50 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 [root@ _2_ ~]# ip a 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever -
啟動備用節點,可看到由於主節點正常運行，備用節點並未搶占主節點ip
[root@ _3_ ~]# service keepalived start Starting keepalived: [ OK ] [root@ _1_ ~]# tail /var/log/message Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Opening file '/etc/keepalived/keepalived.conf'. Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Configuration is using : 7556 Bytes Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Using LinkWatch kernel netlink reflector... [root@ _4_ ~]# ip a 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever -
關閉主節點keepalived服務后
[root@ _6_ ~]# service keepalived stop Stopping keepalived: [ OK ] 以下為主節點日志：keepalived服務關閉，vrrp ip被移除 Nov 1 20:28:17 localhost Keepalived[36349]: Stopping Keepalived v1.2.13 (03/19,2015) Nov 1 20:28:17 localhost Keepalived_vrrp[36352]: VRRP_Instance(VI_1) sending 0 priority Nov 1 20:28:17 localhost Keepalived_vrrp[36352]: VRRP_Instance(VI_1) removing protocol VIPs. Nov 1 20:28:17 localhost Keepalived_healthcheckers[36351]: Netlink reflector reports IP 10.1.7.19 removed 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 以下為備用節點日志：轉換為MASTER角色，配置10.1.7.19的vrrp ip Nov 1 20:28:18 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 20:28:19 localhost Keepalived_healthcheckers[2229]: Netlink reflector reports IP 10.1.7.19 added Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:28:24 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever -
恢復主節點，啟動主節點keepalived服務后
[root@ _8_ ~]# service keepalived start Starting keepalived: [ OK ] 以下為主節點日志，轉換為MASTER角色，搶占10.1.7.19的ip Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 20:34:20 localhost Keepalived_healthcheckers[36430]: Netlink reflector reports IP 10.1.7.19 added Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:34:22 localhost ntpd[2238]: Listen normally on 10 eth0 10.1.7.19 UDP 123 Nov 1 20:34:25 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 以下為備用節點日志，轉換為BACKUP角色，ip 10.1.7.19被移除 Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Received higher prio advert Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Entering BACKUP STATE Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) removing protocol VIPs. Nov 1 20:34:19 localhost Keepalived_healthcheckers[2229]: Netlink reflector reports IP 10.1.7.19 removed 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever
- keepalived 雙主模型(10.1.7.19，10.1.7.20)
-
在單主模型的基礎上，主機1再增加一個vrrp_instance段，改動的地方為
vrrp_instance VI_2 { #vrrp示例編號要修改，不能與其他實例相同 state BACKUP #初始狀態，上一個實例為MASTER，此時這里為BACKUP interface eth0 virtual_router_id 17 priority 98 #優先級要比另一台主機MASTER的優先級低 advert_int 1 authentication { auth_type PASS auth_pass 2a6561b9 #認證字符串要修改 } virtual_ipaddress { 10.1.7.20/16 dev eth0 #配置的另一個ip要修改 } } -
與之對應的另一台主機2上增加的配置為
vrrp_instance VI_2 { state MASTER interface eth0 virtual_router_id 17 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 2a6561b9 } virtual_ipaddress { 10.1.7.20/16 dev eth0 } }
- 測試
-
兩台主機先停止keepalived服務，然后先啟動主機1
以下為主機1日志：VI_1啟動為MASTER角色，配置10.1.7.19的ip，VI_2啟動為MASTER角色，配置10.1.7.20的ip Nov 1 20:57:42 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 20:57:43 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.19 added Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:57:45 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Transition to MASTER STATE Nov 1 20:57:45 localhost ntpd[2238]: Listen normally on 11 eth0 10.1.7.19 UDP 123 Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Entering MASTER STATE Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) setting protocol VIPs. Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 Nov 1 20:57:46 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.20 added Nov 1 20:57:47 localhost ntpd[2238]: Listen normally on 12 eth0 10.1.7.20 UDP 123 Nov 1 20:57:48 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 20:57:51 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global secondary eth0 inet 10.1.7.20/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever -
啟動主機2
以下為主機1日志：VI_2轉換為BACKUP角色，10.1.7.20的ip被移除： Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Received higher prio advert Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Entering BACKUP STATE Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) removing protocol VIPs. Nov 1 21:03:36 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.20 removed 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.19/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 以下為主機2的日志，VI_2轉換為MASTER角色，配置了10.1.7.20的ip Nov 1 21:03:36 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Transition to MASTER STATE Nov 1 21:03:36 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Received lower prio advert, forcing new election Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Entering MASTER STATE Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) setting protocol VIPs. Nov 1 21:03:37 localhost Keepalived_healthcheckers[2378]: Netlink reflector reports IP 10.1.7.20 added Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 Nov 1 21:03:42 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.20/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever -
停止主機1的keepalived服務。
以下為主機2日志：VI_1轉換為MASTER角色，配置了10.1.7.19的ip Nov 1 21:07:47 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 1 21:07:48 localhost Keepalived_healthcheckers[2378]: Netlink reflector reports IP 10.1.7.19 added Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 Nov 1 21:07:53 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.7.20/16 scope global secondary eth0 inet 10.1.7.19/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever
3. Keepalived集群+ipvs（DR）集群
拓撲環境
10.1.6.11和10.1.6.12為兩台real server，提供web服務。
左邊兩台服務器，主節點ip10.1.6.11，備節點ip10.1.6.12
主節點和備節點做成keepalived高可用集群。IP為10.1.8.88
-
兩台real server 安裝httpd，編輯測試主頁，啟動httpd服務，在主節點或備用節點上請求測試主頁
[root@ _2_ ~]# yum -y install httpd [root@ _2_ ~]# cat /var/www/html/index.html <h1>Server 1</h1> [root@ _2_ ~]# yum -y install httpd [root@ _2_ ~]# cat /var/www/html/index.html <h1>Server 2</h1> [root@ _3_ ~]# curl http://10.1.7.11 <h1>Server 1</h1> [root@ _4_ ~]# curl http://10.1.7.12 <h1>Server 2</h1> -
Real Server上編寫DR模型初始配置腳本，兩台real server運行此腳本
#!/bin/bash vip='10.1.8.88' vport='80' netmask='255.255.255.255' iface='lo:0' case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $iface $vip netmask $netmask broadcast $vip up route add -host $vip dev $iface ;; stop) ifconfig $iface down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ;; *) echo "Usage $(basename $0) start|stop" exit 1 ;; esac 檢查配置 lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 10.1.8.88 netmask 255.255.255.255 loop txqueuelen 0 (Local Loopback) -
主節點和備用節點上配置keepalived
以下為主節點配置，備用節點需修改state為BACKUP ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from Keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 vrrp_mcast_group4 224.0.200.158 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 16 priority 98 advert_int 1 authentication { auth_type PASS auth_pass 2a6561b8 } virtual_ipaddress { 10.1.8.88/16 dev eth0 } } -
測試主備節點分別故障時，虛擬ip能來回切換
-
主節點和備節點安裝ipvsadm，測試調度后端real server，確保調度正常
主節點： [root@ _8_ ~]# yum -y install ipvsadm [root@ _8_ ~]# ipvsadm -A -t 10.1.8.88:80 -s rr [root@ _9_ ~]# ipvsadm -a -t 10.1.8.88:80 -r 10.1.7.11 -g -w 1 [root@ _10_ ~]# ipvsadm -a -t 10.1.8.88:80 -r 10.1.7.12 -g -w 1 [root@ _11_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 0 -> 10.1.7.12:80 Route 1 0 0 [root@ _13_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> 調度正常，清空規則 [root@ _25_ ~]# ipvsadm -C 備用節點同樣方法測試一次 -
定義keepalived配置中Virtual Server
主節點和備節點的keepalived配置文件中加入Virtual server配置段 virtual_server 10.1.8.88 80 { #virtual_server ip地址 delay_loop 3 #服務輪詢時間間隔 lb_algo rr #定義調度算法 lb_kind DR #定義lvs的類型 protocol TCP #服務協議，僅支持tcp real_server 10.1.7.11 80 { #real_server ip地址 weight 1 #權重 HTTP_GET { #請求方法 url { path / #定義監控的url status_code 200 #判斷上述檢測機制為健康狀態的響應碼為200 } connect_timeout 1 #連接超時時間 nb_get_retry 3 #重試的次數 delay_before_retry 1 #重試之前延遲時長 } } real_server 10.1.7.12 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } } -
啟動主節點和備節點keepalived服務，查看ip以及ipvs規則
主節點 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever [root@ _33_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 0 -> 10.1.7.12:80 Route 1 0 0 備節點 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link valid_lft forever preferred_lft forever [root@ _28_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 0 -> 10.1.7.12:80 Route 1 0 0 -
使用客戶端對其進行訪問檢測
測試正常 [root@ _16_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> -
使real server中有一個故障，檢測訪問
停掉real server的httpd服務 [root@ _5_ ~]# systemctl stop httpd 主節點上查看ipvs規則，real server 2已下線 [root@ _38_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 10 客戶端請求訪問 [root@ _17_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> <h1>Server 1</h1> 訪問正常 -
使故障的real server恢復，檢測訪問
[root@ _6_ ~]# systemctl start httpd 查看主節點上ipvs規則，real server已加入 [root@ _39_ ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.8.88:80 rr -> 10.1.7.11:80 Route 1 0 0 -> 10.1.7.12:80 Route 1 0 0 客戶端訪問測試 [root@ _18_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> <h1>Server 2</h1> <h1>Server 1</h1> 調度正常
4. keepalived主備節點上配置sorry server
-
主備節點分別安裝httpd，編輯網頁文件，最好都停掉keepalived服務
[root@ _41_ ~]# yum -y install httpd 主節點編輯網頁文件 [root@ _38_ ~]# cat /var/www/html/index.html <h1>LB Cluster Fault,this is Sorry Server 1</h1> 主節點編輯網頁文件 [root@ _38_ ~]# cat /var/www/html/index.html <h1>LB Cluster Fault,this is Sorry Server 2</h1> -
編輯keepalived配置文件，在Virtual server中添加sorry server配置，主節點和備節點都要配置
virtual_server 10.1.8.88 80 { delay_loop 3 lb_algo rr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 10.1.7.11 80 { weight 1 ... -
主備節點都啟動httpd服務，啟動keepalived服務，兩台real server都停止httpd服務
[root@ _48_ ~]# service httpd start [root@ _44_ ~]# service keepalived start Starting keepalived: [ OK ] [root@ _12_ ~]# systemctl stop httpd -
客戶端請求測試
看到sorry server的響應 [root@ _22_ ~]# curl http://10.1.8.88 <h1>LB Cluster Fault,this is Sorry Server 1</h1> -
啟動一台real server的httpd服務，客戶端測試
響應正常 [root@ _23_ ~]# curl http://10.1.8.88 <h1>Server 2</h1>
5.keeplioved調用外部腳本，由結果實時調整優先級
-
腳本的定義與調用介紹
-
（1）腳本先定義 vrrp_script<SCRIPT_NAME> { script "" interval INT weight -INT } （2）再調用 track_script { SCRIPT_NAME_1 SCRIPT_NAME_2 ... } -
主備節點keepalived配置加入腳本段，腳本檢測到/etc/keepalived/down則返回失敗
主節點 ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from Keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 vrrp_mcast_group4 224.0.200.158 } vrrp_script chk_down { #定義腳本名 script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0" #判斷down文件，存在則返回失敗 interval 1 #每隔多長時間腳本執行一次 weight -5 #腳本失敗動作，權重-5，確保-5后低於備用優先級 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 16 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 2a6561b8 } virtual_ipaddress { 10.1.8.88/16 dev eth0 } track_script { #監控的腳本 chk_down #調用的腳本名稱 } } 備節點需要修改state為BACKUP，priority為98 -
主備節點啟動keepalived服務，查看ip
[root@ _72_ /etc/keepalived]# service keepalived stop Starting keepalived: [ OK ] 主節點 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 備節點 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 -
創建/etc/keepalived/down文件，查看ip轉移
[root@ _161_ /etc/keepalived]# touch down 主節點 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link 備節點 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link 主節點檢測腳本，返回失敗，權重-5，轉換為BACKUP角色，ip 10.1.8.88被移除 Nov 3 08:24:02 localhost Keepalived_vrrp[4853]: VRRP_Script(chk_down) failed Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Received higher prio advert Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Entering BACKUP STATE Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) removing protocol VIPs. Nov 3 08:24:03 localhost Keepalived_healthcheckers[4852]: Netlink reflector reports IP 10.1.8.88 removed -
刪除主節點/etc/keepalived/down文件，查看ip轉移
[root@ _163_ /etc/keepalived]# rm -rf down 主節點，ip已奪回 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link 日志 Nov 3 08:32:01 localhost Keepalived_healthcheckers[4852]: Netlink reflector reports IP 10.1.8.88 added Nov 3 08:32:01 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.8.88 Nov 3 08:32:03 localhost ntpd[4558]: Listen normally on 11 eth0 10.1.8.88 UDP 123
6. keepalived結合nginx調度（並使用輔助腳本檢測nginx服務）
-
清空上例在real server上所做的lo：0接口規則以及ARP限制規則，並停止keepalived服務
[root@ _16_ ~]# bash set_dr stop #!/bin/bash vip='10.1.8.88' vport='80' netmask='255.255.255.255' iface='lo:0' case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $iface $vip netmask $netmask broadcast $vip up route add -host $vip dev $iface ;; stop) ifconfig $iface down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ;; *) echo "Usage $(basename $0) start|stop" exit 1 ;; esac -
主備節點停止為sorry server啟動的httpd服務
[root@ _50_ ~]# service httpd stop Stopping httpd: [ OK ] -
主備節點安裝nginx
[root@ _173_ /etc/keepalived]# yum -y install nginx -
編輯nginx配置文件，實現反代
在/etc/nginx/nginx.conf的http上下文中添加 upstream websrvs { server 10.1.7.11; server 10.1.7.12; } 在/etc/nginx/conf.d/default.conf的location上下文中添加 proxy_pass http://websrvs; 如 location / { root /usr/share/nginx/html; proxy_pass http://websrvs; index index.html index.htm; } -
主備節點啟動nginx服務，keepalived服務，訪問測試
[root@ _18_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> [root@ _19_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> [root@ _20_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> [root@ _21_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> 訪問正常 -
主備節點添加vrrp_script腳本，在nginx沒有啟動時觸發
vrrp_script chk_nginx { script "killall -0 nginx && exit 0 || exit 1" interval 1 weigth } 追蹤腳本中也需要加入chk_nginx track_script { chk_down chk_nginx } -
主節點重啟keepalived服務，而后備節點重啟keepalived服務
此時虛擬ip 10.1.8.88在主節點上 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 客戶端訪問正常 [root@ _22_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> [root@ _23_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> [root@ _24_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> -
停止主節點nginx服務
[root@ _12_ ~]# service nginx stop Stopping nginx: [ OK ] 主節點ip已移除 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0 inet6 fe80::20c:29ff:fe9c:147c/64 scope link valid_lft forever preferred_lft forever 日志信息，檢測到vrrp_script失敗，轉換為BACKUP模式，ip已移除 Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Script(chk_nginx) failed Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) Entering FAULT STATE Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) removing protocol VIPs. Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) Now in FAULT state Nov 3 18:00:25 localhost Keepalived_healthcheckers[75163]: Netlink reflector reports IP 10.1.8.88 removed 備節點ip與日志，10.1.8.88地址已獲取，轉換為MASTER角色 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0 inet 10.1.8.88/16 scope global secondary eth0 inet6 fe80::20c:29ff:feaf:fdec/64 scope link Nov 3 18:00:26 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) Transition to MASTER STATE Nov 3 18:00:27 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) Entering MASTER STATE Nov 3 18:00:27 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) setting protocol VIPs. Nov 3 18:00:27 localhost Keepalived_healthcheckers[75083]: Netlink reflector reports IP 10.1.8.88 added -
客戶端訪問測試
調度正常 [root@ _25_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> [root@ _26_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1> [root@ _27_ /etc]# curl http://10.1.8.88 <h1>Server 2</h1> [root@ _28_ /etc]# curl http://10.1.8.88 <h1>Server 1</h1>
補充:keepalived發生角色轉移時運行指定腳本
-
備節點上編寫腳本/etc/keepalived/motify.sh，當角色切換時，給root用戶發送郵件
#!/bin/bash # contact='root@localhost' notify() { mailsubject="$(hostname) to be $1, vip floating." mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master ;; backup) notify backup ;; fault) notify fault ;; *) echo "Usage: $(basename $0) {master|backup|fault}" exit 1 ;; esac -
在vrrp_instance端中調用腳本，並重啟keepalived服務
notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault"
3.主節點停止keepalived服務，查看root用戶郵件
備節點：收到轉換為master角色的郵件
>N 1 root Thu Nov 3 18:41 18/731 "localhost.localdomain to be master, vip floating."
& 1
Message 1:
From root@localhost.localdomain Thu Nov 3 18:41:46 2016
Return-Path: <root@localhost.localdomain>
X-Original-To: root@localhost
Delivered-To: root@localhost.localdomain
Date: Thu, 03 Nov 2016 18:41:46 +0800
To: root@localhost.localdomain
Subject: localhost.localdomain to be master, vip floating.
User-Agent: Heirloom mailx 12.4 7/29/08
Content-Type: text/plain; charset=us-ascii
From: root@localhost.localdomain (root)
Status: R
2016-11-03 18:41:46: vrrp transition, localhost.localdomain changed to be master
ip 10.1.8.88已添加
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0