keepalived作用
在大型網站建設中，我們通常會用一些負載均衡技術（LVS、Nginx、Haproxy），將請求分發到后端的服務集群中。
這時，負載均衡均衡節點就成為單點故障的節點，為了保證系統的高可用，可以引入keepalived，將多個負載均衡節點聯合起來作為一個整體對外服務，從而防止單點故障。
keepalived工作原理
keepalived是以VRRP協議為基礎實現的，VRRP全稱Virtual Router Redundancy Protocol，是一種路由容錯協議，也可以叫做備份路由協議。一組VRRP路由器協同工作，共同構成一台虛擬路由器。該虛擬路由器對外表現為一個具有MAC地址和唯一固定IP地址的邏輯路由器。實際工作時，這一組路由器中的一個作為master節點，其余作為backup節點。工作期間，master會定時發送VRRP協議包，當master節點出現故障，backup節點收不到VRRP包，其中一個backup節點（priority大的）會自動成為master節點，繼續提供服務，此過程完全自動，不需要人工干預。
keepalived利用這個原理，將不同的服務器節點（負載均衡節點、web服務器）組件成一個服務器組，對外提供統一的（VIP）地址，從而保證系統高可用。
keepalived安裝
$ mkdir -p /opt/k8s/keepalive-haproxy
$ cd /opt/k8s/keepalive-haproxy
$ wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz
$ tar -xzvf keepalived-2.0.20.tar.gz
$ cd /opt/k8s/keepalive-haproxy/keepalived-2.0.20
# 執行configure，prefix指定安裝目錄
$ mkdir -p /opt/k8s/keepalived/
$ ./configure --prefix=/opt/k8s/keepalived/
$ make & make install
# 查看安裝目錄
$ ls /opt/k8s/keepalived/
bin etc sbin share
簡單例子
下面來以keepalived結合tomcat來實現一個web服務器的雙機熱備。網絡topology如下：
- 整個系統兩個節點， 節點web1 192.168.0.114 (主節點), 節點web2 192.168.0.107(備用節點)，虛擬IP(對外提供服務的IP 192.168.0.100)
- 兩個節點上面都運行着一個同樣的web服務，暴露在8086端口，在兩個節點上都安裝keepalived，外部通過虛擬IP訪問系統
- 在實際工作過程中虛擬IP在某時刻只能屬於某一個節點，另一個節點作為備用節點存在
- 當主節點正常的時候:節點web1上的keepalived廣播出去的信息中，192.168.0.100 這個IP對應的MAC地址為節點web1網卡的MAC地址
- 同一網段內的機器會更新自己的ARP表，對應192.168.0.100的MAC地址=節點web1網卡的MAC地址
- 當節點web1發生故障的時候，節點web2上的keepalived會檢測到，並且將下面的信息廣播出去:
192.168.8.100 這個IP對應的MAC地址為節點web2網卡的MAC地址 - 同一網段內的其它電腦如客戶端會更新自己的ARP表，對應192.168.8.100的MAC地址=節點web2網卡的MAC地址
-
配置主節點的keepalived
$ mkdir /opt/k8s/keepalived/conf $ cd /opt/k8s/keepalived/conf $ cat > keepalived.conf<<EOF global_defs { router_id web1 } vrrp_instance VI_1 { state MASTER #設置為主服務器 interface wlo1 #監測網絡接口 virtual_router_id 55 #主、備必須一樣 priority 200 #(主、備機取不同的優先級，主機值較大，備份機值較小,值越大優先級越高) advert_int 1 #VRRP Multicast廣播周期秒數 authentication { auth_type PASS #VRRP認證方式，主備必須一致 auth_pass 1111 #(密碼) } virtual_ipaddress { 192.168.0.100/24 #VRRP HA虛擬地址 } } EOF -
配置備用節點的keepalived
$ mkdir /opt/k8s/keepalived/conf $ cd /opt/k8s/keepalived/conf $ cat > keepalived.conf<<EOF global_defs { router_id web2 } vrrp_instance VI_1 { state BACKUP #設置為主服務器 interface wlp3s0 #監測網絡接口 virtual_router_id 55 #主、備必須一樣 priority 100 #(主、備機取不同的優先級，主機值較大，備份機值較小,值越大優先級越高) advert_int 1 #VRRP Multicast廣播周期秒數 authentication { auth_type PASS #VRRP認證方式，主備必須一致 auth_pass 1111 #(密碼) } virtual_ipaddress { 192.168.0.100/24 #VRRP HA虛擬地址 } } EOF -
在主備節點上啟動keepalived服務
$ /opt/k8s/keepalived/sbin/keepalived -f /opt/k8s/keepalived/conf/keepalived.conf -
查看日志
主節點(ubuntu系統，在syslog中）
$ tail -f /var/log/syslog | grep Keepalived Mar 13 17:21:42 slave Keepalived[11843]: Starting Keepalived v2.0.20 (01/22,2020) Mar 13 17:21:42 slave Keepalived[11843]: Running on Linux 5.3.0-28-generic #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 (built for Linux 4.15.18) Mar 13 17:21:42 slave Keepalived[11843]: Command line: '/opt/k8s/keepalived/sbin/keepalived' '-f' '/opt/k8s/keepalived/conf/keepalived.conf' Mar 13 17:21:42 slave Keepalived[11843]: Opening file '/opt/k8s/keepalived/conf/keepalived.conf'. Mar 13 17:21:42 slave Keepalived[11843]: Remove a zombie pid file /run/keepalived.pid Mar 13 17:21:42 slave Keepalived[11844]: Starting VRRP child process, pid=11845 Mar 13 17:21:42 slave Keepalived_vrrp[11845]: Registering Kernel netlink reflector Mar 13 17:21:42 slave Keepalived_vrrp[11845]: Registering Kernel netlink command channel Mar 13 17:21:42 slave Keepalived_vrrp[11845]: Opening file '/opt/k8s/keepalived/conf/keepalived.conf'. Mar 13 17:21:42 slave Keepalived_vrrp[11845]: Registering gratuitous ARP shared channel Mar 13 17:21:42 slave Keepalived_vrrp[11845]: (VI_1) Entering BACKUP STATE (init) Mar 13 17:21:45 slave Keepalived_vrrp[11845]: (VI_1) Entering MASTER STATE從節點
$ tail -f /var/log/syslog | grep Keepalived Mar 13 17:24:30 master Keepalived[3118]: Starting Keepalived v2.0.20 (01/22,2020) Mar 13 17:24:30 master Keepalived[3118]: Running on Linux 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 (built for Linux 4.15.18) Mar 13 17:24:30 master Keepalived[3118]: Command line: '/opt/k8s/keepalived/sbin/keepalived' '-f' '/opt/k8s/keepalived/conf/keepalived.conf' Mar 13 17:24:30 master Keepalived[3118]: Opening file '/opt/k8s/keepalived/conf/keepalived.conf'. Mar 13 17:24:30 master Keepalived[3118]: Remove a zombie pid file /run/keepalived.pid Mar 13 17:24:30 master Keepalived[3120]: Starting VRRP child process, pid=3122 Mar 13 17:24:30 master Keepalived_vrrp[3122]: Registering Kernel netlink reflector Mar 13 17:24:30 master Keepalived_vrrp[3122]: Registering Kernel netlink command channel Mar 13 17:24:30 master Keepalived_vrrp[3122]: Opening file '/opt/k8s/keepalived/conf/keepalived.conf'. Mar 13 17:24:30 master Keepalived_vrrp[3122]: Registering gratuitous ARP shared channel Mar 13 17:24:30 master Keepalived_vrrp[3122]: (VI_1) Entering BACKUP STATE (init) -
查看主節點網路設備
$ ip addr show wlo1 3: wlo1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether c4:8e:8f:d0:33:89 brd ff:ff:ff:ff:ff:ff inet 192.168.0.114/24 brd 192.168.0.255 scope global dynamic noprefixroute wlo1 valid_lft 4948sec preferred_lft 4948sec inet 192.168.0.100/24 scope global secondary wlo1 valid_lft forever preferred_lft forever inet6 fe80::db1e:449b:5431:7392/64 scope link noprefixroute valid_lft forever preferred_lft forever- 可以看到在網卡wlo1下面的多了一個地址192.168.0.100/24
-
部署web服務
基於spring boot構建了一個應用，利用embeded-tomcat，里面提供了一個restful接口,對應暴露的端口設置成8086
@RequestMapping("/health") public String health(HttpServletRequest request) { return "OK"; }在主備節點上都啟動spring boot應用
-
通過虛擬IP訪問
$ curl http://192.168.0.100:8086/health OK
模擬故障以及恢復
-
停掉主節點的keepalived進程
$ ps -elf | grep -e keepa -e PID F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 1 S root 11844 1 0 80 0 - 4188 ep_pol 17:21 ? 00:00:00 /opt/k8s/keepalived/sbin/keepalived -f /opt/k8s/keepalived/conf/keepalived.conf 5 S root 11845 11844 0 80 0 - 4188 ep_pol 17:21 ? 00:00:00 /opt/k8s/keepalived/sbin/keepalived -f /opt/k8s/keepalived/conf/keepalived.conf 0 S root 19389 16663 0 80 0 - 5383 pipe_w 18:06 pts/0 00:00:00 grep --color=auto -e keepa -e PID $ kill -9 11844 -
查看主從節點網路設備
主節點
$ ip addr show wlo1 3: wlo1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether c4:8e:8f:d0:33:89 brd ff:ff:ff:ff:ff:ff inet 192.168.0.114/24 brd 192.168.0.255 scope global dynamic noprefixroute wlo1 valid_lft 5707sec preferred_lft 5707sec inet6 fe80::db1e:449b:5431:7392/64 scope link noprefixroute valid_lft forever preferred_lft forever從節點
$ ip addr show wlp3s0 3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d0:c5:d3:57:73:01 brd ff:ff:ff:ff:ff:ff inet 192.168.0.107/24 brd 192.168.0.255 scope global dynamic noprefixroute wlp3s0 valid_lft 4597sec preferred_lft 4597sec inet 192.168.0.100/24 scope global secondary wlp3s0 valid_lft forever preferred_lft forever inet6 fe80::1fda:e90a:207a:67e4/64 scope link noprefixroute valid_lft forever preferred_lft forever- 可以看到虛擬IP漂移到了從節點的網卡上了
-
通過虛擬IP訪問
$ curl http://192.168.0.100:8086/health OK- 服務正常響應
-
重新啟動web1上的keepalived進程，再次查看主節點上的wlo1網卡，會發現虛擬IP又自動漂移回來
腳本檢測
上一小節我們通過模擬keepalived進程down掉，驗證了VIP自動遷移。可是當主節點上的keepalived功能正常，而只是web服務出現故障，按照之前的簡單配置，VIP是不會自動遷移的，這時服務就變的不可訪問了。解決方法是在keepalived的配置文件中追加vrrp_script
示例如下：
-
編寫檢測腳本
$ mkdir /opt/k8s/keepalived/script $ cd /opt/k8s/keepalived/script $ cat > checkproxy.sh<<EOF #!/bin/bash count = `ps aux | grep -v grep | grep haproxy | wc -l` if [ $count > 0 ]; then exit 0 else exit 1 fi -
在配置文件中追加檢測
... vrrp_script checkhaproxy { script "/home/checkproxy.sh" interval 3 weight -150 } vrrp_instance test { ... track_script { checkhaproxy } ... } -
關於check中的weight
- vrrp_script 里的script返回值為0時認為檢測成功，其它值都會當成檢測失敗；
- weight 為正時，腳本檢測成功時此weight會加到priority上，檢測失敗時不加；
- 主失敗: 主 priority < 從 priority + weight 時會切換。
- 主成功： 主 priority + weight > 從 priority + weight 時，主依然為主
- weight 為負時，腳本檢測成功時此weight不影響priority，檢測失敗時priority – abs(weight)
- 主失敗: 主 priority – abs(weight) < 從priority 時會切換主從
- 主成功: 主 priority > 從priority 主依然為主