上篇中已經搭建了Nginx集群，但還缺少高可用的方案。開源的組件keepalived可以方便發送心跳監控Nginx並實現VIP，所有最終選擇keepalived實現高可用。
1、安裝環境：centos7 、keepalived-1.3.5
2、主機：192.168.30.7 從機：192.168.30.7 VIP：192.168.30.77
由於主從服務器配置一樣，只演示一台機器的配置。
1、安裝keepalived（使用源碼安裝）
==> 安裝庫依賴
yum install -y libnl* yum install -y libnfnetlink-devel zlib zlib-devel gcc gcc-c++ openssl openssl-devel openssh
==> 編譯源碼並安裝（將keepalived-1.3.5.tar.gz下載至當前目錄）
tar xvf keepalived-1.3.5.tar.gz cd keepalived-1.3.5 ./configure --prefix=/usr/local/keepalived make && make install
此時會在對應的路徑生產以下的文件
（1）/usr/local/etc/keepalived/keepalived.conf
（2）/usr/local/etc/sysconfig/keepalived
（3）/usr/local/sbin/keepalived
==> 初始化配置，設置為系統服務
（1）keepalived啟動腳本變量引用文件，copy至默認文件路徑是/etc/sysconfig/，也可以不做軟鏈接，直接修改啟動腳本中文件路徑即可（安裝目錄下）
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
（2）將keepalived主程序加入到環境變量（安裝目錄下）
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived
（3）keepalived啟動腳本（源碼目錄下），放到/etc/init.d/目錄下就可以使用service命令便捷調用
cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/init.d/keepalived
（4）將配置文件放到默認路徑下，此時啟動keepalive服務默認讀取該配置
mkdir -p /etc/keepalived cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
（5）將keepalived設置為系統服務 開機啟動
設置為系統服務開機啟動
chkconfig --add keepalived chkconfig keepalived on
查看開機啟動服務
chkconfig –-list
啟動服務|關閉|重啟命令
service keepalived start|stop|restart
主從配置一樣，這里不贅述
2、配置VIP和nginx可用性檢查
==> 在/etc/keepalived創建 check_nginx.sh
#!/bin/bash
A = `ps aux | grep nginx | grep -v grep | wc -l`
r = 2
if test $[A] -le $[r]
then
/usr/local/nginx/sbin/nginx
sleep 2
re = `ps axu | grep nginx | grep -v grep |wc -l`
if test $[re] -le $[r]
then
service keepalived stop
fi
fi
添加執行權限
chmod 755 /etc/keepalived/check_nginx.sh
==> 修改主節點keepalived.conf（主）
global_defs {
router_id lb-ecg
enable_script_security
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 5
weight -2
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 240
mcast_src_ip 192.168.30.7
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass nginx
}
virtual_ipaddress {
192.168.30.77/24
}
track_interface {
ens33
}
track_script {
chk_nginx
}
}
==> 修改從節點keepalived.conf（從）
global_defs {
router_id lb-ecg
enable_script_security
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 5
weight -2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 240
mcast_src_ip 192.168.30.8
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass nginx
}
virtual_ipaddress {
192.168.30.77/24
}
track_interface {
ens33
}
track_script {
chk_nginx
}
}
==> 啟動keepalived
service keepalived start
注意點：
（1）需要打開 enable_script_security 否則會 【security violation - scripts are being executed but script_security not enabled.】
（2）check_nginx.sh需要賦予755權限否則會 【keepalived unsafe permissions found for script】
3、VIP漂移檢查 和 nginx檢測
當keepalived、nginx配置完成並啟動機器192.168.30.7/8
==> 192.168.30.7機器：此時VIP(192.168.30.77)綁定在192.168.30.7的ens33的網卡上
[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff
inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.30.77/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@localhost keepalived]#
==> 192.168.30.8機器：未綁定VIP(192.168.30.77)
[root@localhost keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:8c:85:62 brd ff:ff:ff:ff:ff:ff
inet 192.168.30.8/24 brd 192.168.30.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::53f5:d47:6465:50c5/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::3be1:5305:816a:3b97/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::9b3b:3a95:5d99:f929/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
[root@localhost keepalived]#
==> 驗證性測試：
（1）kill掉192.168.30.7的keepalived，VIP(192.168.30.77)將漂移到192.168.30.8上。重新啟動192.168.30.7的keepalived，VIP(192.168.30.77)將漂回192.168.30.7。
結果如下： 192.168.30.7機器
[root@localhost keepalived]# ps aux | grep keepalived root 49997 0.0 0.0 48088 1048 ? Ss 00:39 0:00 /usr/local/keepalived/sbin/keepalived -D root 49998 0.0 0.0 48088 1792 ? S 00:39 0:00 /usr/local/keepalived/sbin/keepalived -D root 49999 0.0 0.0 48088 1392 ? S 00:39 0:00 /usr/local/keepalived/sbin/keepalived -D root 92683 0.0 0.0 48088 628 ? S 01:09 0:00 /usr/local/keepalived/sbin/keepalived -D root 92684 0.0 0.0 115304 1496 ? S 01:09 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 92749 0.0 0.0 112712 972 pts/2 R+ 01:09 0:00 grep --color=auto keepalived [root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.30.77/24 scope global secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@localhost keepalived]# [root@localhost keepalived]# kill -9 49997 [root@localhost keepalived]# ps aux | grep keepalived root 93225 0.0 0.0 112712 968 pts/2 R+ 01:10 0:00 grep --color=auto keepalived [root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@localhost keepalived]# [root@localhost keepalived]# service keepalived start Starting keepalived (via systemctl): ^C [root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:14:1e:e9 brd ff:ff:ff:ff:ff:ff inet 192.168.30.7/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.30.77/24 scope global secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link noprefixroute valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@localhost keepalived]#
192.168.30.8機器
。。。此時此時192.168.30.7的keepalived已經被kill掉
[root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:8c:85:62 brd ff:ff:ff:ff:ff:ff inet 192.168.30.8/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet 192.168.30.77/24 scope global secondary ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever
。。。。 此時192.168.30.7的keepalived重啟完畢
[root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:8c:85:62 brd ff:ff:ff:ff:ff:ff inet 192.168.30.8/24 brd 192.168.30.255 scope global noprefixroute ens33 valid_lft forever preferred_lft forever inet6 fe80::53f5:d47:6465:50c5/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::3be1:5305:816a:3b97/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever inet6 fe80::9b3b:3a95:5d99:f929/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever [root@localhost keepalived]#
（2）kill掉192.168.30.7的Nginx, 由於keepalived定時腳本 Nginx進程將自動恢復
[root@localhost keepalived]# ps aux | grep nginx root 113806 0.0 0.0 115304 1496 ? S 01:25 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 113871 0.0 0.0 112712 964 pts/2 R+ 01:25 0:00 grep --color=auto nginx root 127705 0.0 0.0 20568 672 ? Ss 00:04 0:00 nginx: master process /usr/local/nginx/sbin/nginx nobody 127707 0.0 0.0 20960 1636 ? S 00:04 0:00 nginx: worker process [root@localhost keepalived]# /usr/local/nginx/sbin/nginx -s stop [root@localhost keepalived]# ps aux | grep nginx root 114421 0.0 0.0 115304 1496 ? S 01:25 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 114500 0.0 0.0 115304 556 ? S 01:25 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 114502 0.0 0.0 112712 940 ? S 01:25 0:00 grep nginx root 114506 0.0 0.0 112712 964 pts/2 R+ 01:25 0:00 grep --color=auto nginx
。。。等待5秒之后 nginx進程重新啟動
[root@localhost keepalived]# ps aux | grep nginx root 114546 0.0 0.0 20568 676 ? Ss 01:25 0:00 nginx: master process /usr/local/nginx/sbin/nginx nobody 114548 0.0 0.0 20960 1388 ? S 01:25 0:00 nginx: worker process root 114775 0.0 0.0 115304 1500 ? S 01:26 0:00 /bin/bash /etc/keepalived/check_nginx.sh root 114783 0.0 0.0 20564 1640 ? S 01:26 0:00 /usr/local/nginx/sbin/nginx root 114813 0.0 0.0 112712 964 pts/2 R+ 01:26 0:00 grep --color=auto nginx [root@localhost keepalived]#
查看keepalived運行情況也可驗證：
[root@localhost keepalived]# service keepalived status ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: inactive (dead) since Wed 2020-01-08 01:13:22 CST; 15min ago Process: 96569 ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) CGroup: /system.slice/keepalived.service ├─ 96570 /usr/local/keepalived/sbin/keepalived -D ├─ 96571 /usr/local/keepalived/sbin/keepalived -D ├─ 96572 /usr/local/keepalived/sbin/keepalived -D ├─114546 nginx: master process /usr/local/nginx/sbin/nginx ├─114548 nginx: worker process ├─118238 /usr/local/keepalived/sbin/keepalived -D ├─118239 /bin/bash /etc/keepalived/check_nginx.sh └─118302 sleep 2 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.30.77 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77 Jan 08 01:13:24 localhost.localdomain Keepalived_vrrp[96572]: Sending gratuitous ARP on ens33 for 192.168.30.77 Jan 08 01:17:32 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Script(chk_nginx) timed out Jan 08 01:17:32 localhost.localdomain Keepalived_vrrp[96572]: /etc/keepalived/check_nginx.sh exited due to signal 15 Jan 08 01:17:32 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Instance(VI_1) Changing effective priority from 100 to 98 Jan 08 01:17:37 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Script(chk_nginx) succeeded Jan 08 01:17:37 localhost.localdomain Keepalived_vrrp[96572]: VRRP_Instance(VI_1) Changing effective priority from 98 to 100 [root@localhost keepalived]#
