lvs+keepalived+nginx實現高性能負載均衡集群
一、為什么要使用負載均衡技術？
1、系統高可用性
2、 系統可擴展性
3、 負載均衡能力
LVS+keepalived能很好的實現以上的要求，LVS提供負載均衡，keepalived提供健康檢查，故障轉移，提高系統的可用性！采用這樣的架構以后很容易對現有系統進行擴展，只要在后端添加或者減少realserver，只要更改lvs的配置文件，並能實現無縫配置變更！
二、LVS+Keepalived介紹
1、 LVS
LVS是一個開源的軟件，可以實現LINUX平台下的簡單負載均衡。LVS是Linux Virtual Server的縮寫，意思是Linux虛擬服務器。目前有三種IP負載均衡技術（VS/NAT、VS/TUN和VS/DR）；八種調度算法（rr,wrr,lc,wlc,lblc,lblcr,dh,sh）。
2、 keepalived
Keepalived 是運行在lvs 之上，它的主要功能是實現真實機的故障隔離及負載均衡器間的失敗切換，提高系統的可用性
三、環境：
四台服務器，系統全為CentOS6.8：
192.168.2.203 master lvs+keepalived
192.168.2.202 backup lvs+keepalived
192.168.2.204 web1（nginx）
192.168.2.205 web2 （nginx）
vip：192.168.2.13
其中nginx已預裝好，這里不再寫搭建過程
四、搭建並配置
1、分別在backup lvs和master lvs上安裝lvs
1 root@bogon src]# yum -y install ipvsadm
2 已加載插件：fastestmirror
3 設置安裝進程
4 Determining fastest mirrors
5 epel/metalink | 5.4 kB 00:00
6 * base: mirror.lzu.edu.cn
7 ... ...
8 已安裝:
9 ipvsadm.x86_64 0:1.26-4.el6
10
11 作為依賴被安裝:
12 libnl.x86_64 0:1.1.4-2.el6
13
14 完畢！
2、把ipvsadm模塊加載進系統
1 [root@bogon src]# ipvsadm
2 IP Virtual Server version 1.2.1 (size=4096)
3 Prot LocalAddress:Port Scheduler Flags
4 -> RemoteAddress:Port Forward Weight ActiveConn InActConn
5 [root@bogon src]# lsmod | grep ip_vs
6 ip_vs 126897 0
7 libcrc32c 1246 1 ip_vs
8 ipv6 336282 270 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6
3、分別在backup lvs和master lvs上安裝keepalived（keepalived官網：http://www.keepalived.org/）
[root@bogon src]# tar zxf keepalived-1.2.24.tar.gz
[root@bogon src]# cd keepalived-1.2.24
[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/
報錯：
configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
系統缺少openssl-devel包所致
安裝openssl-devel
root@bogon keepalived-1.2.24]# yum -y install openssl-devel
再次編輯安裝
[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/
Keepalived configuration
------------------------
Keepalived version : 1.2.24
Compiler : gcc
Preprocessor flags : -I/lib/modules/2.6.32-642.3.1.el6.x86_64//include
Compiler flags : -Wall -Wunused -Wstrict-prototypes
Linker flags :
Extra Lib : -ldl -lssl -lcrypto
Use IPVS Framework : Yes
IPVS use libnl : No
IPVS syncd attributes : No
IPVS 64 bit stats : No
fwmark socket support : Yes
Use VRRP Framework : Yes
Use VRRP VMAC : Yes
Use VRRP authentication : Yes
With ip rules/routes : Yes
SNMP keepalived support : No
SNMP checker support : No
SNMP RFCv2 support : No
SNMP RFCv3 support : No
SHA1 support : No
Use Debug flags : No
Stacktrace support : No
Memory alloc check : No
libnl version : None
Use IPv4 devconf : No
Use libiptc : No
Use libipset : No
Build genhash : Yes
Build documentation : No
[root@bogon keepalived-1.2.24]# make && make install
[root@bogon keepalived-1.2.24]# ln -s /usr/local/sbin/keepalived /sbin/
[root@bogon keepalived-1.2.24]# chkconfig --add keepalived
[root@bogon keepalived-1.2.24]# chkconfig --level 35 keepalived on
4、配置keepalived
lvs-master的配置文件如下
[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs { #全局配置部分
# notification_email { #email 通知，基本不用此處所以注釋掉
# acassen@firewall.loc
# failover@firewall.loc
# sysadmin@firewall.loc
# }
# notification_email_from Alexandre.Cassen@firewall.loc
# smtp_server 192.168.200.1
# smtp_connect_timeout 30
router_id LVS_DEVEL # 設置lvs的id，在一個網絡內應該是唯一的
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 { #vrrp實例定義部分
state MASTER #設置lvs的狀態，報錯MASTER和BACKUP兩種，必須大寫
interface eth1 #設置對外服務的接口
virtual_router_id 60 #設置虛擬路由標示，這個標示是一個數字，同一個vrrp實例使用唯一標示
priority 100 #定義優先級，數字越大優先級越高，在一個vrrp——instance下，master的優先級必須大於backup
advert_int 1 #設定master與backup負載均衡器之間同步檢查的時間間隔，單位是秒
authentication { #設置驗證類型和密碼
auth_type PASS #主要有PASS和AH兩種
auth_pass 1111 #驗證密碼，同一個vrrp_instance下MASTER和BACKUP密碼必須相同
}
virtual_ipaddress { #設置虛擬ip地址，可以設置多個，每行一個
192.168.2.13
}
}
virtual_server 192.168.2.13 80 { #設置虛擬服務器，需要指定虛擬ip和服務端口
delay_loop 3 #健康檢查時間間隔
lb_algo rr #負載均衡調度算法
lb_kind DR #負載均衡轉發規則
persistence_timeout 50 #設置會話保持時間，對動態網頁非常有用
protocol TCP #指定轉發協議類型，有TCP和UDP兩種
real_server 192.168.2.204 80 { #配置服務器節點1，需要指定real server的真實IP地址和端口
weight 1 #設置權重，數字越大權重越高
TCP_CHECK { #realserver的狀態監測設置部分單位秒
connect_timeout 3 #超時時間
nb_get_retry 3 #重試次數
delay_before_retry 3 #重試間隔
connect_port 80 #監測端口
}
}
real_server 192.168.2.205 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
LVS-backup的配置文件如下
[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
# notification_email {
# acassen@firewall.loc
# failover@firewall.loc
# sysadmin@firewall.loc
# }
# notification_email_from Alexandre.Cassen@firewall.loc
# smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 60
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.13
}
}
virtual_server 192.168.2.13 80 {
delay_loop 3
lb_algo rr
lb_kind DR
persistence_timeout 3
protocol TCP
real_server 192.168.2.204 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.2.205 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
5、realserver的配置
兩台web服務器都要執行下面腳本
[root@bogon www]# cat /etc/rc.d/init.d/realserver.sh
#!/bin/bash
# description: Config realserver lo and apply noarp
SNS_VIP=192.168.2.13
/etc/rc.d/init.d/functions
case "$1" in
start)
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
/sbin/route add -host $SNS_VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
ifconfig lo:0 down
route del $SNS_VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
[root@bogon www]# /etc/rc.d/init.d/realserver.sh start
/etc/rc.d/init.d/realserver.sh: line 6: /etc/rc.d/init.d/functions: 權限不夠
RealServer Start OK
[root@bogon www]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:41:71:DF
inet addr:192.168.12.129 Bcast:192.168.12.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe41:71df/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:728 errors:0 dropped:0 overruns:0 frame:0
TX packets:98 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:137311 (134.0 KiB) TX bytes:7369 (7.1 KiB)
eth1 Link encap:Ethernet HWaddr 00:0C:29:41:71:E9
inet addr:192.168.2.204 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe41:71e9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:119838 errors:0 dropped:0 overruns:0 frame:0
TX packets:31612 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:23411786 (22.3 MiB) TX bytes:2119106 (2.0 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:182 (182.0 b) TX bytes:182 (182.0 b)
lo:0 Link encap:Local Loopback
inet addr:192.168.2.13 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:65536 Metric:1
6、啟動keepalived並進行測試
[root@bogon keepalived-1.2.24]# service keepalived start
正在啟動 keepalived： [確定]
lvs-master
[root@bogon keepalived-1.2.24]# tail -f /var/log/messages
Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:46 bogon Keepalived_healthcheckers[6596]: Netlink reflector reports IP 192.168.2.13 added
Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
[root@bogon keepalived-1.2.24]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.2.13:80 rr persistent 3
-> 192.168.2.204:80 Route 1 0 0
-> 192.168.2.205:80 Route 1 0 0
訪問curl http://192.168.2.13/test.txt
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
關掉web2再次測試
[root@www etc]# curl http://192.168.2.13/test.txt
it is web1
[root@www etc]# curl http://192.168.2.13/test.txt
it is web1
查看lvs-master
[root@bogon keepalived-1.2.24]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.2.13:80 rr persistent 3
-> 192.168.2.204:80 Route 1 0 2
[root@bogon keepalived-1.2.24]# tail -f /var/log/messages
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.
Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.
已經自動把web2剔除
打開web2再次查看
[root@bogon keepalived-1.2.24]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.2.13:80 rr persistent 3
-> 192.168.2.204:80 Route 1 0 0
-> 192.168.2.205:80 Route 1 0 0
[root@bogon keepalived-1.2.24]# tail -f /var/log/messages
Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry.
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80
Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.
Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.
Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 success.
Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Adding service [192.168.2.205]:80 to VS [192.168.2.13]:80
Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.
恢復后已自動添加進來
關掉lvs master的keepalived
[root@bogon keepalived-1.2.24]# service keepalived stop
停止 keepalived： [確定]
訪問web並查看lvs backup
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
[root@www etc]# curl http://192.168.2.13/test.txt
it is web2
[root@lys2 src]# tail -f /var/log/messages
Oct 23 19:03:26 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Transition to MASTER STATE
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Entering MASTER STATE
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) setting protocol VIPs.
Oct 23 19:03:27 lys2 Keepalived_healthcheckers[13123]: Netlink reflector reports IP 192.168.2.13 added
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13
[root@lys2 src]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:89:0f:e3 brd ff:ff:ff:ff:ff:ff
inet 192.168.144.101/24 brd 192.168.144.255 scope global eth0
inet6 fe80::20c:29ff:fe89:fe3/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:89:0f:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.2.202/24 brd 192.168.2.255 scope global eth1
inet 192.168.2.13/32 scope global eth1
inet6 fe80::20c:29ff:fe89:fed/64 scope link
valid_lft forever preferred_lft forever
可以看到lvs backup已自動切換成master狀態並自動綁定了vip
查看lvs master vip
[root@bogon keepalived-1.2.24]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:55:4d:7a brd ff:ff:ff:ff:ff:ff
inet 192.168.12.128/24 brd 192.168.12.255 scope global eth0
inet6 fe80::20c:29ff:fe55:4d7a/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:55:4d:84 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.203/24 brd 192.168.2.255 scope global eth1
inet6 fe80::20c:29ff:fe55:4d84/64 scope link
valid_lft forever preferred_lft forever
已自動解除vip
到處全部結束