一 原理簡介
1.HAProxy
HAProxy提供高可用性、負載均衡以及基於TCP和HTTP應用的代理，支持虛擬主機，它是免費、快速並且可靠的一種解決方案。HAProxy特別適用於那些負載特大的web站點，這些站點通常又需要會話保持或七層處理。HAProxy運行在時下的硬件上，完全可以支持數以萬計的並發連接。並且它的運行模式使得它可以很簡單安全的整合進當前的架構中， 同時可以保護web服務器不被暴露到網絡上。
2.Keepalived
Keepalived 是一個基於VRRP協議來實現的LVS服務高可用方案，可以利用其來避免單點故障。一個LVS服務會有2台服務器運行Keepalived，一台為主服務器（MASTER），一台為備份服務器（BACKUP），但是對外表現為一個虛擬IP，主服務器會發送特定的消息給備份服務器，當備份服務器收不到這個消息的時候，即主服務器宕機的時候，備份服務器就會接管虛擬IP，繼續提供服務，從而保證了高可用性。Keepalived是VRRP的完美實現。
3.vrrp協議
在現實的網絡環境中，兩台需要通信的主機大多數情況下並沒有直接的物理連接。對於這樣的情況，它們之間路由怎樣選擇？主機如何選定到達目的主機的下一跳路由，這個問題通常的解決方法有兩種：
在主機上使用動態路由協議(RIP、OSPF等)
在主機上配置靜態路由
很明顯，在主機上配置動態路由是非常不切實際的，因為管理、維護成本以及是否支持等諸多問題。配置靜態路由就變得十分流行，但路由器(或者說默認網關default gateway)卻經常成為單點故障。VRRP的目的就是為了解決靜態路由單點故障問題，VRRP通過一競選(election)協議來動態的將路由任務交給LAN中虛擬路由器中的某台VRRP路由器。
二 運行環境
1.系統
CentOS Linux release 7.2.1511 (Core)
2.應用軟件
haproxy-1.5.14-3.el7.x86_64
keepalived-1.2.13-7.el7.x86_64
httpd-2.4.6-45.el7.centos.x86_64
php-5.4.16-42.el7.x86_64
mariadb-server-5.5.52-1.el7.x86_64
mariadb.x86_64
wordpress-4.3.1-zh_CN.zip
nfs-utils-1.3.0-0.33.el7.x86_64
rpcbind-0.2.0-38.el7.x86_64
3.IP配置
負載均衡器
DIP1:172.18.67.13
DIP2:172.18.67.14
后端Real Server
RIP1:172.18.67.11
RIP2:172.18.67.12
數據庫服務器
MIP:172.18.67.1
客戶端IP
IP:172.18.67.3
VIP:172,.18.67.33
三 架構拓撲及應用軟件安裝
1.拓撲圖
2.部署應用軟件
在IP為172.18.67.13與172.18.67.14的服務器上安裝部署haproxy、keepalived
[root@inode2 ~]# yum install haproxy keepalived -y [root@inode3 ~]# yum install haproxy keepalived -y
在IP為172.18.67.11與172.18.67.12的服務器上安裝部署httpd、php
[root@inode4 ~]# yum install httpd php -y [root@inode5 ~]# yum install httpd php -y
在IP為172.18.67.1的服務器上部署mariadb、mariadb-server、php-mysql
[root@inode6 ~]# yum install mariadb mariadb-server php-mysql -y
另外由於數據庫采用文件共享的方式，所以還要安裝nfs，分別在Real Server端和數據庫服務器端安裝nfs的應用軟件nfs-utils、rpcbind
[root@inode4 ~]# yum install nfs-utils rpcbind -y [root@inode5 ~]# yum install nfs-utils rpcbind -y [root@inode6 ~]# yum install nfs-utils rpcbind -y
四 配置
1.Real Server配置
在這里我們將IP為172.18.67.11和172.18.67.12的服務器80端口設置為動態資源站，將IP為172.18.67.11和172.18.67.12的8080端口服務器模擬另兩台服務器設置成靜態資源站：將wordpress應用分別解壓至/var/www/html/下，並修改該目錄的屬主和屬組
[root@inode4 ~]# unzip wordpress-4.3.1-zh_CN.zip -C /var/www/html/ [root@inode4 ~]# chown -R apache:apache /var/www/html/wordpress [root@inode5 ~]# unzip wordpress-4.3.1-zh_CN.zip -C /var/www/html/ [root@inode5 ~]# chown -R apache:apache /var/www/html/wordpress
修改后端Server的httpd配置文件將網站目錄從默認的/var/www/html修改為/var/www/html/wordpress。
2.nfs配置
數據庫端修改配置文件
[root@inode6 ~]# vim /etc/exports /data/ 172.18.67.11(rw,async) /data/ 172.18.67.12(rw,async)
修改保存完畢后啟動Real Server和數據庫服務器的nfs應用
[root@inode4 ~]# systemctl start rpcbind [root@inode4 ~]# systemctl start nfs [root@inode5 ~]# systemctl start rpcbind [root@inode5 ~]# systemctl start nfs [root@inode6 ~]# systemctl start rpcbind [root@inode6 ~]# systemctl start nfs
創建數據存放目錄及修改權
[root@inode6 ~]# mkdir /data [root@inode6 ~]# chown -R mysql:mysql /data
修改數據庫配置文件數據存儲目錄
[root@inode6 ~]# vim /etc/my.cnf datadir=/data/
啟動數據庫
[root@inode6 ~]# systemctl start mariadb-server
將數據庫目錄掛載至web動態資源服務器
[root@inode4 ~]# mount -t nfs 172.18.67.1:/data/ /mnt [root@inode5 ~]# mount -t nfs 172.18.67.1:/data/ /mnt
3.keepalived配置
MASTER
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id inode2
vrrp_macst_group4 224.0.67.67
}
vrrp_instance http {
state MASTER
interface eno16777736
virtual_router_id 67
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass lKZvQVv9
}
virtual_ipaddress {
172.18.67.33/16 dev eno16777736
}
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
BACKUP
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id inode2
vrrp_macst_group4 224.0.67.67
}
vrrp_instance http {
state BACKUP
interface eno16777736
virtual_router_id 67
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass lKZvQVv9
}
virtual_ipaddress {
172.18.67.33/16 dev eno16777736
}
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
通知腳本
#!/bin/bash
#
contact='root@localhost'
notify() {
mailsubject="$(hostname) to be $1, vip floating"
mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
;;
esac
4.haproxy配置
兩節點的配置內容是一樣的，如下：
[root@inode2 haproxy]# vim haproxy.cfg global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend web *:80 mode http maxconn 2000 acl url_static path_beg -i /static /images /javascript /stylesheets acl url_static path_end -i .jpg .gif .png .css .js .html .txt .htm use_backend staticsrvs if url_static default_backend appsrvs backend staticsrvs balance roundrobin server stcsrvs1 172.18.67.11:8080 check server stcsrvs2 172.18.67.12:8080 check backend appsrvs balance roundrobin server wp1 172.18.67.11:80 check server wp2 172.18.67.12:80 check listen stats bind :10086 stats enable stats uri /admin?stats stats auth admin:admin stats admin if TRUE
五 啟動服務並測試
1.啟動haproxy和keepalived
[root@inode2 ~]# systemctl restart haproxy [root@inode2 ~]# systemctl restart keepalived [root@inode3 ~]# systemctl restart haproxy [root@inode3 ~]# systemctl restart keepalived
2.測試
inode2：
[root@inode2 ~]# systemctl status -l keepalived ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled) Active: active (running) since Wed 2017-05-17 23:49:45 CST; 6s ago Process: 28940 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 28941 (keepalived) CGroup: /system.slice/keepalived.service ├─28941 /usr/sbin/keepalived -D ├─28942 /usr/sbin/keepalived -D └─28943 /usr/sbin/keepalived -D May 17 23:49:45 inode2 Keepalived_vrrp[28943]: Registering gratuitous ARP shared channel May 17 23:49:45 inode2 Keepalived_vrrp[28943]: Opening file '/etc/keepalived/keepalived.conf'. May 17 23:49:45 inode2 Keepalived_vrrp[28943]: Configuration is using : 63025 Bytes May 17 23:49:45 inode2 Keepalived_vrrp[28943]: Using LinkWatch kernel netlink reflector... May 17 23:49:45 inode2 Keepalived_vrrp[28943]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] May 17 23:49:46 inode2 Keepalived_vrrp[28943]: VRRP_Instance(http) Transition to MASTER STATE May 17 23:49:47 inode2 Keepalived_vrrp[28943]: VRRP_Instance(http) Entering MASTER STATE May 17 23:49:47 inode2 Keepalived_vrrp[28943]: VRRP_Instance(http) setting protocol VIPs. May 17 23:49:47 inode2 Keepalived_healthcheckers[28942]: Netlink reflector reports IP 172.18.67.33 added May 17 23:49:47 inode2 Keepalived_vrrp[28943]: VRRP_Instance(http) Sending gratuitous ARPs on eno16777736 for 172.18.67.33
我們看到inode2節點進入了MASTER模式，再查看inode3的狀態
[root@inode3 ~]# systemctl start keepalived [root@inode3 ~]# systemctl status -l keepalived ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled) Active: active (running) since Wed 2017-05-17 23:51:08 CST; 5s ago Process: 42610 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 42611 (keepalived) CGroup: /system.slice/keepalived.service ├─42611 /usr/sbin/keepalived -D ├─42612 /usr/sbin/keepalived -D └─42613 /usr/sbin/keepalived -D May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Netlink reflector reports IP fe80::20c:29ff:fe78:24c3 added May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Registering Kernel netlink reflector May 17 23:51:08 inode3 Keepalived_healthcheckers[42612]: Using LinkWatch kernel netlink reflector... May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Registering Kernel netlink command channel May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Registering gratuitous ARP shared channel May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Opening file '/etc/keepalived/keepalived.conf'. May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Configuration is using : 63023 Bytes May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Using LinkWatch kernel netlink reflector... May 17 23:51:08 inode3 Keepalived_vrrp[42613]: VRRP_Instance(http) Entering BACKUP STATE May 17 23:51:08 inode3 Keepalived_vrrp[42613]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
不難看出inode3節點處於BACKUP狀態，此時我們將inode2的keepalived服務停掉
[root@inode2 ~]# systemctl stop keepalived [root@inode3 ~]# systemctl status -l keepalived ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled) Active: active (running) since Wed 2017-05-17 23:51:08 CST; 1min 2s ago Process: 42610 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 42611 (keepalived) CGroup: /system.slice/keepalived.service ├─42611 /usr/sbin/keepalived -D ├─42612 /usr/sbin/keepalived -D └─42613 /usr/sbin/keepalived -D May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Opening file '/etc/keepalived/keepalived.conf'. May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Configuration is using : 63023 Bytes May 17 23:51:08 inode3 Keepalived_vrrp[42613]: Using LinkWatch kernel netlink reflector... May 17 23:51:08 inode3 Keepalived_vrrp[42613]: VRRP_Instance(http) Entering BACKUP STATE May 17 23:51:08 inode3 Keepalived_vrrp[42613]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] May 17 23:52:07 inode3 Keepalived_vrrp[42613]: VRRP_Instance(http) Transition to MASTER STATE May 17 23:52:08 inode3 Keepalived_vrrp[42613]: VRRP_Instance(http) Entering MASTER STATE May 17 23:52:08 inode3 Keepalived_vrrp[42613]: VRRP_Instance(http) setting protocol VIPs. May 17 23:52:08 inode3 Keepalived_healthcheckers[42612]: Netlink reflector reports IP 172.18.67.33 added May 17 23:52:08 inode3 Keepalived_vrrp[42613]: VRRP_Instance(http) Sending gratuitous ARPs on eno16777736 for 172.18.67.33
我們發現inode3節點進入了MASTER狀態，因此體現出了高可用的特性
接下來我們測試haproxy的特性，在haproxy的配置文件里有下面這一段
listen stats bind :10086 stats enable stats uri /admin?stats stats auth admin:admin stats admin if TRUE
這段配置可以讓我們在瀏覽器中查看和修改haproxy統計接口啟用相關的參數，在瀏覽器中輸入http://172.18.67.33:10086/admin?stats，就會出現下面這種狀態，輸入賬號和密碼，就進入了haproxy相關參數配置頁面。
在下圖中可以看出負載均衡的兩台web服務器一台負責動態資源解析，另一台負責靜態資源。動態資源的數據存放於后端的nfs服務器上。
接下來我們在瀏覽器中訪問http://172.18.67.33就可以安裝wordpress了。至此一個簡單的高可用負載均衡服務搭建完畢。
