
HAProxy是一種免費,非常快速且可靠的解決方案,
可為基於TCP和HTTP的應用程序提供
高可用性,
負載平衡和代理。它特別適合於流量非常高的網站,並為世界上許多訪問量最大的網站提供支持。多年來,它已成為事實上的標准開源負載平衡器,現在隨大多數主流Linux發行版一起提供,並且通常默認情況下部署在雲平台中。由於它不會自行宣傳,因此我們僅在
管理員報告它時才知道它是否使用:-)
它的操作模式使其非常容易且無風險地集成到現有體系結構中,同時仍提供了不將脆弱的Web服務器暴露在網上的可能性,如下所示:


實驗前准備工作(系統centos7.5)
1.開啟轉發功能
echo "net.ipv4.ip_forward = 1" >>/etc/sysctl.conf
sysctl -p
注:開啟監聽本地不存在的IP
echo "net.ipv4.ip_nonlocal_bind = 1" >>/etc/sysctl.conf
2.關閉防火牆
systemctl stop iptables
3.關閉selinux
setenforce 0
sed -i 's/^SELINUX=/SELINUX=disabled/g' /etc/sysconfig/selinux
4.1 yum安裝
# yum install -y haproxy
# haproxy -v
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <
willy@haproxy.org>
yum安裝的版本較舊,如果要使用新版本建議編譯安裝
4.2 編譯安裝
yum -y install gcc gcc-c++ autoconf automake zlib zlib-devel openssl openssl-devel pcre-devel systemd-devel
tar -xvf haproxy-1.8.23.tar.gz
cd haproxy-1.8.23/
make ARCH=x86_64 TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_CPU_AFFINITY=1 PREFIX=/usr/local/haproxy
make install PREFIX=/usr/local/haproxy
cp haproxy /usr/sbin/
注:make時報錯,可以使用make clean清除上次make的殘留文件
創建啟動腳本:
# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxyLoad Balancer
After=syslog.targetnetwork.target
[Service]
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
[Install]
WantedBy=multi-user.target
創建目錄和用戶:
mkdir /etc/haproxy
useradd haproxy -s /sbin/nologin
mkdir /var/lib/haproxy
chown haproxy.haproxy /var/lib/haproxy/ -R
systemctl restart haproxy
cat /etc/haproxy/haproxy.cfg
……
haproxy.cfg文件中定義了chroot、pidfile、user、group等參數,如果系統沒有相應的資源會導致haproxy無法啟動,具體參考日志文件/var/log/messages
啟動HAProxy:
初始配置文件
# find ./* -name '*.cfg'
./examples/acl-content-sw.cfg
./examples/auth.cfg
./examples/content-sw-sample.cfg
./examples/option-http_proxy.cfg
./examples/ssl.cfg
./examples/transparent_proxy.cfg
./examples/wurfl-example.cfg
# cp ./examples/option-http_proxy.cfg /etc/haproxy/haproxy.cfg
systemctl enable haproxy
systemctl restart haproxy
初始配置文件如下:
# cat /etc/haproxy/haproxy.cfg
#
# demo config for Proxy mode
#
global
maxconn 20000
ulimit-n 16384
log 127.0.0.1 local0
uid 200
gid 200
chroot /var/empty
nbproc 4
daemon
frontend test-proxy
bind 0.0.0.0:8080
mode http
log global
option httplog
option dontlognull
option nolinger
option http_proxy
maxconn 8000
timeout client 30s
# layer3: Valid users
acl allow_host src 192.168.200.150/32
http-request deny if !allow_host
# layer7: prevent private network relaying
acl forbidden_dst url_ip 192.168.0.0/24
acl forbidden_dst url_ip 172.16.0.0/12
acl forbidden_dst url_ip 10.0.0.0/8
http-request deny if forbidden_dst
default_backend test-proxy-srv
backend test-proxy-srv
mode http
timeout connect 5s
timeout server 5s
retries 2
option nolinger
option http_proxy
# layer7: Only GET method is valid
acl valid_method method GET
http-request deny if !valid_method
# layer7: protect bad reply
http-response deny if { res.hdr(content-type) audio/mp3 }
查看haproxy正常運行后的狀態
# systemctl status haproxy
● haproxy.service - HAProxyLoad Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-01-08 15:20:10 CST; 6s ago
Process: 5563 ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q (code=exited, status=0/SUCCESS)
Main PID: 5564 (haproxy)
CGroup: /system.slice/haproxy.service
├─5564 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─5566 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─5567 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─5568 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
└─5569 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
Jan 08 15:20:10 nova-create-vm1.novalocal systemd[1]: haproxy.service: main process exited, code=exited, status=143/n/a
Jan 08 15:20:10 nova-create-vm1.novalocal systemd[1]: Stopped HAProxyLoad Balancer.
Jan 08 15:20:10 nova-create-vm1.novalocal systemd[1]: Unit haproxy.service entered failed state.
Jan 08 15:20:10 nova-create-vm1.novalocal systemd[1]: haproxy.service failed.
Jan 08 15:20:10 nova-create-vm1.novalocal systemd[1]: Starting HAProxyLoad Balancer...
Jan 08 15:20:10 nova-create-vm1.novalocal systemd[1]: Started HAProxyLoad Balancer.
# netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1038/master
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 5566/haproxy
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1070/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1038/master
tcp6 0 0 :::22 :::* LISTEN 1070/sshd
查看haproxy的編譯參數
# haproxy -vv
HA-Proxy version 1.8.23 2019/11/25
Copyright 2000-2019 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
haproxy.cfg參數詳解
HAProxy組成
程序環境:
主程序:/usr/sbin/haproxy
配置文件:/etc/haproxy/haproxy.cfg
Unit file:/usr/lib/systemd/system/haproxy.service
配置段:
global:全局配置段
進程及安全配置相關的參數
性能調整相關參數
Debug參數
proxies:代理配置段
defaults:為frontend, backend, listen提供默認配置
frontend:前端,相當於nginx中的server {}
backend:后端,相當於nginx中的upstream {}
listen:同時擁有前端和后端配置
Haproxy配置-global
global配置參數:
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3
chroot#鎖定運行目錄
deamon#以守護進程運行
#stats socket /var/lib/haproxy/haproxy.sockmode 600 level admin #socket文件
user, group, uid, gid#運行haproxy的用戶身份
nbproc#開啟的haproxy進程數,與CPU保持一致
nbthread#指定每個haproxy進程開啟的線程數,默認為每個進程一個線程
cpu-map 1 0 #綁定haproxy進程至指定CPU
maxconn#每個haproxy進程的最大並發連接數
maxsslconn#SSL每個haproxy進程ssl最大連接數
maxconnrate#每個進程每秒最大連接數
spread-checks #后端server狀態check隨機提前或延遲百分比時間,建議2-5(20%-50%)之間
pidfile#指定pid文件路徑
log 127.0.0.1 local3 info #定義全局的syslog服務器;最多可以定義兩個
HAProxy Proxies配置
defaults [<name>] #默認配置項,針對以下的frontend、backend和lsiten生效,可以多個name
frontend <name> #前端servername,類似於Nginx的一個虛擬主機server。
backend <name> #后端服務器組,等於nginx的upstream
listen <name> #將frontend和backend合並在一起配置
注:name字段只能使用”-”、”_”、”.”、和”:”,並且嚴格區分大小寫,例如:Web和web是完全不同的兩組服務器。
Proxies配置-defaults
defaults 配置參數:
option redispatch #當server Id對應的服務器掛掉后,強制定向到其他健康的服務器
option abortonclose #當服務器負載很高的時候,自動結束掉當前隊列處理比較久的鏈接
option http-keep-alive 60#開啟會話保持
option forwardfor #開啟IP透傳
mode http #默認工作類型
timeout connect 120s #轉發客戶端請求到后端server的最長連接時間(TCP之前)
timeout server 600s #轉發客戶端請求到后端服務端的超時超時時長(TCP之后)
timeout client 600s #與客戶端的最長空閑時間
timeout http-keep-alive 120s #session 會話保持超時時間,范圍內會轉發到相同的后端服務器
#timeout check 5s #對后端服務器的檢測超時時間
Proxies配置-frontend配置參數
bind:指定HAProxy的監聽地址,可以是IPV4或IPV6,可以同時監聽多個IP或端口,可同時用於listen字段中
bind [<address>]:<port_range> [, ...] [param*]
mode http/tcp #指定負載協議類型
use_backend backend_name #調用的后端服務器組名稱
示例:
frontend WEB_PORT
bind :80,:8080
bind 192.168.7.102:10080,192.168.7.102:10043
use_backend backend_name
Proxies配置-backend配置參數
mode http/tcp #指定負載協議類型
option #配置選項
server #定義后端realserver
注意:option后面加httpchk,smtpchk, mysql-check, pgsql-check,ssl-hello-chk方法,可用於實現更多應用層檢測功能。
后端服務器狀態監測及相關配置
check #對指定real進行健康狀態檢查,默認不開啟
addr IP#可指定的健康狀態監測IP
port num#指定的健康狀態監測端口
inter num#健康狀態檢查間隔時間,默認2000 ms
fall num#后端服務器失效檢查次數,默認為3
rise num#后端服務器從下線恢復檢查次數,默認為2
weight #默認為1,最大值為256,0表示不參與負載均衡
backup #將后端服務器標記為備份狀態
disabled #將后端服務器標記為不可用狀態
redirect prefix http://www.magedu.com/#將請求臨時重定向至其它URL,只適用於http模式
maxconn <maxconn>:當前后端server的最大並發連接數
backlog <backlog>:當server的連接數達到上限后的后援隊列長度
案例環境
centos7-vm1 172.16.99.117 192.168.9.109 web1
centos7-vm2 172.16.99.114 192.168.9.102 web2
nova-create-vm1 172.16.99.131 192.168.9.211 nginx proxy #
frontend/ backend 配置案例
[root@nova-create-vm1 ~]# cat /etc/haproxy/haproxy.cfg
global
maxconn 65535
ulimit-n 131111
log 127.0.0.1 local3 info
user haproxy
group haproxy
chroot /var/lib/haproxy
nbproc 6
nbthread 3
daemon
defaults
option http-keep-alive
maxconn 65536
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
listen admin_stats
bind 0.0.0.0:1080
mode http
maxconn 10
stats enable
stats refresh 30s
stats uri /haproxy_status
stats auth admin:admin
stats hide-version
frontend WEB_PORT_80
bind 0.0.0.0:80
mode http
use_backend web_port_http_nodes
backend web_port_http_nodes
mode http
option forwardfor
server centos7-vm1 192.168.9.109:80 check inter 3000 fall 3 rise 5
server centos7-vm2 192.168.9.102:80 check inter 3000 fall 3 rise 5
frontend MYSQL_PORT_3306
bind 0.0.0.0:80
mode tcp
use_backend mysql_port_nodes
backend mysql_port_nodes
mode tcp
server centos7-vm1_mysql-node1 172.16.99.117:3306 check inter 3000 fall 3 rise 5
server centos7-vm2_mysql-node2 172.16.99.114:3306 check inter 3000 fall 3 rise 5
返回結果
# for i in `seq 10` ;do curl -L http://172.16.99.131/ ;done
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
Proxies配置-listen
使用listen替換frontend和backend的配置方式:
使用http模式
listen WEB_PORT_80
bind 0.0.0.0:80
mode http
option forwardfor
server centos7-vm1 192.168.9.109:80 check inter 3000 fall 3 rise 5
server centos7-vm2 192.168.9.102:80 check inter 3000 fall 3 rise 5
或者使用TCP模式
listen WEB_PORT_80
bind 0.0.0.0:80
mode tcp
server centos7-vm1 192.168.9.109:80 check inter 3000 fall 3 rise 5
server centos7-vm2 192.168.9.102:80 check inter 3000 fall 3 rise 5
listen MYSQL_PORT_3306
bind 0.0.0.0:3306
mode tcp
server centos7-vm1_mysql-node1 172.16.99.117:3306 check inter 3000 fall 3 rise 5
server centos7-vm2_mysql-node2 172.16.99.114:3306 check inter 3000 fall 3 rise 5
返回結果
web頁面測試結果
# for i in `seq 10` ;do curl -L http://172.16.99.131/ ;done
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
centos7-vm2 172.16.99.114 192.168.9.102 web2
centos7-vm1 172.16.99.117 192.168.9.109 web1
使用TCP模式反向代理mysql集群的測試結果
# for i in `seq 10`;do mysql -h172.16.99.131 -uroot -p'123456' -e "show status like 'wsrep_gcomm_%';" | grep wsrep ;done
wsrep_gcomm_uuid 8be166cf-31d6-11ea-a4cf-7f05a956f41d
wsrep_gcomm_uuid 1a346279-31d6-11ea-b372-3a1907be8814
wsrep_gcomm_uuid 8be166cf-31d6-11ea-a4cf-7f05a956f41d
wsrep_gcomm_uuid 1a346279-31d6-11ea-b372-3a1907be8814
wsrep_gcomm_uuid 8be166cf-31d6-11ea-a4cf-7f05a956f41d
wsrep_gcomm_uuid 1a346279-31d6-11ea-b372-3a1907be8814
wsrep_gcomm_uuid 8be166cf-31d6-11ea-a4cf-7f05a956f41d
wsrep_gcomm_uuid 1a346279-31d6-11ea-b372-3a1907be8814
wsrep_gcomm_uuid 8be166cf-31d6-11ea-a4cf-7f05a956f41d
wsrep_gcomm_uuid 1a346279-31d6-11ea-b372-3a1907be8814
注:上面TCP案例的完整配置
global
maxconn 65535
ulimit-n 131111
log 127.0.0.1 local3 info
user haproxy
group haproxy
chroot /var/lib/haproxy
nbproc 6
nbthread 3
daemon
defaults
option http-keep-alive
maxconn 65536
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
listen admin_stats
bind 0.0.0.0:1080
mode http
maxconn 10
stats enable
stats refresh 30s
stats uri /haproxy_status
stats auth admin:admin
stats hide-version
listen WEB_PORT_80
bind 172.16.99.131:80
mode tcp
server centos7-vm1 192.168.9.109:80 check inter 3000 fall 3 rise 5
server centos7-vm2 192.168.9.102:80 check inter 3000 fall 3 rise 5
listen MYSQL_PORT_3306
bind 172.16.99.131:3306
mode tcp
server centos7-vm1_mysql-node1 172.16.99.117:3306 check inter 3000 fall 3 rise 5
server centos7-vm2_mysql-node2 172.16.99.114:3306 check inter 3000 fall 3 rise 5
配置支持多進程多線程
haproxy.cfg的global配置下添加或修改
nbproc 6
nbthread 3
#systemctl reload haproxy
# ps -ef | grep haproxy
root 1999 1 0 14:30 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
haproxy 2046 1999 0 14:32 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
haproxy 2047 1999 0 14:32 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
haproxy 2048 1999 0 14:32 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
haproxy 2049 1999 0 14:32 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
haproxy 2050 1999 0 14:32 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
haproxy 2051 1999 0 14:32 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
# # pstree -p 1999
haproxy(1999)─┬─haproxy(2046)─┬─{haproxy}(2058)
│ └─{haproxy}(2059)
├─haproxy(2047)─┬─{haproxy}(2060)
│ └─{haproxy}(2061)
├─haproxy(2048)─┬─{haproxy}(2056)
│ └─{haproxy}(2057)
├─haproxy(2049)─┬─{haproxy}(2064)
│ └─{haproxy}(2065)
├─haproxy(2050)─┬─{haproxy}(2062)
│ └─{haproxy}(2063)
└─haproxy(2051)─┬─{haproxy}(2054)
└─{haproxy}(2055)
● haproxy.service - HAProxyLoad Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2020-01-10 14:30:40 CST; 4min 40s ago
Process: 2045 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
Process: 1998 ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q (code=exited, status=0/SUCCESS)
Main PID: 1999 (haproxy)
CGroup: /system.slice/haproxy.service
├─1999 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
├─2046 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
├─2047 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
├─2048 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
├─2049 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
├─2050 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
└─2051 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 2024 2025 2026 2027
Jan 10 14:31:33 nova-create-vm1.novalocal haproxy[1999]: [WARNING] 009/143133 (1999) : Former worker 2006 exited with code 0
Jan 10 14:31:33 nova-create-vm1.novalocal haproxy[1999]: [WARNING] 009/143133 (1999) : Former worker 2003 exited with code 0
Jan 10 14:32:57 nova-create-vm1.novalocal systemd[1]: Reloading HAProxyLoad Balancer.
Jan 10 14:32:57 nova-create-vm1.novalocal systemd[1]: Reloaded HAProxyLoad Balancer.
Jan 10 14:32:57 nova-create-vm1.novalocal haproxy[1999]: [WARNING] 009/143133 (1999) : Reexecuting Master process
檢查haproxy配置語法是否正確
# haproxy -f /etc/haproxy/haproxy.cfg -c
Configuration file is valid
啟動haproxy服務
systemctl start haproxy
systemctl enable haproxy
#下面的配置為haproxy web監控,可以不需要
listen admin_stats
bind 0.0.0.0:1080
mode http
maxconn 10
stats enable
stats refresh 30s
stats uri /haproxy_status
stats auth admin:admin
stats hide-version
