HAProxy的四層與七層的區別及透傳IP實戰案例

　　　　　　　　　　HAProxy的四層與七層的區別及透傳IP實戰案例

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正傑

版權聲明：原創作品，謝絕轉載！否則將追究法律責任。

 

 

一.HAProxy在四層的工作原理

　　在四層負載設備中，把client發送的報文目標地址(原來是負載均衡設備的IP地址)，根據均衡設備設置的選擇web服務器的規則選擇對應的web服務器IP地址，這樣client就可以直接跟此服務器建立TCP連接並發送數據。

　　我們其實可以把haproxy在四層的工作流程分為兩個階段:
　　　　第一階段:
　　　　　　 client發送http請求報文到haproxy，haproxy由於工作在第四層，因此通過拆包可以查看到TCP/IP包頭信息，haproxy根據定義的規則將client請求的目標地址修改為后端的web服務器的IP地址，與此同時還將源地址修改為haproxy的IP地址並轉發給后端的web服務器。
　　　　第二階段:
　　　　 　　后端的web服務器將響應的結果封裝成響應報文發送給haproxy服務器后，haproxy再將報文的源地址修改為haproxy的IP地址，目標地址修改為client的IP地址並轉發給client。

 

二.HAProxy在七層的工作原理

　　七層負載均衡服務器起了一個代理服務器的作用，服務器建立一次TCP連接要三次握手，而client要訪問webserver要先與七層負載設備進行三次握手后建立TCP連接，把要訪問的報文信息發送給七層負載均衡；然后七層負載均衡再根據設置的均衡規則選擇特定的webserver，然后通過三次握手與此台webserver建立TCP連接，然后webserver把需要的數據發送給七層負載均衡設備，負載均衡設備再把數據發送給client；所以，七層負載均衡設備起到了代理服務器的作用。

　　我們其實可以把haproxy在七層的工作流程分為四個階段:
　　　　第一階段:
　　　　　　client發送http請求報文到haproxy，haproxy接收到client的http請求報文后，會查看client的請求報文，根據http請求報文的信息在結合haproxy自身定義的規則判斷出client要訪問后端的web服務器;
　　　　第二階段:
　　　　　　haproxy根據client的請求報文判斷要訪問的后端的web服務器后，會代替客戶端的重新發起http請求報文到后端的web服務器;
　　　　第三階段:
　　　　　　后端的web服務器接收到haproxy的http請求報文后，如果訪問的資源存在且權限允許的話會將haproxy請求的資源封裝成響應報文並發送給haproxy服務器;
　　　　第四階段：
　　　　　　haproxy接收到web服務器的響應報文后，haproxy將后端web服務器返回的結果封裝成響應報文並發送給client。

 

三.HAProxy的四層與七層的區別

　　如上圖所示，HAProxy的四層與七層的主要區別就在於是否重新發起了http請求，如果重新發來http請求則說明工作在第七層，若沒有重新發起http請求則說明haproxy工作在第四次。

　　這也就是為什么HAProxy在反向代理MySQL，Redis，RDP等服務時需要特別指定mode為TCP模式的主要原因，因為工作在七層會重新發起http請求，而重新發起的http請求MySQL,Redis,RDP等服務壓根就不認識，因為這些服務都有着自己默認的協議。

 

四.七層(基於http)負載實現IP透傳案例

1>.nginx服務器端配置

[[e-mail] ~]# cat /yinzhengjie/softwares/nginx/conf/nginx.conf　　　　　　　　　　　　　　#編輯nginx的主配置文件
worker_processes  4;
worker_cpu_affinity 00000001 00000010 00000100 00001000; 
 
events {
   worker_connections  100000;
   use epoll;
   accept_mutex on;
   multi_accept on; 
}
  
http {
     include       mime.types;
       
     default_type  text/html;
    
     server_tokens off; 
      
     charset utf-8;
   
     log_format my_access_json '{"@timestamp":"$time_iso8601",' 
                               '"host":"$server_addr",' 
                               '"clientip":"$remote_addr",' 
                               '"size":$body_bytes_sent,' 
                               '"responsetime":$request_time,' 
                               '"upstreamtime":"$upstream_response_time",' 
                               '"upstreamhost":"$upstream_addr",' 
                               '"http_host":"$host",' 
                               '"uri":"$uri",' 
                               '"domain":"$host",' 
                               '"xff":"$http_x_forwarded_for",' 
                               '"referer":"$http_referer",' 
                               '"tcp_xff":"$proxy_protocol_addr",' 
                               '"http_user_agent":"$http_user_agent",' 
                               '"status":"$status"}';
   
    access_log logs/access_json.log my_access_json;
 
    ssl_certificate /yinzhengjie/softwares/nginx/certs/www.yinzhengjie.org.cn.crt;
    ssl_certificate_key /yinzhengjie/softwares/nginx/certs/www.yinzhengjie.org.cn.key;
    ssl_session_cache shared:sslcache:20m;
    ssl_session_timeout 10m;
  
    include /yinzhengjie/softwares/nginx/conf.d/*.conf;
}
[[e-mail] ~]# 
[[e-mail] ~]# nginx -t
nginx: the configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf test is successful
[[e-mail] ~]# 
[[e-mail] ~]# 

[[e-mail] ~]# cat /yinzhengjie/softwares/nginx/conf.d/node101_yinzhengjie_org.cn.conf 　　　　　　#編輯nginx的子配置文件
server {
    listen 80;
    listen 443 ssl;
    server_name node101.yinzhengjie.org.cn;
 
    access_log /yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_access.log my_access_json;
    error_log /yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_error.log;

    location / {
       root /yinzhengjie/data/web/nginx;
       index index.html;
    }

    location = /favicon.ico {
       root /yinzhengjie/data/web/nginx/images/jd;
    }

    location ~ \.php$ {
        #"$document_root"會調用root目錄,若不寫root指令對應的目錄路徑,則"$document_root"所對應的值為空.
        #root /yinzhengjie/data/web/php;
        #指定PHP服務器地址
        fastcgi_pass 172.30.1.106:9000;
        fastcgi_index index.php;
        #fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        #如果"SCRIPT_FILENAME"后面是絕對路徑則可以省略上面的"root /data/nginx/php;"
        fastcgi_param SCRIPT_FILENAME /yinzhengjie/data/web/php$fastcgi_script_name;
        #千萬別忘記添加改行,若不添加"include fastcgi_params;"盡管上面配置的都正確，nginx也無法將fastcgi程序的處理結果返回給瀏覽器,返回給瀏覽器的只能是一個零字節的文件。
        include fastcgi_params;
    }
}
[[e-mail] ~]# 
[[e-mail] ~]# nginx -t
nginx: the configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf test is successful
[[e-mail] ~]# 

[[e-mail] ~]# ss -ntl
State      Recv-Q Send-Q                                         Local Address:Port                                                        Peer Address:Port              
LISTEN     0      128                                                        *:22                                                                     *:*                  
LISTEN     0      128                                                       :::22                                                                    :::*                  
[[e-mail] ~]# 
[[e-mail] ~]# nginx 　　　　　　　　　　#啟動nginx服務
[[e-mail] ~]# 
[[e-mail] ~]# ss -ntl
State      Recv-Q Send-Q                                         Local Address:Port                                                        Peer Address:Port              
LISTEN     0      128                                                        *:80                                                                     *:*                  
LISTEN     0      128                                                        *:22                                                                     *:*                  
LISTEN     0      128                                                        *:443                                                                    *:*                  
LISTEN     0      128                                                       :::22                                                                    :::*                  
[[e-mail] ~]# 

2>.使用瀏覽器直接訪問nginx服務器地址("http://node101.yinzhengjie.org.cn/")

[[e-mail] ~]# >/yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_access.log　　　　　　　　　　#為了便於觀測，我將之前日志清空。
[[e-mail] ~]# 
[[e-mail] ~]# tail -10f /yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_access.log　　　　 #觀察訪問日志
{"@timestamp":"2020-01-03T21:50:46+08:00","host":"172.30.1.101","clientip":"172.30.1.254","size":0,"responsetime":0.000,"upstreamtime":"
-","upstreamhost":"-","http_host":"node101.yinzhengjie.org.cn","uri":"/index.html","domain":"node101.yinzhengjie.org.cn","xff":"-","refe
rer":"-","tcp_xff":"","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.39
45.88 Safari/537.36","status":"304"}
{"@timestamp":"2020-01-03T21:50:46+08:00","host":"172.30.1.101","clientip":"172.30.1.254","size":25214,"responsetime":0.000,"upstreamtim
e":"-","upstreamhost":"-","http_host":"node101.yinzhengjie.org.cn","uri":"/favicon.ico","domain":"node101.yinzhengjie.org.cn","xff":"-",
"referer":"http://node101.yinzhengjie.org.cn/","tcp_xff":"","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537
.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36","status":"200"}

3>.編輯haproxy服務器的配置文件

[[e-mail] ~]# cat /etc/haproxy/haproxy.cfg 　　　　　　　　　　　　　　#使用haproxy配置反向代理nginx
global
maxconn 100000
chroot /yinzhengjie/softwares/haproxy
#如果需要使用動態調度算法需要將socket功能打開
stats socket /yinzhengjie/softwares/haproxy/haproxy.sock mode 600 level admin
user haproxy
group haproxy
daemon
nbproc 2
cpu-map 1 0
cpu-map 2 1
nbthread 2
pidfile /yinzhengjie/softwares/haproxy/haproxy.pid
log 127.0.0.1 local5 info

defaults
option http-keep-alive
option forwardfor
option redispatch
option abortonclose
maxconn 100000
mode http
timeout connect 300000ms
timeout client  300000ms
timeout server  300000ms

listen stats
 mode http
 bind 0.0.0.0:9999
 stats enable
 log global
 stats uri     /haproxy-status
 stats auth    haadmin:q1w2e3r4ys

listen WEB_PORT_80
 bind 172.30.1.102:80 mode http server web01 172.30.1.101:80 check inter 3000 fall 3 rise 5
[[e-mail] ~]# 
[[e-mail] ~]# 
[[e-mail] ~]# systemctl restart haproxy　　　　　　　　　　　　　　　　#別忘記重啟haproxy使配置生效。
[[e-mail] ~]# 

4>.使用瀏覽器直接訪問haproxy服務器地址("http://node102.yinzhengjie.org.cn/")

[[e-mail] ~]# >/yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_access.log
[[e-mail] ~]# 
[[e-mail] ~]# tail -10f /yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_access.log
{"@timestamp":"2020-01-03T22:08:44+08:00","host":"172.30.1.101","clientip":"172.30.1.102","size":0,"responsetime":0.000,"upstreamtime":"-","upstream
host":"-","http_host":"node102.yinzhengjie.org.cn","uri":"/index.html","domain":"node102.yinzhengjie.org.cn","xff":"172.30.1.254","referer":"-","tcp
_xff":"","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36","sta
tus":"304"}

 

五.四層負載實現IP透傳案例

1>.修改nginx的主配置文件(標紅色的參數必須的配置)

[[e-mail] ~]# cat /yinzhengjie/softwares/nginx/conf/nginx.conf 
worker_processes  4;
worker_cpu_affinity 00000001 00000010 00000100 00001000; 
 
events {
   worker_connections  100000;
   use epoll;
   accept_mutex on;
   multi_accept on; 
}
  
http {
     include       mime.types;
       
     default_type  text/html;
    
     server_tokens off; 
      
     charset utf-8;
   
     log_format my_access_json '{"@timestamp":"$time_iso8601",' 
                               '"host":"$server_addr",' 
                               '"clientip":"$remote_addr",' 
                               '"size":$body_bytes_sent,' 
                               '"responsetime":$request_time,' 
                               '"upstreamtime":"$upstream_response_time",' 
                               '"upstreamhost":"$upstream_addr",' 
                               '"http_host":"$host",' 
                               '"uri":"$uri",' 
                               '"domain":"$host",' 
                               '"xff":"$http_x_forwarded_for",' 
                               '"referer":"$http_referer",' 
                             '"tcp_xff":"$proxy_protocol_addr",' 
                               '"http_user_agent":"$http_user_agent",' 
                               '"status":"$status"}';
   
    access_log logs/access_json.log my_access_json;
 
    ssl_certificate /yinzhengjie/softwares/nginx/certs/www.yinzhengjie.org.cn.crt;
    ssl_certificate_key /yinzhengjie/softwares/nginx/certs/www.yinzhengjie.org.cn.key;
    ssl_session_cache shared:sslcache:20m;
    ssl_session_timeout 10m;
  
    include /yinzhengjie/softwares/nginx/conf.d/*.conf;
}
[[e-mail] ~]# 
[[e-mail] ~]# 
[[e-mail] ~]# nginx -t
nginx: the configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf test is successful
[[e-mail] ~]# 

2>.修改nginx的子配置文件(標紅色的參數必須的配置)

[[e-mail] ~]# cat /yinzhengjie/softwares/nginx/conf.d/node101_yinzhengjie_org.cn.conf 
server {
 listen 80 proxy_protocol;
    listen 443 ssl;
    server_name node101.yinzhengjie.org.cn;
 
    access_log /yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_access.log my_access_json;
    error_log /yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_error.log;

    location / {
       root /yinzhengjie/data/web/nginx;
       index index.html;
    }

    location = /favicon.ico {
       root /yinzhengjie/data/web/nginx/images/jd;
    }
}
[[e-mail] ~]# 
[[e-mail] ~]# nginx -t
nginx: the configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /yinzhengjie/softwares/nginx/conf/nginx.conf test is successful
[[e-mail] ~]# 

3>.重新加載nginx的配置文件

[[e-mail] ~]# ps -ef | grep nginx | grep -v grep
root      2825     1  0 21:43 ?        00:00:00 nginx: master process nginx
nginx     2984  2825  0 22:24 ?        00:00:00 nginx: worker process
nginx     2985  2825  0 22:24 ?        00:00:00 nginx: worker process
nginx     2986  2825  0 22:24 ?        00:00:00 nginx: worker process
nginx     2987  2825  0 22:24 ?        00:00:00 nginx: worker process
[[e-mail] ~]# 
[[e-mail] ~]# nginx -s reload
[[e-mail] ~]# 
[[e-mail] ~]# ps -ef | grep nginx | grep -v grep
root      2825     1  0 21:43 ?        00:00:00 nginx: master process nginx
nginx     3025  2825  7 22:26 ?        00:00:00 nginx: worker process
nginx     3026  2825  7 22:26 ?        00:00:00 nginx: worker process
nginx     3027  2825 10 22:26 ?        00:00:00 nginx: worker process
nginx     3028  2825 11 22:26 ?        00:00:00 nginx: worker process
[[e-mail] ~]# 
[[e-mail] ~]# 

4>.編輯haproxy的配置文件(標紅色的參數必須的配置)

[[e-mail] ~]# cat /etc/haproxy/haproxy.cfg 
global
maxconn 100000
chroot /yinzhengjie/softwares/haproxy
#如果需要使用動態調度算法需要將socket功能打開
stats socket /yinzhengjie/softwares/haproxy/haproxy.sock mode 600 level admin
user haproxy
group haproxy
daemon
nbproc 2
cpu-map 1 0
cpu-map 2 1
nbthread 2
pidfile /yinzhengjie/softwares/haproxy/haproxy.pid
log 127.0.0.1 local5 info

defaults
option http-keep-alive
option  forwardfor
option redispatch
option abortonclose
maxconn 100000
mode http
timeout connect 300000ms
timeout client  300000ms
timeout server  300000ms

listen stats
 mode http
 bind 0.0.0.0:9999
 stats enable
 log global
 stats uri     /haproxy-status
 stats auth    haadmin:q1w2e3r4ys

listen WEB_PORT_80
    bind 172.30.1.102:80
  mode tcp
    server web01 172.30.1.101:80  send-proxy check inter 3000 fall 3 rise 5
[[e-mail] ~]# 
[[e-mail] ~]# systemctl restart haproxy　　　　　　　　#別忘記重啟haproxy使配置生效。
[[e-mail] ~]# 

5>.使用瀏覽器直接訪問haproxy服務器地址("http://node102.yinzhengjie.org.cn/")

[[e-mail] ~]# 
[[e-mail] ~]# >/yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_access.log
[[e-mail] ~]# 
[[e-mail] ~]# tail -10f /yinzhengjie/softwares/nginx/logs/node101_yinzhengjie_org_cn_access.log
{"@timestamp":"2020-01-03T22:30:36+08:00","host":"172.30.1.101","clientip":"172.30.1.102","size":0,"responsetime":0.000,"upstreamtime":"-",
"upstreamhost":"-","http_host":"node102.yinzhengjie.org.cn","uri":"/index.html","domain":"node102.yinzhengjie.org.cn","xff":"-","referer":"
-","tcp_xff":"172.30.1.254","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0
.3945.88 Safari/537.36","status":"304"}