30.第24章 企業級調度器LVS

一.LVS的四種工作模式

LVS相關術語

1. DS：Director Server。指的是前端負載均衡器節點
2. RS：Real Server。后端真實的工作服務器
3. VIP：向外部直接面向用戶請求，作為用戶請求的目標的IP地址
4. DIP：Director Server IP，主要用於和內部主機通訊的IP地址
5. RIP：Real Server IP，后端服務器的IP地址
6. CIP：Client IP，訪問客戶端的IP地址

1、LVS NAT模式

LVS NAT原理詳解圖

LVS NAT原理：用戶請求LVS到達director，director將請求的報文的目的IP改為RIP，同時將報文的目標端口也改為realserver的相應端口，最后將報文發送到realserver上，realserver將數據返回給director，director再把數據發送給用戶

LVS NAT特性：

• NAT模式修改的是目的ip，直接走的是switch不需要修改mac地址，所以VIP和RIP不需要在同一個網段內
• NAT的包的進出都需要經過LVS，所以LVS可能會成為一個系統的瓶頸問題

2、LVS DR模式

LVS DR原理詳解圖

LVS DR原理：用戶請求LVS到達director，director將請求的報文的目的MAC地址改為后端的realserver的MAC地址，目的IP為VIP(不變)，源IP為client IP地址(不變)，然后director將報文發送到realserver，realserver檢測到目的地址為自己本地的VIP，如果在同一網段，將請求直接返回給用戶，如果用戶跟realserver不在同一個網段，則需要通過網關返回給用戶。

LVS DR特性：

• 前端路由將目標地址為VIP報文統統發給Director Server
• RS跟Director Server必須有一個網卡在同一個物理網絡中
• 所有的請求報文經由Director Server，但響應報文必須不能進過Director Server
• 所有的real server機器上都有VIP地址

3、LVS TUN原理

LVS TUN原理詳解圖

LVS TUN原理：用戶請求LVS到達director，director通過IP-TUN加密技術將請求報文的包封裝到一個新的IP包里面，目的IP為VIP(不變)，然后director將報文發送到realserver，realserver基於IP-TUN解密，然后解析出來包的目的為VIP，檢測網卡是否綁定了VIP，綁定了就處理這個包，如果在同一個網段，將請求直接返回給用戶，否則通過網關返回給用戶；如果沒有綁定VIP就直接丟掉這個包

LVS TUN特性：

• TUNNEL必須在所有的realserver上綁定VIP
• realserver直接把包發給client
• 隧道模式運維起來會比較難，所以一般不用

4、LVS FULLNAT模式

LVS FULLNAT報文變化

LVS FULLNAT特性：

• FULLNAT模式也不需要DIP和RIP在同一網段
• FULLNAT和NAT相比的話：會保證RS的回包一定可到達LVS
• FULLNAT需要更新源IP，所以性能正常比NAT模式下降10%****

5、四種模式的比較

• 是否需要VIP和realserver在同一網段
  DR模式因為只修改包的MAC地址，需要通過ARP廣播找到realserver，所以VIP和realserver必須在同一個網段，也就是說DR模式需要先確認這個IP是否只能掛在這個LVS下面；其他模式因為都會修改目的地址為realserver的IP地址，所以不需要在同一個網段內
• 是否需要在realserver上綁定VIP
  realserver在收到包之后會判斷目的地址是否是自己的IP
  DR模式的目的地址沒有修改，還是VIP，所以需要在realserver上綁定VIP
  IP TUN模式值是對包重新包裝了一層，realserver解析后的包的IP仍然是VIP，所以也需要在realserver上綁定VIP
• 四種模式的性能比較
  DR模式、IP TUN模式都是在包進入的時候經過LVS，在包返回的時候直接返回給client；所以二者的性能比NAT高
  但TUN模式更加復雜，所以性能不如DR
  FULLNAT模式不僅更換目的IP還更換了源IP，所以性能比NAT下降10%
  性能比較：DR>TUN>NAT>FULLNAT

二.LVS的十種調度算法

1.靜態方法

1.1 輪詢調度 rr
均等地對待每一台服務器，不管服務器上的實際連接數和系統負載

1.2 加權輪詢 wrr
調度器可以自動問詢真實服務器的負載情況，並動態調整權值.

1.3 源地址散列調度算法 sh
與目標地址散列調度算法類似，但它是根據源地址散列算法進行靜態分配固定的服務器資源。

1.4 目標地址散列調度算法 dh
該算法是根據目標 IP 地址通過散列函數將目標 IP 與服務器建立映射關系，出現服務器不可用或負載過高的情況下，發往該目標 IP 的請求會固定發給該服務器。

2 動態方法

主要根據每RS當前的負載狀態及調度算法進行調度Overhead=value 較小的RS將被調度

2.1 最少鏈接 lc
動態地將網絡請求調度到已建立的連接數最少的服務器上
如果集群真實的服務器具有相近的系統性能，采用該算法可以較好的實現負載均衡

Overhead=activeconns*256+inactiveconns

2.2 加權最少鏈接 wlc
調度器可以自動問詢真實服務器的負載情況，並動態調整權值
帶權重的誰不干活就給誰分配，機器配置好的權重高

Overhead=(activeconns*256+inactiveconns)/weight

2.3 最少期望延遲 sed
不考慮非活動鏈接，誰的權重大，優先選擇權重大的服務器來接收請求，但權重大的機器會比較忙

Overhead=(activeconns+1)*256/weight

2.4 永不排隊 nq
無需隊列，如果有realserver的連接數為0就直接分配過去

2.5 基於局部性的最少連接調度算法 lblc
這個算法是請求數據包的目標 IP 地址的一種調度算法，該算法先根據請求的目標 IP 地址尋找最近的該目標 IP 地址所有使用的服務器，如果這台服務器依然可用，並且有能力處理該請求，調度器會盡量選擇相同的服務器，否則會繼續選擇其它可行的服務器

2.6 復雜的基於局部性最少的連接算法 lblcr
記錄的不是要給目標 IP 與一台服務器之間的連接記錄，它會維護一個目標 IP 到一組服務器之間的映射關系，防止單點服務器負載過高。

三.LVS-NAT模式案例

[root@centos7 ~]# hostnamectl set-hostname rs1

[root@centos7-2 ~]# hostnamectl set-hostname rs2

[root@rs1 ~]# yum -y install httpd;systemctl enable --now httpd;(hostname -I;hostname) > /var/www/html/index.html

[root@rs2 ~]# yum -y install httpd;systemctl enable --now httpd;(hostname -I;hostname) > /var/www/html/index.html

[root@centos8 ~]# curl 10.0.0.7
10.0.0.7 
rs1
[root@centos8 ~]# curl 10.0.0.17
10.0.0.17 
rs2

[root@rs1 ~]# yum -y install mariadb-server

[root@rs2 ~]# yum -y install mariadb-server

[root@rs1 ~]# cdnet
[root@rs1 network-scripts]# vim ifcfg-eth0
GATEWAY=10.0.0.8
:wq
[root@rs1 network-scripts]# ip route
default via 10.0.0.8 dev eth0 proto static metric 100 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.7 metric 100

[root@rs2 ~]# cdnet
[root@rs2 network-scripts]# vim ifcfg-eth0
GATEWAY=10.0.0.8
:wq
[root@rs2 network-scripts]# systemctl restart network
[root@rs2 network-scripts]# ip route
default via 10.0.0.8 dev eth0 proto static metric 100 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.17 metric 100

[root@centos8 ~]# cdnet
[root@centos8 network-scripts]# ls
ifcfg-eth0
[root@centos8 network-scripts]# cp ifcfg-eth0 ifcfg-eth1
[root@centos8 network-scripts]# vim ifcfg-eth1
DEVICE=eth1
NAME=eth1
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.10.100
PREFIX=24
:wq

[root@centos8 ~]# hostnamectl set-hostname lvs

[root@lvs ~]# reboot
[root@lvs ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:3f:19:17 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.8/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe3f:1917/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:3f:19:21 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.100/24 brd 192.168.10.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe3f:1921/64 scope link 
       valid_lft forever preferred_lft forever

root@ubuntu2004:/etc/netplan# vim 01-netcfg.yaml 
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses: [192.168.10.200/24]
:wq

root@ubuntu2004:/etc/netplan# netplan apply

root@ubuntu2004:~# ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.438 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.445 ms
^C
--- 192.168.10.100 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1017ms
rtt min/avg/max/mdev = 0.438/0.441/0.445/0.003 ms

[root@lvs ~]# ping 10.0.0.7
PING 10.0.0.7 (10.0.0.7) 56(84) bytes of data.
64 bytes from 10.0.0.7: icmp_seq=1 ttl=64 time=0.452 ms
^C
--- 10.0.0.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.452/0.452/0.452/0.000 ms
[root@lvs ~]# ping 10.0.0.17
PING 10.0.0.17 (10.0.0.17) 56(84) bytes of data.
64 bytes from 10.0.0.17: icmp_seq=1 ttl=64 time=0.700 ms
64 bytes from 10.0.0.17: icmp_seq=2 ttl=64 time=0.287 ms
^C
--- 10.0.0.17 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 27ms
rtt min/avg/max/mdev = 0.287/0.493/0.700/0.207 ms
[root@lvs ~]# ping 192.168.10.200
PING 192.168.10.200 (192.168.10.200) 56(84) bytes of data.
64 bytes from 192.168.10.200: icmp_seq=1 ttl=64 time=0.277 ms
^C
--- 192.168.10.200 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.277/0.277/0.277/0.000 ms

[root@lvs ~]# dnf -y install ipvsadm

[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@lvs ~]# ipvsadm -A -t 192.168.10.100:80 -s rr
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.100:80 rr

[root@lvs ~]# ipvsadm -a -t 192.168.10.100:80 -r 10.0.0.7 -m
[root@lvs ~]# ipvsadm -a -t 192.168.10.100:80 -r 10.0.0.17 -m
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.100:80 rr
  -> 10.0.0.7:80                  Masq    1      0          0         
  -> 10.0.0.17:80                 Masq    1      0          0   

root@ubuntu2004:~# curl 192.168.10.100

[root@lvs ~]# tcpdump -i eth1 -nn port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:49.008394 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699730811 ecr 0,nop,wscale 7], length 0
20:49:50.017583 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699731820 ecr 0,nop,wscale 7], length 0
20:49:52.033188 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699733836 ecr 0,nop,wscale 7], length 0
20:49:56.289533 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699738092 ecr 0,nop,wscale 7], length 0
20:50:04.480518 IP 192.168.10.200.47720 > 192.168.10.100.80: Flags [S], seq 1024236654, win 64240, options [mss 1460,sackOK,TS val 3699746284 ecr 0,nop,wscale 7], length 0

[root@lvs ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward=1
:wq

[root@lvs ~]# sysctl -p
net.ipv4.ip_forward = 1

root@ubuntu2004:~# curl 192.168.10.100
10.0.0.17 
rs2
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7 
rs1

[root@rs2 network-scripts]# vim /etc/httpd/conf/httpd.conf
Listen 8080
:wq

[root@rs2 ~]# systemctl restart httpd
[root@rs2 ~]# ss -ntl
State       Recv-Q Send-Q                  Local Address:Port                                 Peer Address:Port              
LISTEN      0      128                                 *:22                                              *:*                  
LISTEN      0      100                         127.0.0.1:25                                              *:*                  
LISTEN      0      128                              [::]:8080                                         [::]:*                  
LISTEN      0      128                              [::]:22                                           [::]:*                  
LISTEN      0      100                             [::1]:25                                           [::]:*    

root@ubuntu2004:~# curl 192.168.10.100
curl: (7) Failed to connect to 192.168.10.100 port 80: Connection refused
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7 
rs1
                                                                                    
[root@lvs ~]# systemctl stop ipvsadm

[root@lvs ~]# ipvsadm-save >ipvsadm.log

[root@lvs ~]# cat ipvsadm.log > /etc/sysconfig/ipvsadm
[root@lvs ~]# cat /etc/sysconfig/ipvsadm
-A -t lvs:http -s rr
-a -t lvs:http -r 10.0.0.7:http -m -w 1
-a -t lvs:http -r 10.0.0.17:http -m -w 1

[root@lvs ~]# vim /etc/sysconfig/ipvsadm
-A -t 190.168.10.100:80 -s rr
-a -t 192.168.10.100:80 -r 10.0.0.7:80 -m -w 1
-a -t 192.168.10.100:80 -r 10.0.0.17:8080 -m -w 1:wq

[root@lvs ~]# systemctl start ipvsadm

[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.100:80 rr
  -> 10.0.0.7:80                  Masq    1      0          0         
  -> 10.0.0.17:8080               Masq    1      0          0

[root@lvs ~]# systemctl enable ipvsadm
Created symlink /etc/systemd/system/multi-user.target.wants/ipvsadm.service → /usr/lib/systemd/system/ipvsadm.service.

[root@lvs ~]# reboot

oot@ubuntu2004:~# curl 192.168.10.100
10.0.0.17 
rs2
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7 
rs1

[root@lvs ~]# ipvsadm -Ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  192.168.10.100:80                   4       26        8     1636      941
  -> 10.0.0.7:80                         2       13        4      818      470
  -> 10.0.0.17:80                        2       13        4      818      471

[root@lvs ~]# systemctl stop ipvsadm
[root@lvs ~]# vim /etc/sysconfig/ipvsadm
-A -t 192.168.10.100:80 -s wrr
-a -t 192.168.10.100:80 -r 10.0.0.7:80 -m -w 3                                                                                 
-a -t 192.168.10.100:80 -r 10.0.0.17:8080 -m -w 1

[root@lvs ~]# systemctl start ipvsadm
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.100:80 wrr
  -> 10.0.0.7:80                  Masq    3      0          1         
  -> 10.0.0.17:8080               Masq    1      0          1        

root@ubuntu2004:~# curl 192.168.10.100
10.0.0.17 
rs2
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7 
rs1
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7 
rs1
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7 
rs1
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.17 
rs2
root@ubuntu2004:~# curl 192.168.10.100
10.0.0.7 
rs1

四.LVS-DR模式單網段案例

root@ubuntu2004:~# vim /etc/netplan/01-netcfg.yaml 

network:                                                                                                              
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses: [192.168.10.123/24]
      gateway4: 192.168.10.200
:wq

root@ubuntu2004:~# reboot

root@ubuntu2004:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:fb:a9:38 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.123/24 brd 192.168.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fefb:a938/64 scope link 
       valid_lft forever preferred_lft forever
root@ubuntu2004:~# ip route
default via 192.168.10.200 dev eth0 proto static 
192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.123

[root@centos7-3 ~]# hostnamectl set-hostname route

[root@route ~]# cdnet
[root@route network-scripts]# vim ifcfg-eth0 
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.0.0.200
PREFIX=24
:wq

[root@route network-scripts]# cp ifcfg-eth0 ifcfg-eth1
[root@route network-scripts]# vim ifcfg-eth1
DEVICE=eth1
NAME=eth1
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.10.200                                                                                                          
PREFIX=24
:wq

[root@route network-scripts]# reboot

[root@route ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6c:54:0a brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.200/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe6c:540a/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6c:54:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.200/24 brd 192.168.10.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe6c:5414/64 scope link 
       valid_lft forever preferred_lft forever

[root@route ~]# sysctl -a |grep ip_forward
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.eth0.stable_secret"
sysctl: reading key "net.ipv6.conf.eth1.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_use_pmtu = 0

[root@route ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
:wq

[root@route ~]# sysctl -p
net.ipv4.ip_forward = 1

[root@rs1 network-scripts]# vim ifcfg-eth0 
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.0.0.7
PREFIX=24
GATEWAY=10.0.0.200                                                                                                             
DNS1=223.5.5.5
DNS2=180.76.76.76
:wq

[root@rs1 network-scripts]# reboot

[root@rs2 network-scripts]# vim ifcfg-eth0 
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.0.0.17
PREFIX=24
GATEWAY=10.0.0.200                                                                                                             
DNS1=223.5.5.5
DNS2=180.76.76.76
:wq

[root@rs2 network-scripts]# reboot

[root@centos8 ~]# dnf -y install ipvsadm

[root@centos8 network-scripts]# vim ifcfg-eth0 
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.0.0.8
PREFIX=24
GATEWAY=10.0.0.200                                                                                                             
DNS1=223.5.5.5
DNS2=180.76.76.76
:wq

[root@centos8 network-scripts]# reboot

[root@rs2 ~]# ss -ntl
State       Recv-Q Send-Q                  Local Address:Port                                 Peer Address:Port              
LISTEN      0      128                                 *:22                                              *:*                  
LISTEN      0      100                         127.0.0.1:25                                              *:*                  
LISTEN      0      128                              [::]:8080                                         [::]:*                  
LISTEN      0      128                              [::]:22                                           [::]:*                  
LISTEN      0      100                             [::1]:25                                           [::]:*                  
[root@rs2 ~]# vim /etc/httpd/conf/httpd.conf
Listen 80  
:wq

[root@rs2 ~]# systemctl restart httpd
[root@rs2 ~]# ss -ntl
State       Recv-Q Send-Q                  Local Address:Port                                 Peer Address:Port              
LISTEN      0      128                                 *:22                                              *:*                  
LISTEN      0      100                         127.0.0.1:25                                              *:*                  
LISTEN      0      128                              [::]:80                                           [::]:*                  
LISTEN      0      128                              [::]:22                                           [::]:*                  
LISTEN      0      100                             [::1]:25                                           [::]:*         

[root@centos8 ~]# hostnamectl set-hostname lvs

[root@lvs ~]# ip route
default via 10.0.0.200 dev eth0 proto static metric 100 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.8 metric 100 
[root@lvs ~]# ping 10.0.0.7
PING 10.0.0.7 (10.0.0.7) 56(84) bytes of data.
64 bytes from 10.0.0.7: icmp_seq=1 ttl=64 time=0.366 ms
^C
--- 10.0.0.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.366/0.366/0.366/0.000 ms
[root@lvs ~]# ping 10.0.0.17
PING 10.0.0.17 (10.0.0.17) 56(84) bytes of data.
64 bytes from 10.0.0.17: icmp_seq=1 ttl=64 time=1.13 ms
^C
--- 10.0.0.17 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.126/1.126/1.126/0.000 ms

root@ubuntu2004:~# ping 10.0.0.7
PING 10.0.0.7 (10.0.0.7) 56(84) bytes of data.
64 bytes from 10.0.0.7: icmp_seq=1 ttl=63 time=0.776 ms
^C
--- 10.0.0.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.776/0.776/0.776/0.000 ms

[root@rs1 ~]# ping 10.0.0.8
PING 10.0.0.8 (10.0.0.8) 56(84) bytes of data.
64 bytes from 10.0.0.8: icmp_seq=1 ttl=64 time=0.280 ms
^C
--- 10.0.0.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.280/0.280/0.280/0.000 ms

[root@rs2 ~]# ping 10.0.0.8
PING 10.0.0.8 (10.0.0.8) 56(84) bytes of data.
64 bytes from 10.0.0.8: icmp_seq=1 ttl=64 time=0.276 ms
^C
--- 10.0.0.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.276/0.276/0.276/0.000 ms

root@ubuntu2004:~# ping 10.0.0.17
PING 10.0.0.17 (10.0.0.17) 56(84) bytes of data.
64 bytes from 10.0.0.17: icmp_seq=1 ttl=63 time=0.756 ms
64 bytes from 10.0.0.17: icmp_seq=2 ttl=63 time=1.32 ms
^C
--- 10.0.0.17 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1031ms
rtt min/avg/max/mdev = 0.756/1.036/1.316/0.280 ms

[root@rs1 ~]# ping 192.168.10.123
PING 192.168.10.123 (192.168.10.123) 56(84) bytes of data.
64 bytes from 192.168.10.123: icmp_seq=1 ttl=63 time=0.947 ms
64 bytes from 192.168.10.123: icmp_seq=2 ttl=63 time=0.647 ms
^C
--- 192.168.10.123 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.647/0.797/0.947/0.150 ms

[root@rs1 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@rs1 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@rs1 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@rs1 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce

[root@rs2 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@rs2 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@rs2 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@rs2 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce

[root@rs1 ~]# ifconfig lo:1 10.0.0.100/32
[root@rs1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/0 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6a:7e:b8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.7/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe6a:7eb8/64 scope link 
       valid_lft forever preferred_lft forever

[root@rs2 network-scripts]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/0 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:be:39:4d brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.17/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:febe:394d/64 scope link 
       valid_lft forever preferred_lft forever

[root@lvs ~]# ifconfig lo:1 10.0.0.100/32
[root@lvs ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/0 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:3f:19:17 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.8/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe3f:1917/64 scope link 
       valid_lft forever preferred_lft forever

[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@lvs ~]# ipvsadm -A -t 10.0.0.100:80 -s rr
[root@lvs ~]# ipvsadm -a -t 10.0.0.100:80 -r 10.0.0.7 -g
[root@lvs ~]# ipvsadm -a -t 10.0.0.100:80 -r 10.0.0.17 -g
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.100:80 rr
  -> 10.0.0.7:80                  Route   1      0          0         
  -> 10.0.0.17:80                 Route   1      0          0  

root@ubuntu2004:~# curl 10.0.0.100
10.0.0.17 
rs2
root@ubuntu2004:~# curl 10.0.0.100
10.0.0.7 
rs1
root@ub

[root@lvs ~]# ip route
default via 10.0.0.200 dev eth0 proto static metric 100 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.8 metric 100 
[root@lvs ~]# ip route del default via 10.0.0.200 dev eth0 proto static metric 100
[root@lvs ~]# ip route
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.8 metric 100

root@ubuntu2004:~# while true;do curl 10.0.0.100;sleep 1;done

[root@lvs ~]# ip route add default via 10.0.0.222 dev eth0 proto static metric 100
[root@lvs ~]# ip route
default via 10.0.0.222 dev eth0 proto static metric 100 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.8 metric 100

root@ubuntu2004:~# while true;do curl 10.0.0.100;sleep 1;done
10.0.0.7 
rs1
10.0.0.17 
rs2

LVS上必須加網關，可以隨便加網關只要跟route是一個網段即可
lo網卡子網掩碼必須是32，不然不能通訊

五.LVS-DR模式多網段案例

[root@route ~]# ip a a 172.16.0.200/24 dev eth0 label eth0:1
[root@route ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6c:54:0a brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.200/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 172.16.0.200/24 scope global eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe6c:540a/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6c:54:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.200/24 brd 192.168.10.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe6c:5414/64 scope link 
       valid_lft forever preferred_lft forever

[root@route ~]# ip route
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.200 metric 100 
172.16.0.0/24 dev eth0 proto kernel scope link src 172.16.0.200 
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.200 metric 101

[root@lvs ~]# reboot

[root@rs1 ~]# reboot

[root@rs2 ~]# reboot

[root@rs1 ~]# ip route
default via 10.0.0.200 dev eth0 proto static metric 100 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.7 metric 100

[root@rs1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6a:7e:b8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.7/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe6a:7eb8/64 scope link 
       valid_lft forever preferred_lft forever

[root@rs2 ~]# ip route
default via 10.0.0.200 dev eth0 proto static metric 100 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.17 metric 100 
[root@rs2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:be:39:4d brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.17/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:febe:394d/64 scope link 
       valid_lft forever preferred_lft forever

[root@rs1 ~]# vim lvs_dr_rs.sh 
#!/bin/bash
#Author:wangxiaochun
#Date:2017-08-13
vip=172.16.0.100
mask='255.255.255.255'
dev=lo:1
#rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
#service httpd start &> /dev/null && echo "The httpd Server is Ready!"
#echo "`hostname -I`" > /var/www/html/index.html

case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ifconfig $dev $vip netmask $mask #broadcast $vip up
    #route add -host $vip dev $dev
    echo "The RS Server is Ready!"
    ;;
stop)
    ifconfig $dev down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "The RS Server is Canceled!"
    ;;
*) 
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac
:wq
[root@rs1 ~]# bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@rs1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 172.16.0.100/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6a:7e:b8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.7/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe6a:7eb8/64 scope link 
       valid_lft forever preferred_lft forever

[root@rs2 ~]# vim lvs_dr_rs.sh 
#!/bin/bash
#Author:wangxiaochun
#Date:2017-08-13
vip=172.16.0.100
mask='255.255.255.255'
dev=lo:1
#rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
#service httpd start &> /dev/null && echo "The httpd Server is Ready!"
#echo "`hostname -I`" > /var/www/html/index.html

case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ifconfig $dev $vip netmask $mask #broadcast $vip up
    #route add -host $vip dev $dev
    echo "The RS Server is Ready!"
    ;;
stop)
    ifconfig $dev down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "The RS Server is Canceled!"
    ;;
*) 
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac
:wq
[root@rs1 ~]# bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@rs1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 172.16.0.100/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:6a:7e:b8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.7/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe6a:7eb8/64 scope link 
       valid_lft forever preferred_lft forever

[root@rs2 ~]# bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@rs2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 172.16.0.100/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:be:39:4d brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.17/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:febe:394d/64 scope link 
       valid_lft forever preferred_lft forever

[root@lvs ~]# vim lvs_dr_vs.sh 
#!/bin/bash
#Author:wangxiaochun
#Date:2017-08-13
vip='172.16.0.100'
iface='lo:1'
mask='255.255.255.255'
port='80'
rs1='10.0.0.7'
rs2='10.0.0.17'
scheduler='wrr'
type='-g'
rpm -q ipvsadm &> /dev/null || yum -y install ipvsadm &> /dev/null

case $1 in
start)
    ifconfig $iface $vip netmask $mask #broadcast $vip up
    iptables -F
 
    ipvsadm -A -t ${vip}:${port} -s $scheduler
    ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
    ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
    echo "The VS Server is Ready!"
    ;;
stop)
    ipvsadm -C
    ifconfig $iface down
    echo "The VS Server is Canceled!"
    ;;
*)
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac
:wq

[root@lvs ~]# bash lvs_dr_vs.sh start
The VS Server is Ready!
[root@lvs ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.0.100:80 wrr
  -> 10.0.0.7:80                  Route   1      0          0         
  -> 10.0.0.17:80                 Route   1      0          2         
[root@lvs ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 172.16.0.100/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:3f:19:17 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.8/24 brd 10.0.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe3f:1917/64 scope link 
       valid_lft forever preferred_lft forever

root@ubuntu2004:~# while true;do curl 172.16.0.100;sleep 1;done
10.0.0.17 
rs2
10.0.0.7 
rs1