K8S網絡模型
解決容器在集群之間的互通。
K8S設計了網絡模型，但卻將它的實現交給了網絡插件，CNI網絡插件最主要的功能就是實現POD資源能夠跨宿主機進行通信。
常見的CNI網絡插件
- Flannel(重點)：依賴於etcd去存儲網絡信息的
- Calico：網絡限制，網絡規則的內容
- Canal：把上面兩個結合了，前半部分用F，后半部分用C（沒有啥用）
- Contiv：思科開源的
- OpenContrail
- NSX-T
- kube-router：試圖取代kube-proxy
集群規划
| 主機名 | 角色 | IP |
|---|---|---|
| HDSS7-21.host.com | flannel | 10.4.7.21 |
| HDSS7-22.host.com | flannel | 10.4.7.22 |
下載軟件，解壓，做軟鏈接
在HDSS7-21.host.com上
[root@hdss7-21 src]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@hdss7-21 src]# mkdir /opt/flannel-v0.11.0
[root@hdss7-21 src]# tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0
[root@hdss7-21 src]# ln -s /opt/flannel-v0.11.0/ /opt/flannel
做軟鏈接，方便以后版本升級，只需要移除舊的軟鏈接，創建新的軟連接即可。
[root@hdss7-21 opt]# cd flannel
[root@hdss7-21 flannel]# ll
total 34436
-rwxr-xr-x 1 root root 35249016 Jan 29 2019 flannel
-rwxr-xr-x 1 root root 2139 Oct 23 2018 mk-docker-opts.sh
-rw-r--r-- 1 root root 4300 Oct 23 2018 README.md
[root@hdss7-21 flannel]# mkdir cert
[root@hdss7-21 flannel]# scp hdss7-200:/opt/certs/ca.pem .
[root@hdss7-21 flannel]# scp hdss7-200:/opt/certs/client.pem .
[root@hdss7-21 flannel]# scp hdss7-200:/opt/certs/client-key.pem .
創建env配置文件
[root@hdss7-21 cert]# cd ..
[root@hdss7-21 flannel]# vim subnet.env
FLANNEL_NETWORK=172.7.0.0/16
FLANNEL_SUBNET=172.7.21.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false
創建啟動腳本
在HDSS7-21.host.com上
[root@hdss7-21 flannel]# vim flanneld.sh
#!/bin/sh
./flanneld \
--public-ip=10.4.7.21 \
--etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
--etcd-keyfile=./cert/client-key.pem \
--etcd-certfile=./cert/client.pem \
--etcd-cafile=./cert/ca.pem \
--iface=eth0 \
--subnet-file=./subnet.env \
--healthz-port=2401
授權
[root@hdss7-21 flannel]# chmod +x flanneld.sh
創建日志目錄
[root@hdss7-21 flannel]# mkdir -p /data/logs/flanneld
創建supervisor配置
在HDSS7-21.host.com上
[root@hdss7-21 flannel]# vim /etc/supervisord.d/flannel.ini
[program:flanneld-7-21]
command=/opt/flannel/flanneld.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/flannel ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=root ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/flanneld/flanneld.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
操作etcd，增加host-gw
[root@hdss7-21 flannel]# cd /opt/etcd/
[root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}
[root@hdss7-21 etcd]# ./etcdctl get /coreos.com/network/config
{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}
查看誰是master
[root@hdss7-21 etcd]# ./etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.22:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.21:2379 isLeader=true
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false
啟動服務並檢查
[root@hdss7-21 etcd]# supervisorctl status
etcd-server-7-21 RUNNING pid 1029, uptime 4 days, 0:41:33
kube-apiserver-7-21 RUNNING pid 1030, uptime 4 days, 0:41:33
kube-controller-manager-7-21 RUNNING pid 89625, uptime 1 day, 4:53:09
kube-kubelet-7-21 RUNNING pid 1027, uptime 4 days, 0:41:33
kube-proxy-7-21 RUNNING pid 1032, uptime 4 days, 0:41:33
kube-scheduler-7-21 RUNNING pid 90832, uptime 1 day, 4:49:07
[root@hdss7-21 etcd]# supervisorctl update
flanneld-7-21: added process group
[root@hdss7-21 flannel]# supervisorctl status
etcd-server-7-21 RUNNING pid 1029, uptime 4 days, 1:03:48
flanneld-7-21 RUNNING pid 50922, uptime 0:01:44
kube-apiserver-7-21 RUNNING pid 1030, uptime 4 days, 1:03:48
kube-controller-manager-7-21 RUNNING pid 48165, uptime 0:11:30
kube-kubelet-7-21 RUNNING pid 1027, uptime 4 days, 1:03:48
kube-proxy-7-21 RUNNING pid 1032, uptime 4 days, 1:03:48
kube-scheduler-7-21 RUNNING pid 48158, uptime 0:11:30
ping
[root@hdss7-22 ~]# ping 172.7.21.2
PING 172.7.21.2 (172.7.21.2) 56(84) bytes of data.
64 bytes from 172.7.21.2: icmp_seq=1 ttl=63 time=0.710 ms
64 bytes from 172.7.21.2: icmp_seq=2 ttl=63 time=0.429 ms
查看路由
[root@hdss7-22 etcd]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.4.7.254 0.0.0.0 UG 100 0 0 eth0
10.4.7.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
172.7.21.0 10.4.7.21 255.255.255.0 UG 0 0 0 eth0
172.7.22.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
凡是172.7.21.0/24的網絡想要通信或者被訪問，它的網關是10.4.7.21。
[root@hdss7-21 flannel]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.4.7.254 0.0.0.0 UG 100 0 0 eth0
10.4.7.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
172.7.21.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
172.7.22.0 10.4.7.22 255.255.255.0 UG 0 0 0 eth0
凡是要去172.7.22.0網絡的包，必須要經過10.4.7.22這個網關。同樣，凡是要去172.7.21.0的包，必須要通過10.4.7.21這個網關。
宿主機IP是10.4.7.21和10.4.7.22，兩台宿主機上的docker的NAT網段分別是10.7.21.0/24和10.7.22.0/24網段。
而flannel其實什么都沒干，就是給對應的宿主機加了靜態路由。
PS：手動加靜態路由
[root@hdss7-21 ~]# route add -net 172.7.22.0/24 gw 10.4.7.21 dev eth0
[root@hdss7-22 ~]# route add -net 172.7.21.0/24 gw 10.4.7.22 dev eth0
Flannel的host-gw靜態路由模型非常重要的前提條件——所有的宿主機要同處在一個二層網絡下，也就是說他們要指向同一個網關。這樣才能用host-gw模型，才能通過維護靜態路由表的方式去讓docker能夠跨宿主機通信。CNI網絡模型，效率最高的就是Flannel模型。因為是通過內核轉發，只需要在宿主機上維護一張路由表即可，沒有任何額外的資源開銷。
Flannel的VxLAN模型
可以看到兩個宿主機在不通的二層網絡下，用host-gw模型就不能解決問題了。只能使用vxlan模型。
如上圖，flannel.1相當於是實例化出來的虛擬網卡，就是一個Flannel網絡隧道，網絡適配。
停止flannel之后，需要再查看一下相關進程是否還依然存在，如果存在，需要強制殺掉進程。
手動刪掉靜態路由
[root@hdss7-22 ~]# route del -net 172.7.21.0/24 gw 10.4.7.21
[root@hdss7-21 ~]# route del -net 172.7.22.0/24 gw 10.4.7.22
容器網絡通了，接下來還要做其他事情。
[root@hdss7-21 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ds-nmgjn 1/1 Running 1 11d
nginx-ds-v7hrn 1/1 Running 1 11d
[root@hdss7-21 ~]# kubectl exec -it nginx-ds-nmgjn /bin/bash
root@nginx-ds-nmgjn:/# ping 172.7.22.2
PING 172.7.22.2 (172.7.22.2): 48 data bytes
56 bytes from 172.7.22.2: icmp_seq=0 ttl=62 time=0.854 ms
56 bytes from 172.7.22.2: icmp_seq=1 ttl=62 time=0.289 ms
修改yaml配置文件
apiVersion: extensions/v1beta1
image: harbor.od.com/public/nginx:curl
進入容器
[root@hdss7-21 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ds-nmgjn 1/1 Running 1 11d
nginx-ds-v7hrn 1/1 Running 1 11d
[root@hdss7-21 ~]# kubectl delete pod nginx-ds-nmgjn
pod "nginx-ds-nmgjn" deleted
[root@hdss7-21 ~]# kubectl delete pod nginx-ds-v7hrn
pod "nginx-ds-v7hrn" deleted
[root@hdss7-21 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ds-gwswr 1/1 Running 0 7s
nginx-ds-jh2x5 1/1 Running 0 15s
[root@hdss7-21 ~]# kubectl exec -it nginx-ds-jh2x5 /bin/bash
root@nginx-ds-jh2x5:/# curl 172.7.22.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
在7-22上查看日志
[root@hdss7-22 ~]# kubectl logs -f nginx-ds-gwswr
10.4.7.21 - - [22/Aug/2020:09:18:01 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
發現訪問的IP地址是10.4.7.21，是宿主機IP，而不是容器的IP地址，說明還是有問題。
在兩台機器都做如下操作
[root@hdss7-22 ~]# yum install iptables-services -y
[root@hdss7-22 ~]# systemctl start iptables && systemctl enable iptables
[root@hdss7-21 ~]# systemctl start iptables && systemctl enable iptables
查看iptable規則
[root@hdss7-21 ~]# iptables-save |grep -i postrouting
:POSTROUTING ACCEPT [19:1155]
:KUBE-POSTROUTING - [0:0]
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
重點是這個
-A POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
把規則刪掉
[root@hdss7-21 ~]# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
插入一條新的規則
[root@hdss7-21 ~]# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
原地址是172.7.21.0/24，不是(!)從docker0這個出口出去的，目標地址(-d)不是172.7.0.0/16的網絡，才做snat地址轉換。
保存
[root@hdss7-21 ~]# iptables-save > /etc/sysconfig/iptables
如果不通
[root@hdss7-21 ~]# kubectl exec -it nginx-ds-jh2x5 /bin/bash
root@nginx-ds-jh2x5:/# curl 172.7.22.2
curl: (7) Failed to connect to 172.7.22.2 port 80: No route to host
查看reject
[root@hdss7-21 ~]# iptables-save | grep -i reject
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
兩台機器的規則都要刪掉
[root@hdss7-21 ~]# iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-22 ~]# iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-22 ~]# iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited
再保存一次
[root@hdss7-21 ~]# iptables-save > /etc/sysconfig/iptables
[root@hdss7-22 ~]# iptables-save > /etc/sysconfig/iptables
再curl一次
[root@hdss7-21 ~]# kubectl exec -it nginx-ds-jh2x5 /bin/bash
root@nginx-ds-jh2x5:/# curl 172.7.22.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
查看7-22的日志
[root@hdss7-22 ~]# kubectl logs -f nginx-ds-gwswr
10.4.7.21 - - [22/Aug/2020:09:18:01 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
172.7.21.2 - - [22/Aug/2020:09:46:18 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
在7-22也刪除並新建規則
[root@hdss7-22 ~]# iptables-save |grep -i postrouting
:POSTROUTING ACCEPT [16:972]
:KUBE-POSTROUTING - [0:0]
-A POSTROUTING -s 172.7.22.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
[root@hdss7-22 ~]# iptables -t nat -D POSTROUTING -s 172.7.22.0/24 ! -o docker0 -j MASQUERADE
[root@hdss7-22 ~]# iptables -t nat -I POSTROUTING -s 172.7.22.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
同樣，在7-22執行curl命令
[root@hdss7-22 ~]# kubectl exec -it nginx-ds-gwswr /bin/bash
root@nginx-ds-gwswr:/# curl 172.7.21.2
# 在宿主機上curl
[root@hdss7-22 ~]# curl 172.7.21.2
查看7-21容器的日志
[root@hdss7-21 ~]# kubectl logs -f nginx-ds-jh2x5
172.7.22.2 - - [22/Aug/2020:10:02:30 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
10.4.7.22 - - [22/Aug/2020:10:04:30 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
總結：
- 在集群內部根本不需要nat轉換，如果在集群內部也做轉換的話，查看日志會發現全是宿主機IP，而不是對應容器的IP，排查問題根本無從下手。
- 因此一定要做snat規則優化，讓容器互相看到對方真實的IP，而不是宿主機IP。