iptables
iptables的方式则是利用了linux的iptables的nat转发进行实现。在本例中，创建了名为mysql-service的service。
apiVersion: v1 kind: Service metadata: labels: name: mysql role: service name: mysql-service spec: ports: - port: 3306 targetPort: 3306 nodePort: 30964 type: NodePort selector: mysql-service: "true" mysql-service对应的nodePort暴露出来的端口为30964，对应的cluster IP(10.254.162.44)的端口为3306，进一步对应于后端的pod的端口为3306。
mysql-service后端代理了两个pod，ip分别是192.168.125.129和192.168.125.131。先来看一下iptables。
[root@localhost ~]# iptables -S -t nat ... -A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES -A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES -A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000 -A KUBE-NODEPORTS -p tcp -m comment --comment "default/mysql-service:" -m tcp --dport 30964 -j KUBE-MARK-MASQ -A KUBE-NODEPORTS -p tcp -m comment --comment "default/mysql-service:" -m tcp --dport 30964 -j KUBE-SVC-67RL4FN6JRUPOJYM -A KUBE-SEP-ID6YWIT3F6WNZ47P -s 192.168.125.129/32 -m comment --comment "default/mysql-service:" -j KUBE-MARK-MASQ -A KUBE-SEP-ID6YWIT3F6WNZ47P -p tcp -m comment --comment "default/mysql-service:" -m tcp -j DNAT --to-destination 192.168.125.129:3306 -A KUBE-SEP-IN2YML2VIFH5RO2T -s 192.168.125.131/32 -m comment --comment "default/mysql-service:" -j KUBE-MARK-MASQ -A KUBE-SEP-IN2YML2VIFH5RO2T -p tcp -m comment --comment "default/mysql-service:" -m tcp -j DNAT --to-destination 192.168.125.131:3306 -A KUBE-SERVICES -d 10.254.162.44/32 -p tcp -m comment --comment "default/mysql-service: cluster IP" -m tcp --dport 3306 -j KUBE-SVC-67RL4FN6JRUPOJYM -A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS -A KUBE-SVC-67RL4FN6JRUPOJYM -m comment --comment "default/mysql-service:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ID6YWIT3F6WNZ47P -A KUBE-SVC-67RL4FN6JRUPOJYM -m comment --comment "default/mysql-service:" -j KUBE-SEP-IN2YML2VIFH5RO2T 下面来逐条分析
首先如果是通过node的30964端口访问，则会进入到以下链
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/mysql-service:" -m tcp --dport 30964 -j KUBE-MARK-MASQ -A KUBE-NODEPORTS -p tcp -m comment --comment "default/mysql-service:" -m tcp --dport 30964 -j KUBE-SVC-67RL4FN6JRUPOJYM 然后进一步跳转到KUBE-SVC-67RL4FN6JRUPOJYM的链
-A KUBE-SVC-67RL4FN6JRUPOJYM -m comment --comment "default/mysql-service:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ID6YWIT3F6WNZ47P -A KUBE-SVC-67RL4FN6JRUPOJYM -m comment --comment "default/mysql-service:" -j KUBE-SEP-IN2YML2VIFH5RO2T 这里利用了iptables的--probability的特性，使连接有50%的概率进入到KUBE-SEP-ID6YWIT3F6WNZ47P链，50%的概率进入到KUBE-SEP-IN2YML2VIFH5RO2T链。
KUBE-SEP-ID6YWIT3F6WNZ47P的链的具体作用就是将请求通过DNAT发送到192.168.125.129的3306端口。
-A KUBE-SEP-ID6YWIT3F6WNZ47P -s 192.168.125.129/32 -m comment --comment "default/mysql-service:" -j KUBE-MARK-MASQ -A KUBE-SEP-ID6YWIT3F6WNZ47P -p tcp -m comment --comment "default/mysql-service:" -m tcp -j DNAT --to-destination 192.168.125.129:3306 同理KUBE-SEP-IN2YML2VIFH5RO2T的作用是通过DNAT发送到192.168.125.131的3306端口。
-A KUBE-SEP-IN2YML2VIFH5RO2T -s 192.168.125.131/32 -m comment --comment "default/mysql-service:" -j KUBE-MARK-MASQ -A KUBE-SEP-IN2YML2VIFH5RO2T -p tcp -m comment --comment "default/mysql-service:" -m tcp -j DNAT --to-destination 192.168.125.131:3306 分析完nodePort的工作方式，接下里说一下clusterIP的访问方式。
对于直接访问cluster IP(10.254.162.44)的3306端口会直接跳转到KUBE-SVC-67RL4FN6JRUPOJYM。
-A KUBE-SERVICES -d 10.254.162.44/32 -p tcp -m comment --comment "default/mysql-service: cluster IP" -m tcp --dport 3306 -j KUBE-SVC-67RL4FN6JRUPOJYM
root@ubuntu:~# cat web-ngx-svc.yml apiVersion: v1 kind: Service metadata: name: nodeport-svc spec: type: NodePort selector: app: web-nginx ports: - protocol: TCP port: 3000 targetPort: 8087 nodePort: 30090
root@ubuntu:~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 244d my-nginx ClusterIP 10.110.79.116 <none> 8280/TCP 37d my-nginx-np NodePort 10.99.1.231 <none> 8081:31199/TCP 36d nodeport-svc NodePort 10.97.11.232 <none> 3000:30090/TCP 60m web2 NodePort 10.110.171.213 <none> 8097:31866/TCP 20d
service cluster IP
10.97.11.232 3000
root@ubuntu:~# iptables -S -t nat | grep KUBE-SVC-GFPAJ7EGCNM4QF4H -N KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport-svc:" -m tcp --dport 30090 -j KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-SERVICES -d 10.97.11.232/32 -p tcp -m comment --comment "default/nodeport-svc: cluster IP" -m tcp --dport 3000 -j KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OACTDSBVK7HADL5Z -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -j KUBE-SEP-4WFPOQNWZU6CJPY5
目的地址是10.97.11.232 3000 执行
-j KUBE-SVC-GFPAJ7EGCNM4QF4H
root@ubuntu:~# iptables -S -t nat | grep KUBE-SVC-GFPAJ7EGCNM4QF4H -N KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport-svc:" -m tcp --dport 30090 -j KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-SERVICES -d 10.97.11.232/32 -p tcp -m comment --comment "default/nodeport-svc: cluster IP" -m tcp --dport 3000 -j KUBE-SVC-GFPAJ7EGCNM4QF4H
-A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OACTDSBVK7HADL5Z -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -j KUBE-SEP-4WFPOQNWZU6CJPY5
dnat
KUBE-SEP-OACTDSBVK7HADL5Z和KUBE-SEP-4WFPOQNWZU6CJPY5
root@ubuntu:~# iptables -S -t nat | grep KUBE-SEP-OACTDSBVK7HADL5Z -N KUBE-SEP-OACTDSBVK7HADL5Z -A KUBE-SEP-OACTDSBVK7HADL5Z -s 10.244.0.22/32 -m comment --comment "default/nodeport-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-OACTDSBVK7HADL5Z -p tcp -m comment --comment "default/nodeport-svc:" -m tcp -j DNAT [unsupported revision] -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OACTDSBVK7HADL5Z root@ubuntu:~# iptables -S -t nat | grep KUBE-SEP-4WFPOQNWZU6CJPY5 -N KUBE-SEP-4WFPOQNWZU6CJPY5 -A KUBE-SEP-4WFPOQNWZU6CJPY5 -s 10.244.2.6/32 -m comment --comment "default/nodeport-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-4WFPOQNWZU6CJPY5 -p tcp -m comment --comment "default/nodeport-svc:" -m tcp -j DNAT [unsupported revision] -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -j KUBE-SEP-4WFPOQNWZU6CJPY5 root@ubuntu:~#
KUBE DNAT [unsupported revision]
nodeport
root@ubuntu:~# iptables -S -t nat | grep 30090 -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport-svc:" -m tcp --dport 30090 -j KUBE-MARK-MASQ -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport-svc:" -m tcp --dport 30090 -j KUBE-SVC-GFPAJ7EGCNM4QF4H
先mark（后续会做snat），然后执行
KUBE-SVC-GFPAJ7EGCNM4QF4H
KUBE-SVC-GFPAJ7EGCNM4QF4H
root@ubuntu:~# iptables -S -t nat | grep KUBE-SVC-GFPAJ7EGCNM4QF4H -N KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport-svc:" -m tcp --dport 30090 -j KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-SERVICES -d 10.97.11.232/32 -p tcp -m comment --comment "default/nodeport-svc: cluster IP" -m tcp --dport 3000 -j KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OACTDSBVK7HADL5Z -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -j KUBE-SEP-4WFPOQNWZU6CJPY5
root@ubuntu:~# iptables -S -t nat | grep KUBE-SEP-OACTDSBVK7HADL5Z -N KUBE-SEP-OACTDSBVK7HADL5Z -A KUBE-SEP-OACTDSBVK7HADL5Z -s 10.244.0.22/32 -m comment --comment "default/nodeport-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-OACTDSBVK7HADL5Z -p tcp -m comment --comment "default/nodeport-svc:" -m tcp -j DNAT [unsupported revision] -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OACTDSBVK7HADL5Z
root@ubuntu:~# iptables -S -t nat | grep KUBE-SEP-4WFPOQNWZU6CJPY5 -N KUBE-SEP-4WFPOQNWZU6CJPY5 -A KUBE-SEP-4WFPOQNWZU6CJPY5 -s 10.244.2.6/32 -m comment --comment "default/nodeport-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-4WFPOQNWZU6CJPY5 -p tcp -m comment --comment "default/nodeport-svc:" -m tcp -j DNAT [unsupported revision] -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -j KUBE-SEP-4WFPOQNWZU6CJPY5 root@ubuntu:~#
KUBE-MARK-MASQ和snat
root@ubuntu:~# iptables -nvL -t nat | grep 10.97.11.232 3 180 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.97.11.232 /* default/nodeport-svc: cluster IP */ tcp dpt:3000 3 180 KUBE-SVC-GFPAJ7EGCNM4QF4H tcp -- * * 0.0.0.0/0 10.97.11.232 /* default/nodeport-svc: cluster IP */ tcp dpt:3000
root@ubuntu:~# iptables -S -t nat | grep KUBE-MARK-MASQ -N KUBE-MARK-MASQ -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
root@ubuntu:~# iptables -S -t nat | grep KUBE-POSTROUTING -N KUBE-POSTROUTING -A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE root@ubuntu:~# iptables -S -t nat | grep KUBE-PREROUTING root@ubuntu:~#
对nat表的自定义链“KUBE-POSTROUTING”写入如下规则：
-A KUBE-POSTROUTING -m comment –comment “kubernetes service traffic requiring SNAT” -m mark –mark 0x4000/0x4000 -j MASQUERADE3、 对nat表的自定义链“KUBE-MARK-MASQ”写入如下规则：
-A KUBE-MARK-MASQ -j MARK –set-xmark 0x4000/0x4000这里2和3做的事情的实际含义是kubernetes会让所有kubernetes集群内部产生的数据包流经nat表的自定义链“KUBE-MARK-MASQ”，然后在这里kubernetes会对这些数据包打一个标记（0x4000/0x4000），接着在nat的自定义链“KUBE-POSTROUTING”中根据上述标记匹配所有的kubernetes集群内部的数据包，匹配的目的是kubernetes会对这些包做SNAT操作。