在1.19版本之前,kubeadm部署方式啟用ipvs模式時,初始化配置文件需要添加以下內容:
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration featureGates: SupportIPVSProxyMode: true mode: ipvs
本次在1.20.2版本中,使用kubeadm進行集群初始化時,雖然可以正常部署,但是查看pod情況的時候可以看到kube-proxy無法運行成功,報錯部分內容如下:
]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
....#省略其他輸出內容
kube-system kube-proxy-7vrbv 0/1 CrashLoopBackOff 9 43m
kube-system kube-proxy-ghs7h 0/1 CrashLoopBackOff 9 43m
kube-system kube-proxy-l9twb 0/1 CrashLoopBackOff 1 7s
kube-system kube-proxy-mzfrf 0/1 CrashLoopBackOff 9 42m
kube-system kube-proxy-nxpls 0/1 CrashLoopBackOff 9 3h4m
kube-system kube-proxy-pmmtq 0/1 CrashLoopBackOff 8 42m
#查看日志信息
]# kubectl logs kube-proxy-l9twb -n kube-system F0114 12:58:34.042769 1 server.go:488] failed complete: unrecognized feature gate: SupportIPVSProxyMode goroutine 1 [running]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00000e001, 0xc0004b6000, 0x6e, 0xc0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9 k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x29b65c0, 0xc000000003, 0x0, 0x0, 0xc0003d8230, 0x28edbc9, 0x9, 0x1e8, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x19b k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printf(0x29b65c0, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1b3a573, 0x13, 0xc000431310, 0x1, ...) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:750 +0x191 k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatalf(...)
通過報錯可以看到kube-proxy無法識別SupportIPVSProxyMode這個字段,於是訪問官方查看最新版本ipvs開啟的正確配置,通過https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md可以看到官方說明:
Cluster Created by Kubeadm
If you are using kubeadm with a configuration file, you have to add mode: ipvs below the kubeProxy field as part of the kubeadm configuration.
...
kubeProxy:
config:
mode: ipvs
...
由於集群已經初始化成功了,所以現在改kubeadm初始化配置文件沒有意義,因為我們需要直接修改kube-proxy的啟動配置
通過查看kube-pxory的資源清單可以知道, kube-proxy的配置文件是通過configmap方式掛載到容器中的,因此我們只需要對應修改configmap中的配置內容,就可以將無效字段刪除
]# kubectl -n kube-system get pod kube-proxy-24tkb -o yaml apiVersion: v1 kind: Pod metadata: ..... #其他內容省略 containers: - command: - /usr/local/bin/kube-proxy - --config=/var/lib/kube-proxy/config.conf - --hostname-override=$(NODE_NAME) ..... #其他內容省略 volumeMounts: - mountPath: /var/lib/kube-proxy name: kube-proxy ..... #其他內容省略 volumes: - configMap: defaultMode: 420 name: kube-proxy name: kube-proxy
]# kubectl get cm -n kube-system NAME DATA AGE coredns 1 5h18m extension-apiserver-authentication 6 5h18m kube-proxy 2 5h18m kube-root-ca.crt 1 5h18m kubeadm-config 2 5h18m kubelet-config-1.20 1 5h18m
]# kubectl edit cm kube-proxy -n kube-system
#在編輯模式中找到以下字段,刪除后保存退出
featureGates:
SupportIPVSProxyMode: true
然后將刪除所有kube-proxy進行重啟,查看pod運行情況
]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE ... #其他內容省略 kube-proxy-24tkb 1/1 Running 0 122m kube-proxy-9cl7j 1/1 Running 0 123m kube-proxy-cxbg5 1/1 Running 0 123m kube-proxy-cxgqk 1/1 Running 0 123m kube-proxy-hkq54 1/1 Running 0 123m kube-proxy-ttdqb 1/1 Running 0 122m
在服務器上安裝ipvsadm,查看ipvs模式是否啟用成功
]# yum install ipvsadm -y ]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.96.0.1:443 rr -> 和諧ip:6443 Masq 1 0 0 -> 和諧ip:6443 Masq 1 0 0 -> 和諧ip:6443 Masq 1 0 0 TCP 10.96.0.10:53 rr TCP 10.96.0.10:9153 rr UDP 10.96.0.10:53 rr