本文記錄的是將 kubernetes 集群從 1.17.0 升級至最新版 1.20.2 的實際操作步驟，由於 1.17.0 無法直接升級到 1.20.2，需要進行2次過濾升級，1.17.0 -> 1.18.0 -> 1.19.0 -> 1.20.2。
先在 control plane node 上進行升級操作，通過下面的命令查看最新版 kubeadm 的版本號
apt update
apt-cache madison kubeadm
最新版是 1.20.2-00，用下面的命令安裝最新版 kubeadm
apt-get update && \
apt-get install -y --allow-change-held-packages kubeadm=1.20.2-00
然后查看升級計划
kubeadm upgrade plan
提示無法直接升級1.20.2
[upgrade/config] FATAL: this version of kubeadm only supports deploying clusters with the control plane version >= 1.19.0. Current version: v1.17.0
於是嘗試先升級到 1.19.0
apt-get update && \
apt-get install -y --allow-change-held-packages --allow-downgrades kubeadm=1.19.0-00
查看升級計划1.19.0
[upgrade/config] FATAL: this version of kubeadm only supports deploying clusters with the control plane version >= 1.18.0. Current version: v1.17.0
只能先升級到1.18.0
apt-get update && \
apt-get install -y --allow-change-held-packages --allow-downgrades kubeadm=1.18.0-00
查看升級計划
kubeadm upgrade plan
啟動升級操作
kubeadm upgrade apply v1.18.0
集群成功升級至 1.18.0
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.18.0". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
分別在 control plane node 與 worker node 上 將 kubelet 和 kubectl 升級至 1.18.0（如果是生產環境，在升級之前先 uncordon 要升級的節點）
apt-get update && \
apt-get install -y --allow-change-held-packages kubelet=1.18.0-00 kubectl=1.18.0-00
systemctl daemon-reload && systemctl restart kubelet
繼續，向 1.19.0 升級！
升級 kubeadm 至 1.19.0
apt-get install -y --allow-change-held-packages --allow-downgrades kubeadm=1.19.0-00
升級集群至 1.19.0
kubeadm upgrade plan
kubeadm upgrade apply v1.19.0
升級 kubelet 和 kubectl 至 1.19.0
apt-get install -y --allow-change-held-packages kubelet=1.19.0-00 kubectl=1.19.0-00
systemctl daemon-reload && systemctl restart kubelet
沖刺，向最新版升級！
升級 kubeadm 至 1.20.2
apt-get install -y --allow-change-held-packages kubeadm=1.20.2-00
升級集群至 1.20.2
kubeadm upgrade plan
kubeadm upgrade apply v1.20.2
各個 node 升級 kubelet、kubectl、kubeadm 至 1.20.2
apt-get install -y --allow-change-held-packages kubelet=1.20.2-00 kubectl=1.20.2-00 kubeadm=1.20.2-00
systemctl daemon-reload && systemctl restart kubelet
勝利，升級成功！
NAME STATUS ROLES AGE VERSION
k8s-master0 Ready control-plane,master 371d v1.20.2
kube-node1 Ready <none> 2d15h v1.20.2
kube-node2 Ready <none> 38h v1.20.2
kube-node3 Ready <none> 17h v1.20.2
升級過程中遇到的問題
我們在 coredns 配置中使用了 rewrite answer ，升級時出現下面的錯誤提示
[preflight] Some fatal errors occurred:
[ERROR CoreDNSUnsupportedPlugins]: CoreDNS cannot migrate the following plugins:
[Option "answer" in plugin "rewrite" is unsupported by this migration tool in 1.6.5. Option "answer"
通過 plan --ignore-preflight-errors 參數忽略了這個錯誤
kubeadm upgrade plan --ignore-preflight-errors=CoreDNSUnsupportedPlugins
所以升級后 coredns 的配置沒有遷移過來，需要在升級前導出 coredns 的 ConfigMap
kubectl get cm coredns -n kube-system -o yaml > coredns_cm.yaml
然后在升級后替換 coredns 的配置
kubectl replace -n kube-system -f coredns_cm.yaml