更好的閱讀體驗建議點擊下方原文鏈接。
原文地址：http://maoqide.live/post/cloud/build-docker-image-in-a-pod-in-kubernetes
升級 kubeadm 部署的 kubernetes 集群。
當前環境
[root@centos10 ~]$ uname -a
Linux centos10 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@centos10 ~]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"7a578febe155a7366767abce40d8a16795a96371", GitTreeState:"clean", BuildDate:"2019-05-01T04:14:38Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"7a578febe155a7366767abce40d8a16795a96371", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
確認升級版本
由於目前集群版本和最新版本相差版本較多，采用逐個版本升級的策略，首先確認要升級到的具體版本。
yum list --showduplicates | grep kubeadm
選取要升級版本的最新小版本，即1.12.10-0。
升級 master
# backup config
kubeadm config view > kubeadm.yaml
# upgrade kubeadm
yum upgrade -y kubeadm-1.12.x --disableexcludes=kubernetes
# confirm version
kubeadm version
kubeadm upgrade plan查看升級組件后版本
[root@centos10 v1.12.10]# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.11.10
[upgrade/versions] kubeadm version: v1.12.10
I1031 13:42:09.427100 7721 version.go:93] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get https://dl.k8s.io/release/stable.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1031 13:42:09.427154 7721 version.go:94] falling back to the local client version: v1.12.10
[upgrade/versions] Latest stable version: v1.12.10
I1031 13:42:19.466181 7721 version.go:93] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.11.txt": Get https://dl.k8s.io/release/stable-1.11.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1031 13:42:19.466211 7721 version.go:94] falling back to the local client version: v1.12.10
[upgrade/versions] Latest version in the v1.11 series: v1.12.10
External components that should be upgraded manually before you upgrade the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Etcd 3.3.13 3.2.24
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 3 x v1.11.10 v1.12.10
Upgrade to the latest version in the v1.11 series:
COMPONENT CURRENT AVAILABLE
API Server v1.11.10 v1.12.10
Controller Manager v1.11.10 v1.12.10
Scheduler v1.11.10 v1.12.10
Kube Proxy v1.11.10 v1.12.10
CoreDNS 1.1.3 1.2.2
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.12.10
_____________________________________________________________________
可以看到升級后 k8s 各組件版本，由於無法直接訪問 gcr 鏡像倉庫，需要手動拉取對應版本。
k8s組件的鏡像命名基本為 k8s.gcr.io/<component-name>-<arch>:<component-version> 或 gcr.io/google-containers/<component-name>:<component-version>, 二者實際上是同一倉庫，國內可以使用azure 的鏡像倉庫下載，只要將后者gcr.io替換為gcr.azk8s.cn即可。也可以使用https://raw.githubusercontent.com/maoqide/utils/master/pullgcr.sh腳本一鍵拉取。
pullgcr k8s.gcr.io/kube-apiserver:v1.12.10
pullgcr k8s.gcr.io/kube-controller-manager:v1.12.10
pullgcr k8s.gcr.io/kube-scheduler:v1.12.10
pullgcr k8s.gcr.io/kube-proxy:v1.12.10
pullgcr k8s.gcr.io/coredns:1.2.2
pullgcr k8s.gcr.io/etcd:3.2.24
如果使用的內部鏡像倉庫，將以上鏡像tag后推送到內部的鏡像倉庫中。
# using dry-run flag for validation
kubeadm upgrade apply v1.12.10 --config kubeadm.yaml --dry-run
使用 kubeadm config view 保存的配置，有可能有遺漏，在升級時遇到主節點的 nodeName 配置沒有保存，所以在更新時，使用默認的 hostname 為 node name，導致在拉起apiserver的靜態pod時，以 kube-apiserver-<node-name>的規則拼接 pod 名出錯，一直找不到已有的 apiserver 的靜態pod，進而導致 upgrade 超時而失敗。
遇到kubeadm upgrade一直卡住並最終超時的情況時，首先 kubeadm upgrade apply v1.12.10 --config kubeadm.yaml -v 5的時候添加 -v參數指定 log 等級，一般到5就比較詳細了，然后通過日志來定位問題，以上問題就是通過如下日志看到一直在請求錯誤的 apiserver 的pod名稱，從而定位到錯誤。
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.12.10"...
I1031 16:03:08.955081 25594 request.go:530] Throttling request took 185.13957ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-centos10
解決：kubeadm config print-default 可以查看 kubeadm 的 config 文件的所有默認的配置，查看后發現有如下nodeRegistration配置, 將node name默認指定成主機的 hostname，導致了上述的錯誤，在之前通過kubeadm config view生成的配置文件中，加上
nodeRegistration:
name: <real-node-name>
並重新執行kubeadm upgrade apply v1.12.10 --config kubeadm.yaml即可。
...
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: centos10
tlsBootstrapToken: abcdef.0123456789abcdef
token: abcdef.0123456789abcdef
...
[root@centos10 v1.12.10]$ kubeadm upgrade apply v1.12.10 --config kubeadm.yaml -v 5
I1031 16:18:09.848434 31033 apply.go:86] running preflight checks
[preflight] Running pre-flight checks.
I1031 16:18:09.848496 31033 apply.go:92] fetching configuration from file kubeadm.yaml
I1031 16:18:09.848507 31033 masterconfig.go:167] loading configuration from the given file
I1031 16:18:09.855269 31033 feature_gate.go:206] feature gates: &{map[]}
I1031 16:18:09.855302 31033 apply.go:150] [upgrade/apply] verifying health of cluster
I1031 16:18:09.855307 31033 apply.go:151] [upgrade/apply] retrieving configuration from cluster
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration options from a file: kubeadm.yaml
[upgrade/apply] Respecting the --cri-socket flag that is set with higher priority than the config file.
I1031 16:18:09.877158 31033 apply.go:163] [upgrade/apply] validating requested and actual version
I1031 16:18:09.877171 31033 apply.go:181] [upgrade/version] enforcing version skew policies
[upgrade/version] You have chosen to change the cluster version to "v1.12.10"
[upgrade/versions] Cluster version: v1.11.10
[upgrade/versions] kubeadm version: v1.12.10
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
I1031 16:18:11.377401 31033 apply.go:195] [upgrade/apply] creating prepuller
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
I1031 16:18:14.046733 31033 request.go:530] Throttling request took 64.263528ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-controller-manager
I1031 16:18:14.246235 31033 request.go:530] Throttling request took 263.748054ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-scheduler
I1031 16:18:14.647280 31033 request.go:530] Throttling request took 173.534831ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-apiserver
I1031 16:18:14.846492 31033 request.go:530] Throttling request took 367.060734ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-controller-manager
I1031 16:18:15.045493 31033 request.go:530] Throttling request took 566.033546ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-scheduler
I1031 16:18:15.245604 31033 request.go:530] Throttling request took 271.241012ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-etcd
I1031 16:18:15.445656 31033 request.go:530] Throttling request took 471.269391ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-apiserver
I1031 16:18:15.646784 31033 request.go:530] Throttling request took 668.609111ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-controller-manager
I1031 16:18:15.845868 31033 request.go:530] Throttling request took 370.912219ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-etcd
I1031 16:18:16.045929 31033 request.go:530] Throttling request took 570.956625ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-apiserver
I1031 16:18:16.245422 31033 request.go:530] Throttling request took 767.553384ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-scheduler
I1031 16:18:16.445445 31033 request.go:530] Throttling request took 468.71903ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-etcd
[upgrade/prepull] Prepulled image for component etcd.
I1031 16:18:16.645553 31033 request.go:530] Throttling request took 666.826898ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-controller-manager
I1031 16:18:16.845616 31033 request.go:530] Throttling request took 371.917463ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-apiserver
I1031 16:18:17.047923 31033 request.go:530] Throttling request took 568.641489ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-scheduler
I1031 16:18:17.245839 31033 request.go:530] Throttling request took 273.128186ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-apiserver
[upgrade/prepull] Prepulled image for component kube-apiserver.
I1031 16:18:17.445456 31033 request.go:530] Throttling request took 466.94069ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-controller-manager
[upgrade/prepull] Prepulled image for component kube-controller-manager.
I1031 16:18:17.646001 31033 request.go:530] Throttling request took 167.6097ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dupgrade-prepull-kube-scheduler
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
I1031 16:18:17.653778 31033 apply.go:202] [upgrade/apply] performing upgrade
I1031 16:18:17.653785 31033 apply.go:268] checking if cluster is self-hosted
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.12.10"...
I1031 16:18:17.845963 31033 request.go:530] Throttling request took 184.620923ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-172.27.32.165
Static pod: kube-apiserver-172.27.32.165 hash: c5c5b18e12d5e605ef55d516c4559ba0
I1031 16:18:18.045736 31033 request.go:530] Throttling request took 193.159946ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-172.27.32.165
Static pod: kube-controller-manager-172.27.32.165 hash: 6e79e209a8bc468fa21969e807be68e3
I1031 16:18:18.245404 31033 request.go:530] Throttling request took 197.471715ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-172.27.32.165
Static pod: kube-scheduler-172.27.32.165 hash: dcb7e042ad7733f6f04068629eb4b8b9
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests040725233"
I1031 16:18:18.247454 31033 manifests.go:44] [controlplane] creating static pod files
I1031 16:18:18.247469 31033 manifests.go:117] [controlplane] getting StaticPodSpecs
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests040725233/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests040725233/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests040725233/kube-scheduler.yaml"
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-10-31-16-18-17/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s
I1031 16:18:18.446010 31033 request.go:530] Throttling request took 178.44808ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-172.27.32.165
Static pod: kube-apiserver-172.27.32.165 hash: 7d9abb385d9679f7575617a4e03828be
I1031 16:18:18.648914 31033 request.go:530] Throttling request took 200.383408ms, request: GET:https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver
[apiclient] Error getting Pods with label selector "component=kube-apiserver" [Get https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 172.27.32.165:6443: connect: connection refused]
[apiclient] Error getting Pods with label selector "component=kube-apiserver" [Get https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 172.27.32.165:6443: connect: connection refused]
[apiclient] Error getting Pods with label selector "component=kube-apiserver" [Get https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 172.27.32.165:6443: connect: connection refused]
[apiclient] Error getting Pods with label selector "component=kube-apiserver" [Get https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 172.27.32.165:6443: connect: connection refused]
[apiclient] Error getting Pods with label selector "component=kube-apiserver" [Get https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: dial tcp 172.27.32.165:6443: connect: connection refused]
[apiclient] Error getting Pods with label selector "component=kube-apiserver" [Get https://172.27.32.165:6443/api/v1/namespaces/kube-system/pods?labelSelector=component%3Dkube-apiserver: net/http: TLS handshake timeout]
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-10-31-16-18-17/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s
Static pod: kube-controller-manager-172.27.32.165 hash: 6e79e209a8bc468fa21969e807be68e3
Static pod: kube-controller-manager-172.27.32.165 hash: a9ef5d96e631ef6bb37aa7399b39ce31
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-10-31-16-18-17/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s
Static pod: kube-scheduler-172.27.32.165 hash: dcb7e042ad7733f6f04068629eb4b8b9
Static pod: kube-scheduler-172.27.32.165 hash: 92d30ca7866fe98c3147c1f8f4c3d413
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
I1031 16:18:47.231899 31033 apply.go:208] [upgrade/postupgrade] upgrading RBAC rules and addons
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "172.27.32.165" as an annotation
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
I1031 16:18:47.794738 31033 clusterinfo.go:79] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I1031 16:18:47.849852 31033 request.go:530] Throttling request took 50.242056ms, request: POST:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/rolebindings
I1031 16:18:48.049867 31033 request.go:530] Throttling request took 195.937473ms, request: PUT:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/rolebindings/kubeadm:bootstrap-signer-clusterinfo
I1031 16:18:48.249312 31033 request.go:530] Throttling request took 189.877158ms, request: POST:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/clusterroles
I1031 16:18:48.449307 31033 request.go:530] Throttling request took 195.86853ms, request: PUT:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/system:coredns
I1031 16:18:48.650121 31033 request.go:530] Throttling request took 198.286357ms, request: POST:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings
I1031 16:18:48.849300 31033 request.go:530] Throttling request took 196.19422ms, request: PUT:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/system:coredns
[addons] Applied essential addon: CoreDNS
I1031 16:18:49.049271 31033 request.go:530] Throttling request took 146.459988ms, request: POST:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings
I1031 16:18:49.249379 31033 request.go:530] Throttling request took 195.347278ms, request: PUT:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kubeadm:node-proxier
I1031 16:18:49.449396 31033 request.go:530] Throttling request took 198.120697ms, request: POST:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles
I1031 16:18:49.649555 31033 request.go:530] Throttling request took 197.170914ms, request: POST:https://172.27.32.165:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.10". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
升級成功后升級 kubectl：
# upgrade kubectl
yum upgrade -y kubectl-1.12.10 --disableexcludes=kubernetes
升級 kubelet
逐台節點升級 kubelet。
升級前最好先備份下當前kubelet的配置：
cp /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 10-kubeadm.conf.bak
# upgrade kubelet/kubeadm
yum upgrade -y kubelet-1.12.10 kubeadm-1.12.10 --disableexcludes=kubernetes
# kubeadm upgrade node config
kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d ' ' -f 2)
# restart
systemctl daemon-reload
systemctl restart kubelet
至此升級完成，可以執行kubectl get nodes，進行驗證，是否所有 node 都正常啟動。