kubernetes NotReady
部署節點 NotReady
#kubectl get nodes NAME STATUS ROLES AGE VERSION centos107 NotReady control-plane,master 41h v1.23.4 centos117 NotReady <none> 41h v1.23.4 centos127 NotReady <none> 41h v1.23.4 centos147 NotReady <none> 41h v1.23.4
查看node信息
#kubectl describe node
...
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason M
---- ------ ----------------- ------------------ ------ -
NetworkUnavailable False Sat, 26 Feb 2022 16:46:51 +0800 Sat, 26 Feb 2022 16:46:51 +0800 FlannelIsUp F on this node
MemoryPressure Unknown Sat, 26 Feb 2022 22:16:55 +0800 Sun, 27 Feb 2022 16:32:30 +0800 NodeStatusUnknown Ksting node status.
DiskPressure Unknown Sat, 26 Feb 2022 22:16:55 +0800 Sun, 27 Feb 2022 16:32:30 +0800 NodeStatusUnknown Ksting node status.
PIDPressure Unknown Sat, 26 Feb 2022 22:16:55 +0800 Sun, 27 Feb 2022 16:32:30 +0800 NodeStatusUnknown Ksting node status.
Ready Unknown Sat, 26 Feb 2022 22:16:55 +0800 Sun, 27 Feb 2022 16:32:30 +0800 NodeStatusUnknown
.
查看kubelete通信
#systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Mon 2022-02-28 09:35:52 CST; 7min ago Docs: https://kubernetes.io/docs/ Main PID: 820 (kubelet) Tasks: 16 Memory: 124.5M CGroup: /system.slice/kubelet.service └─820 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/boots... Feb 28 09:43:37 node3 kubelet[820]: E0228 09:43:37.017497 820 kubelet.go...d" Feb 28 09:43:37 node3 kubelet[820]: E0228 09:43:37.119369 820 kubelet.go...d" Feb 28 09:43:37 node3 kubelet[820]: E0228 09:43:37.220334 820 kubelet.go...d" Feb 28 09:43:37 node3 kubelet[820]: E0228 09:43:37.322111 820 kubelet.go...d" Feb 28 09:43:37 node3 kubelet[820]: E0228 09:43:37.422352 820 kubelet.go...d"
查看kubelete日志信息
#journalctl -u kubelet.service Feb 28 09:35:39 master kubelet[819]: E0228 09:35:39.117414 819 kubelet.go:2422] "Error getting node" err="node \"master\" not found" Feb 28 09:35:39 master kubelet[819]: E0228 09:35:39.240928 819 kubelet.go:2422] "Error getting node" err="node \"master\" not found" Feb 28 09:35:39 master kubelet[819]: E0228 09:35:39.341755 819 kubelet.go:2422] "Error getting node" err="node \"master\" not found" Feb 28 09:35:39 master kubelet[819]: W0228 09:35:39.425709 819 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list * Feb 28 09:35:39 master kubelet[819]: E0228 09:35:39.425795 819 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch
發現問題，節點的主機名和kublete的主機名不一致，被修改。添加hosts解析,未果
# cat /etc/hosts 10.0.0.107 master centos107 10.0.0.117 node1 centos117 10.0.0.127 node2 centos127 10.0.0.147 node3 centos147
解決方法
master修改hostname，或將master節點reset也可，鏡像都已pull執行不受影響
#hostnamectl set-hostname centos107 && bash
#systemctl restart kubelet
node重新加入節點
#kubeadm reset -f && rm -rf $HOME/.kube/config #kubeadm join 10.0.0.107:6443 --token asdf3yp.71joyabpn1234 --discovery-token-ca-cert-hash sha256:3e7aacad24a5bc4c4587010179eee04a36ae8fc5e9338774930d65e32e74ee0a token過期報錯 error execution phase preflight: couldn't validate the identity of the API Server: could not find a JWS signature in the cluster-info ConfigMap for token ID "hvg2gc" To see the stack trace of this error execute with --v=5 or higher #kubeadm token create wjb7yp.71joyabpn190e8kl
獲取ca證書sha256編碼hash值[本次未更改CA，所以未更換hash值]
# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' 替換token重新加入節點 #kubeadm join 10.0.0.107:6443 --token wjb7yp.71joyabpn190e8kl --discovery-token-ca-cert-hash sha256:3e7aacad24a5bc4c4587010179eee04a36ae8fc5e9338774930d65e32e74ee0a
#kubectl get nodes
W0228 10:42:03.859157 24068 loader.go:221] Config not found: /etc/kubernetes/admin.conf
The connection to the server localhost:8080 was refused - did you specify the right host or port?
報錯原因為缺少admin.conf配置文件，拷貝
#scp /etc/kubernetes/admin.conf node2:/etc/kubernetes/admin.conf
#kubectl get node
NAME STATUS ROLES AGE VERSION
centos107 NOtReady control-plane,master 42h v1.23.4
centos117 NotReady <none> 42h v1.23.4
centos127 NotReady <none> 42h v1.23.4
centos147 NotReady <none> 42h v1.23.4
node1 Ready <none> 3m23s v1.23.4
#刪除沒用節點信息
kubectl delete nodes centos117 centos127 centos147
查看節點
#kubectl get nodes NAME STATUS ROLES AGE VERSION centos107 Ready control-plane,master 42h v1.23.4 node1 Ready <none> 37m v1.23.4 node2 Ready <none> 22m v1.23.4 node3 Ready <none> 18m v1.23.4
總結
hostname在初始化時，按照初始化規范配置好，后續不要再次修改以免造成不必要影響，同時主機名命名主機名盡量不要使用下划線，按照各自規范執行初始化配置