記一次k8s部署問題

問題描述

在k8s master節點按步驟創建成功后，執行從節點的加入操作，發現一直超時無法加入

問題日志

[root@node02 kubernetes-1.14]# kubeadm join 10.0.0.121:6443 --token oxqn4k.2olj80b59gndjmdj --discovery-token-unsafe-skip-ca-verification
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
error execution phase kubelet-start: timed out waiting for the condition

問題解決

1.按網上查了並不是token過期的原因

2.反復操作過程中，發現master非常卡，后來執行命令都無法返回結果了

kubeadm token list
failed to list bootstrap tokens: Get https://10.0.0.121:6443/api/v1/namespaces/kube-system/secrets?fieldSelector=type%3Dbootstrap.kubernetes.io%2Ftoken: net/http: TLS handshake timeout

3.判斷可能是內存不足

[root@node01 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           972M        524M         61M         50M        386M         76M
Swap:            0B          0B          0B

• 只有1G內存

4.於是打開虛擬機設置，增加1G內存

[root@node01 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        552M        595M         50M        848M        1.2G
Swap:            0B          0B          0B

發現master機流暢多了

5.重新執行節點加入，成功加入

kubeadm join 10.0.0.121:6443 --token befijo.vtrzxgw2ds7m5bak     --discovery-token-ca-cert-hash sha256:5ea9439afdd166995f92d9d42a6fcb73aa8c92e5af5570850a709273f361b5b0
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details

[root@node01 ~]# kubectl get nodes
NAME     STATUS   ROLES    AGE   VERSION
node01   Ready    master   20h   v1.14.1
node02   Ready    <none>   30m   v1.14.1

問題總結

• 主節點由於內存不足無法正常服務導致從節點無法加入
• 又是一起簡單的原因導致難查的BUG
• 由此可知k8s部署主節點至少需要分配2G內存