K8S部署---故障處理

本文轉載自查看原文 2021-06-02 10:31 21903 K8S

一、kubeadm部署K8S集群故障排查

問題1：kubeadm初始化master01失敗？

[root@k8s-master01 ~]# kubeadm init --config=new-kubeadm-config.yaml  --upload-certs |tee kubeadm-init.log
W0819 14:05:46.581489   81698 strict.go:47] unknown configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta3", Kind:"InitConfiguration"} for scheme definitions in "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go:31" and "k8s.io/kubernetes/cmd/kubeadm/app/componentconfigs/scheme.go:28"
no kind "InitConfiguration" is registered for version "kubeadm.k8s.io/v1beta3" in scheme "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go:31"
To see the stack trace of this error execute with --v=5 or higher

原因分析：yaml文件中apiVersion版本錯誤。

解決方法：修改yaml文件中apiVersion版本即可。

問題2：kubeadm初始化master01成功后，沒有出現加入其他master節點的提示命令？

原因分析：初始化master的yaml文件存在問題。

解決方法：修改適合k8s版本的yaml添加“controlPlaneEndpoint: 192.168.1.236:16443”即可。

問題3：kubeadm方式部署K8S集群時，node加入K8S集群時卡頓並失敗？

[root@k8s-node01 ~]# kubeadm join 192.168.1.201:6443 --token 1qo7ms.7atall1jcecf10qz --discovery-token-ca-cert-hash sha256:d1d102ceb6241a3617777f6156cd4e86dc9f9edd9e1d6d73266d6ca7f6280890
[preflight] Running pre-flight checks

原因分析：初始化主機點后發現部分組件異常；集群時間不同步；

[root@k8s-master01 ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-54d67798b7-28w5q 0/1 Pending 0 3m39s
coredns-54d67798b7-sxqpm 0/1 Pending 0 3m39s
etcd-k8s-master01 1/1 Running 0 3m53s
kube-apiserver-k8s-master01 1/1 Running 0 3m53s
kube-controller-manager-k8s-master01 1/1 Running 0 3m53s
kube-proxy-rvj6w 0/1 CrashLoopBackOff 5 3m40s
kube-scheduler-k8s-master01 1/1 Running 0 3m53s

解決方法：同步時間后，重新進行初始化主節點，再行加入其他節點。

###所有節點執行如下內容

ntpdate ntp1.aliyun.com;hwclock --systohc

date;hwclock

kubeadm reset -f;ipvsadm --clear;rm -rf ./.kube

###首個節點初始化

kubeadm init --config=new-kubeadm-config.yaml --upload-certs |tee kubeadm-init.log

###其他節點加入集群

問題3：kubeadm方式部署K8S集群時，node加入K8S集群失敗？

[root@k8s-node01 ~]# kubeadm join 192.168.1.201:6443 --token 2g9k0a.tsm6xe31rdb7jbo8 --discovery-token-ca-cert-hash sha256:d1d102ceb6241a3617777f6156cd4e86dc9f9edd9e1d6d73266d6ca7f6280890
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
error execution phase preflight: unable to fetch the kubeadm-config ConfigMap: failed to get config map: Unauthorized
To see the stack trace of this error execute with --v=5 or higher

原因分析：token過期了；

解決方法：重新生成不過期的token即可；

[root@k8s-master01 ~]# kubeadm token create --ttl 0 --print-join-command
W0819 12:00:27.541838 7855 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join 192.168.1.201:6443 --token 6xyv8a.cueltqmpe9qa8nxu --discovery-token-ca-cert-hash sha256:bd78dfd370e47dfca742b5f6934c21014792168fa4dc19c9fa63bfdd87270097

問題3：kubeadm方式部署K8S集群時，node狀態為NotReady，且coredns的pod狀態為pending，僅初始化的master節點kubeproxy為running？

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS     ROLES                  AGE   VERSION
k8s-master01   NotReady   control-plane,master   35m   v1.22.0
k8s-master02   NotReady   control-plane,master   32m   v1.22.0
k8s-master03   NotReady   control-plane,master   22m   v1.22.0
k8s-node01     NotReady   node                   32m   v1.22.0
k8s-node02     NotReady   node                   32m   v1.22.0
[root@k8s-master01 ~]# kubectl get pod -n kube-system
NAME                                   READY   STATUS             RESTARTS      AGE
coredns-7d89d9b6b8-ddjjp               0/1     Pending            0             35m
coredns-7d89d9b6b8-sbd8k               0/1     Pending            0             35m
etcd-k8s-master01                      1/1     Running            3 (18m ago)   35m
etcd-k8s-master02                      1/1     Running            0             32m
etcd-k8s-master03                      0/1     Running            1 (18m ago)   22m
kube-apiserver-k8s-master01            0/1     Running            2             34m
kube-apiserver-k8s-master02            0/1     Running            0             32m
kube-apiserver-k8s-master03            0/1     Running            0             22m
kube-controller-manager-k8s-master01   1/1     Running            4 (19m ago)   34m
kube-controller-manager-k8s-master02   1/1     Running            0             32m
kube-controller-manager-k8s-master03   1/1     Running            0             22m
kube-proxy-6wxkt                       0/1     Error              8 (21m ago)   32m
kube-proxy-9286f                       1/1     Running            8 (21m ago)   32m
kube-proxy-h9jcv                       0/1     CrashLoopBackOff   5 (19m ago)   22m
kube-proxy-v944j                       0/1     Error              8 (21m ago)   32m
kube-proxy-zfrrk                       0/1     Error              8 (23m ago)   35m
kube-scheduler-k8s-master01            1/1     Running            3 (32m ago)   35m
kube-scheduler-k8s-master02            1/1     Running            1 (19m ago)   32m
kube-scheduler-k8s-master03            1/1     Running            0             22m
[root@k8s-master01 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   35m

原因分析：網絡插件沒有安裝好。

解決方法：安裝網絡插件，如flannel、calico等。

問題4、kubeadm方式部署K8S集群時，部署calico組件失敗？

[root@k8s-master01 calico]# kubectl get pod -A -o wide | grep calico
kube-system   calico-kube-controllers-cdd5755b9-phg94   0/1     Running             0               6m5s   192.168.1.203   k8s-master03   <none>           <none>
kube-system   calico-node-cb6c6                         0/1     CrashLoopBackOff    5 (99s ago)     6m5s   192.168.1.202   k8s-master02   <none>           <none>
kube-system   calico-node-knr8g                         0/1     CrashLoopBackOff    5 (78s ago)     6m5s   192.168.1.203   k8s-master03   <none>           <none>
kube-system   calico-node-l4fp9                         0/1     CrashLoopBackOff    5 (98s ago)     6m5s   192.168.1.204   k8s-node01     <none>           <none>
kube-system   calico-node-mwqlt                         0/1     CrashLoopBackOff    5 (96s ago)     6m5s   192.168.1.205   k8s-node02     <none>           <none>
kube-system   calico-node-zjkz6                         0/1     CrashLoopBackOff    5 (93s ago)     6m5s   192.168.1.201   k8s-master01   <none>           <none>

原因分析：calico-node的yaml文件存在問題。

[root@k8s-master01 ~]# kubectl describe pod calico-node-cb6c6  -n kube-system
Name:                 calico-node-cb6c6
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 k8s-master02/192.168.1.202
Start Time:           Tue, 14 Sep 2021 17:14:28 +0800
Labels:               controller-revision-hash=b5f966fbb
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          <none>
Status:               Running
IP:                   192.168.1.202
IPs:
  IP:           192.168.1.202
Controlled By:  DaemonSet/calico-node
Init Containers:
  install-cni:
    Container ID:  docker://3e7d03495a0daa469b0f6b15acb451fa228c8d97a7d41fda0642d733cb21de9c
    Image:         registry.cn-beijing.aliyuncs.com/dotbalo/cni:v3.15.3
    Image ID:      docker-pullable://registry.cn-beijing.aliyuncs.com/dotbalo/cni@sha256:a559d264c7a75a7528560d11778dba2d3b55c588228aed99be401fd2baa9b607
    Port:          <none>
    Host Port:     <none>
    Command:
      /install-cni.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 14 Sep 2021 17:14:51 +0800
      Finished:     Tue, 14 Sep 2021 17:14:51 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      CNI_CONF_NAME:       10-calico.conflist
      CNI_NETWORK_CONFIG:  <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      ETCD_ENDPOINTS:      <set to the key 'etcd_endpoints' of config map 'calico-config'>      Optional: false
      CNI_MTU:             <set to the key 'veth_mtu' of config map 'calico-config'>            Optional: false
      SLEEP:               false
    Mounts:
      /calico-secrets from etcd-certs (rw)
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tfw5v (ro)
  flexvol-driver:
    Container ID:   docker://c8fe47bd4805c83fa3fad054470c3f3eb9dbccc1e64455cae085f050b11897c1
    Image:          registry.cn-beijing.aliyuncs.com/dotbalo/pod2daemon-flexvol:v3.15.3
    Image ID:       docker-pullable://registry.cn-beijing.aliyuncs.com/dotbalo/pod2daemon-flexvol@sha256:6bd1246d0ea1e573a6a050902995b1666ec0852339e5bda3051f583540361b55
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 14 Sep 2021 17:14:56 +0800
      Finished:     Tue, 14 Sep 2021 17:14:56 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tfw5v (ro)
Containers:
  calico-node:
    Container ID:   docker://7e209f5b426027b10aa3460b4bab2b003ad4a853f46e24353f119211abb42cee
    Image:          registry.cn-beijing.aliyuncs.com/dotbalo/node:v3.15.3
    Image ID:       docker-pullable://registry.cn-beijing.aliyuncs.com/dotbalo/node@sha256:988095dbe39d2066b1964aafaa4a302a1286b149a4a80c9a1eb85544f2a0cdd0
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 14 Sep 2021 17:21:43 +0800
      Finished:     Tue, 14 Sep 2021 17:21:45 +0800
    Ready:          False
    Restart Count:  6
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ETCD_ENDPOINTS:                     <set to the key 'etcd_endpoints' of config map 'calico-config'>  Optional: false
      ETCD_CA_CERT_FILE:                  <set to the key 'etcd_ca' of config map 'calico-config'>         Optional: false
      ETCD_KEY_FILE:                      <set to the key 'etcd_key' of config map 'calico-config'>        Optional: false
      ETCD_CERT_FILE:                     <set to the key 'etcd_cert' of config map 'calico-config'>       Optional: false
      CALICO_K8S_NODE_REF:                 (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_IPV4POOL_CIDR:               172.168.0.0/12
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGSEVERITYSCREEN:            info
      FELIX_HEALTHENABLED:                true
    Mounts:
      /calico-secrets from etcd-certs (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tfw5v (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  etcd-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-etcd-secrets
    Optional:    false
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  kube-api-access-tfw5v:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  7m19s                  default-scheduler  Successfully assigned kube-system/calico-node-cb6c6 to k8s-master02
  Normal   Pulling    7m19s                  kubelet            Pulling image "registry.cn-beijing.aliyuncs.com/dotbalo/cni:v3.15.3"
  Normal   Pulled     6m57s                  kubelet            Successfully pulled image "registry.cn-beijing.aliyuncs.com/dotbalo/cni:v3.15.3" in 22.396538132s
  Normal   Created    6m56s                  kubelet            Created container install-cni
  Normal   Started    6m56s                  kubelet            Started container install-cni
  Normal   Pulling    6m55s                  kubelet            Pulling image "registry.cn-beijing.aliyuncs.com/dotbalo/pod2daemon-flexvol:v3.15.3"
  Normal   Pulling    6m51s                  kubelet            Pulling image "registry.cn-beijing.aliyuncs.com/dotbalo/node:v3.15.3"
  Normal   Pulled     6m51s                  kubelet            Successfully pulled image "registry.cn-beijing.aliyuncs.com/dotbalo/pod2daemon-flexvol:v3.15.3" in 3.78191146s
  Normal   Created    6m51s                  kubelet            Created container flexvol-driver
  Normal   Started    6m51s                  kubelet            Started container flexvol-driver
  Normal   Pulled     6m12s                  kubelet            Successfully pulled image "registry.cn-beijing.aliyuncs.com/dotbalo/node:v3.15.3" in 39.059498113s
  Warning  Unhealthy  6m7s (x4 over 6m12s)   kubelet            Readiness probe failed: calico/node is not ready: BIRD is not ready: Failed to stat() nodename file: stat /var/lib/calico/nodename: no such file or directory
 Normal Created 5m49s (x3 over 6m12s) kubelet Created container calico-node
  Normal   Started    5m49s (x3 over 6m12s)  kubelet            Started container calico-node
  Normal   Pulled     5m49s (x2 over 6m10s)  kubelet            Container image "registry.cn-beijing.aliyuncs.com/dotbalo/node:v3.15.3" already present on machine
  Warning  BackOff    2m10s (x22 over 6m7s)  kubelet            Back-off restarting failed containerp

pod詳細信息顯示：Readiness probe failed導致重啟pod失敗。

[root@k8s-master01 ~]# kubectl logs calico-node-cb6c6  -n kube-system
2021-09-14 09:21:43.819 [INFO][8] startup/startup.go 349: Early log level set to info
2021-09-14 09:21:43.819 [INFO][8] startup/startup.go 369: Using HOSTNAME environment (lowercase) for node name
2021-09-14 09:21:43.819 [INFO][8] startup/startup.go 377: Determined node name: k8s-master02
2021-09-14 09:21:43.819 [INFO][8] startup/startup.go 109: Skipping datastore connection test
2021-09-14 09:21:43.830 [INFO][8] startup/startup.go 460: Building new node resource Name="k8s-master02"
2021-09-14 09:21:45.832 [ERROR][8] startup/startup.go 149: failed to query kubeadm's config map error=Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=2s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021-09-14 09:21:45.832 [WARNING][8] startup/startup.go 1264: Terminating
Calico node failed to start

pod日志顯示：未能成功訪問kubeadm's config導致calico啟動失敗。

解決方法：calico-node默認訪問kubernetes SVC的443端口，這將導致無法訪問apiserver，需要在yaml文件添加apiserver的IP和端口，字段名：KUBERNETES_SERVICE_HOST、KUBERNETES_SERVICE_PORT、KUBERNETES_SERVICE_PORT_HTTPS。

問題5：kubeadm方式部署K8S集群時，安裝calico網絡組件成功后，但coredns依然處於ContainerCreating？

[root@k8s-master01 .kube]# kubectl get pod -A -o wide
NAMESPACE     NAME                                      READY   STATUS              RESTARTS         AGE   IP              NODE           NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-cdd5755b9-phg94   0/1     Running             0                73m   192.168.1.203   k8s-master03   <none>           <none>
kube-system   calico-node-5d6xr                         1/1     Running             0                50m   192.168.1.202   k8s-master02   <none>           <none>
kube-system   calico-node-k7qmf                         1/1     Running             0                50m   192.168.1.204   k8s-node01     <none>           <none>
kube-system   calico-node-p47mm                         1/1     Running             0                50m   192.168.1.201   k8s-master01   <none>           <none>
kube-system   calico-node-v7q76                         1/1     Running             0                50m   192.168.1.203   k8s-master03   <none>           <none>
kube-system   calico-node-wjw4d                         1/1     Running             0                50m   192.168.1.205   k8s-node02     <none>           <none>
kube-system   coredns-7d89d9b6b8-ddjjp                  0/1     ContainerCreating   0                92m   <none>          k8s-master03   <none>           <none>
kube-system   coredns-7d89d9b6b8-sbd8k                  0/1     ContainerCreating   0                92m   <none>          k8s-master03   <none>           <none>

原因分析：

[root@k8s-master01 ~]# kubectl logs coredns-7d89d9b6b8-ddjjp -n kube-system
Error from server (BadRequest): container "coredns" in pod "coredns-7d89d9b6b8-ddjjp" is waiting to start: ContainerCreating

日志顯示：容器創建中。

[root@k8s-master01 ~]# kubectl describe pod coredns-7d89d9b6b8-ddjjp -n kube-system
Name:                 coredns-7d89d9b6b8-ddjjp
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 k8s-master03/192.168.1.203
Start Time:           Tue, 14 Sep 2021 17:14:54 +0800
Labels:               k8s-app=kube-dns
                      pod-template-hash=7d89d9b6b8
Annotations:          <none>
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-7d89d9b6b8
Containers:
  coredns:
    Container ID:  
    Image:         registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4
    Image ID:      
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7m9rx (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-7m9rx:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------

  Normal   SandboxChanged          6m21s (x2093 over 136m)  kubelet  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  82s (x2154 over 136m)    kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "089f15d2961982244b2ce122b738afc70e29fa17eb79a4e7b88a35ce5be4868b" network for pod "coredns-7d89d9b6b8-ddjjp": networkPlugin cni failed to set up pod "coredns-7d89d9b6b8-ddjjp_kube-system" network: Get "https://[10.96.0.1]:443/api/v1/namespaces/kube-system": dial tcp 10.96.0.1:443: connect: connection refused

pod詳細信息顯示：調度節點失敗。

[root@k8s-master01 .kube]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Tue 2021-09-14 16:56:06 CST; 1h 42min ago
     Docs: https://kubernetes.io/docs/
 Main PID: 28526 (kubelet)
    Tasks: 14
   Memory: 78.6M
   CGroup: /system.slice/kubelet.service
           └─28526 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --net...

Sep 14 18:37:41 k8s-master01 kubelet[28526]: I0914 18:37:41.703532   28526 scope.go:110] "RemoveContainer" containerID="a9e6b2c2df1bd9530394a4077af6c159830efd7e2b2d9156cbf1580086bf6d99"
Sep 14 18:37:41 k8s-master01 kubelet[28526]: E0914 18:37:41.703788   28526 pod_workers.go:747] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-proxy\" with ...
Sep 14 18:37:52 k8s-master01 kubelet[28526]: I0914 18:37:52.704002   28526 scope.go:110] "RemoveContainer" containerID="a9e6b2c2df1bd9530394a4077af6c159830efd7e2b2d9156cbf1580086bf6d99"
Sep 14 18:37:52 k8s-master01 kubelet[28526]: E0914 18:37:52.704281   28526 pod_workers.go:747] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-proxy\" with ...
Sep 14 18:38:07 k8s-master01 kubelet[28526]: I0914 18:38:07.704831   28526 scope.go:110] "RemoveContainer" containerID="a9e6b2c2df1bd9530394a4077af6c159830efd7e2b2d9156cbf1580086bf6d99"
Sep 14 18:38:07 k8s-master01 kubelet[28526]: E0914 18:38:07.705058   28526 pod_workers.go:747] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-proxy\" with ...
Sep 14 18:38:22 k8s-master01 kubelet[28526]: I0914 18:38:22.705015   28526 scope.go:110] "RemoveContainer" containerID="a9e6b2c2df1bd9530394a4077af6c159830efd7e2b2d9156cbf1580086bf6d99"
Sep 14 18:38:22 k8s-master01 kubelet[28526]: E0914 18:38:22.705279   28526 pod_workers.go:747] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-proxy\" with ...
Sep 14 18:38:34 k8s-master01 kubelet[28526]: I0914 18:38:34.709678   28526 scope.go:110] "RemoveContainer" containerID="a9e6b2c2df1bd9530394a4077af6c159830efd7e2b2d9156cbf1580086bf6d99"
Sep 14 18:38:34 k8s-master01 kubelet[28526]: E0914 18:38:34.711197   28526 pod_workers.go:747] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-proxy\" with ...
Hint: Some lines were ellipsized, use -l to show in full.

解決方法:修改kubelet服務配置即可。

問題5：kubeadm方式部署K8S集群時，部署flannel組件失敗？

kube-flannel-ds-amd64-8cqqz 0/1 CrashLoopBackOff 3 84s 192.168.66.10 k8s-master01 <none> <none>

原因分析：查看日志，發現注冊網絡失敗，原因在於主節點初始化時yaml文件存在問題。

kubectl logs kube-flannel-ds-amd64-8cqqz -n kubesystem

I0602 01:53:54.021093 1 main.go:514] Determining IP address of default interface
I0602 01:53:54.022514 1 main.go:527] Using interface with name ens33 and address 192.168.66.10
I0602 01:53:54.022619 1 main.go:544] Defaulting external address to interface address (192.168.66.10)
I0602 01:53:54.030311 1 kube.go:126] Waiting 10m0s for node controller to sync
I0602 01:53:54.030555 1 kube.go:309] Starting kube subnet manager
I0602 01:53:55.118656 1 kube.go:133] Node controller sync successful
I0602 01:53:55.118754 1 main.go:244] Created subnet manager: Kubernetes Subnet Manager - k8s-master01
I0602 01:53:55.118765 1 main.go:247] Installing signal handlers
I0602 01:53:55.119057 1 main.go:386] Found network config - Backend type: vxlan
I0602 01:53:55.119146 1 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
E0602 01:53:55.119470 1 main.go:289] Error registering network: failed to acquire lease: node "k8s-master01" pod cidr not assigned
I0602 01:53:55.119506 1 main.go:366] Stopping shutdownHandler...

解決方法：修改kubeadm-config.yaml后重新初始化主節點即可。

問題6：K8S集群初始化主節點失敗？

[init] Using Kubernetes version: v1.15.1

[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 18.09
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR Port-6443]: Port 6443 is in use
[ERROR Port-10251]: Port 10251 is in use
[ERROR Port-10252]: Port 10252 is in use
[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
[ERROR Port-10250]: Port 10250 is in use
[ERROR Port-2379]: Port 2379 is in use
[ERROR Port-2380]: Port 2380 is in use
[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

原因分析：K8S集群已進行過初始化主節點。

解決方法：重置K8S后重新初始化。

kubeadm reset

kubeadm init --config=kubeadm-config.yaml --upload-certs |tee kubeadm-init.log

問題7：重置K8S成功后，是否需要刪除相關文件？

[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0602 10:20:53.656954 76680 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

原因分析：無。

解決方法：可根據提示刪除相關文件，避免主節點初始化后引起其他問題。

問題8：主節點初始化成功后，查看節點信息失敗？

Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

原因分析：初始化主節點后，未導入kubelet的config文件。

解決方法：導入config文件即可。

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

問題9、master節點加入K8S集群失敗？

[root@k8s-master02 ~]# kubeadm join 192.168.1.201:6443 --token 6xyv8a.cueltqmpe9qa8nxu --discovery-token-ca-cert-hash sha256:bd78dfd370e47dfca742b5f6934c21014792168fa4dc19c9fa63bfdd87270097 \
> --control-plane --certificate-key b464a8d23d3313c4c0bb5b65648b039cb9b1177dddefbf46e2e296899d0e4516
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
error execution phase preflight:
One or more conditions for hosting a new control plane instance is not satisfied.

unable to add a new control plane instance a cluster that doesn't have a stable controlPlaneEndpoint address

Please ensure that:
* The cluster has a stable controlPlaneEndpoint address.
* The certificates that must be shared among control plane instances are provided.

To see the stack trace of this error execute with --v=5 or higher

原因分析：證書未共享。

解決方法：共享證書即可！！！

##########其他master節點執行如下內容#################################
mkdir -p /etc/kubernetes/pki/etcd/

##########master01節點執行如下內容#################################
cd /etc/kubernetes/pki/
scp ca.* front-proxy-ca.* sa.* 192.168.1.202:/etc/kubernetes/pki/
scp ca.* front-proxy-ca.* sa.* 192.168.1.203:/etc/kubernetes/pki/

##########其他master節點執行如下內容#################################
kubeadm join 192.168.1.201:6443 --token 6xyv8a.cueltqmpe9qa8nxu --discovery-token-ca-cert-hash sha256:bd78dfd370e47dfca742b5f6934c21014792168fa4dc19c9fa63bfdd87270097 --control-plane --certificate-key b464a8d23d3313c4c0bb5b65648b039cb9b1177dddefbf46e2e296899d0e4516

問題10、部署prometheus失敗？

unable to recognize "0prometheus-operator-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.co m/v1"

unable to recognize "alertmanager-alertmanager.yaml": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1"
unable to recognize "alertmanager-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "grafana-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "kube-state-metrics-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/ v1"
unable to recognize "node-exporter-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "prometheus-prometheus.yaml": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1"
unable to recognize "prometheus-rules.yaml": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"
unable to recognize "prometheus-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
unable to recognize "prometheus-serviceMonitorApiserver.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com /v1"
unable to recognize "prometheus-serviceMonitorCoreDNS.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v 1"
unable to recognize "prometheus-serviceMonitorKubeControllerManager.yaml": no matches for kind "ServiceMonitor" in version "monitorin g.coreos.com/v1"
unable to recognize "prometheus-serviceMonitorKubeScheduler.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos .com/v1"
unable to recognize "prometheus-serviceMonitorKubelet.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v 1"

原因分析：不明。

解決方法：重新執行發布命令。

問題7：無法下載K8S高可用安裝包，但能直接通過瀏覽器下載git包？

[root@k8s-master01 install-k8s-v1.17]# git clone https://github.com/dotbalo/k8s-ha-install.git

Cloning into 'k8s-ha-install'...
error: RPC failed; result=35, HTTP code = 0
fatal: The remote end hung up unexpectedly

原因分析： git buffer 太小了。

解決方法：git config --global http.postBuffer 100M #git buffer增大。

問題8：無法下載K8S高可用安裝包，但能直接通過瀏覽器下載git包？

[root@k8s-master01 install-k8s-v1.17]# git clone https://github.com/dotbalo/k8s-ha-install.git
Cloning into 'k8s-ha-install'...
fatal: unable to access 'https://github.com/dotbalo/k8s-ha-install.git/': Empty reply from server

原因分析：不明；

解決方法：手動下載並上傳git包；

問題9：K8S查看分支失敗？

[root@k8s-master01 k8s-ha-install-master]# git branch -a
fatal: Not a git repository (or any of the parent directories): .git

原因分析：缺少.git本地倉庫。

解決方法：初始化git即可。

[root@k8s-master01 k8s-ha-install-master]# git init
Initialized empty Git repository in /root/install-k8s-v1.17/k8s-ha-install-master/.git/
[root@k8s-master01 k8s-ha-install-master]# git branch -a

問題10：K8S切換分支失敗？

[root@k8s-master01 k8s-ha-install-master]# git checkout manual-installation-v1.20.x
error: pathspec 'manual-installation-v1.20.x' did not match any file(s) known to git.

原因分析：沒有發現分支。

解決方法：必須通過git下載，不能通過瀏覽器下載zip格式的文件；或將git下載的文件打包后再次解壓也不可使用。

git clone https://github.com/dotbalo/k8s-ha-install.git

[root@k8s-master01 k8s-ha-install]# git checkout manual-installation-v1.20.x

Branch manual-installation-v1.20.x set up to track remote branch manual-installation-v1.20.x from origin.
Switched to a new branch 'manual-installation-v1.20.x'

問題11：apiserver聚合證書生成失敗？

[root@k8s-master01 pki]# cfssl gencert \
> -ca=/etc/kubernetes/pki/front-proxy-ca.pem \
> -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes \
> front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client
2021/07/13 16:23:31 [INFO] generate received request
2021/07/13 16:23:31 [INFO] received CSR
2021/07/13 16:23:31 [INFO] generating key: rsa-2048
2021/07/13 16:23:31 [INFO] encoded CSR
2021/07/13 16:23:31 [INFO] signed certificate with serial number 723055319564838988438867282088171857759978642034
2021/07/13 16:23:31 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements")

原因分析：命令缺少hosts參數，不適合用於網站；但不影響apiserver組件與其他組件通信。

解決方法：無需關注。

問題12：二進制安裝K8S組件后，出現警告？

[root@k8s-master01 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

原因分析：kubectl組件安裝成功，但沒有成功連接kubernetes apiserver；

解決方法：安裝證書並配置組件即可。

問題13：二進制安裝K8S進行證書配置后，啟動ETCD失敗？

[root@k8s-master02 ~]# systemctl status etcd
● etcd.service - Etcd Service
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: activating (start) since Wed 2021-07-21 08:51:42 CST; 1min 13s ago
Docs: https://coreos.com/etcd/docs/latest/
Main PID: 1424 (etcd)
Tasks: 9
Memory: 33.1M
CGroup: /system.slice/etcd.service
└─1424 /usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml

Jul 21 08:52:55 k8s-master02 etcd[1424]: rejected connection from "192.168.0.107:38222" (error "remote error: tls: bad certificate", ServerName "")
Jul 21 08:52:55 k8s-master02 etcd[1424]: rejected connection from "192.168.0.107:38224" (error "remote error: tls: bad certificate", ServerName "")

原因分析：認證失敗；可能原因：創建證書的csr文件問題導致證書錯誤，IP未加入hosts中。

解決方法：重新生成證書即可，處理方法詳見問題21.

問題14：haproxy啟動失敗？

[root@k8s-master01 pki]# systemctl enable --now haproxy;systemctl status haproxy
Created symlink from /etc/systemd/system/multi-user.target.wants/haproxy.service to /usr/lib/systemd/system/haproxy.service.
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2021-07-21 10:25:35 CST; 3ms ago
Process: 2255 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid $OPTIONS (code=exited, status=1/FAILURE)
Main PID: 2255 (code=exited, status=1/FAILURE)

Jul 21 10:25:35 k8s-master01 haproxy-systemd-wrapper[2255]: [ALERT] 201/102535 (2256) : parsing [/etc/haproxy/haproxy.cfg:59] : 'server 192.168.0.107:6443' : invalid address...in 'check'
Jul 21 10:25:35 k8s-master01 haproxy-systemd-wrapper[2255]: [ALERT] 201/102535 (2256) : parsing [/etc/haproxy/haproxy.cfg:60] : 'server 192.168.0.108:6443' : invalid address...in 'check'
Jul 21 10:25:35 k8s-master01 haproxy-systemd-wrapper[2255]: [ALERT] 201/102535 (2256) : parsing [/etc/haproxy/haproxy.cfg:61] : 'server 192.168.0.109:6443' : invalid address...in 'check'
Jul 21 10:25:35 k8s-master01 haproxy-systemd-wrapper[2255]: [ALERT] 201/102535 (2256) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Jul 21 10:25:35 k8s-master01 haproxy-systemd-wrapper[2255]: [WARNING] 201/102535 (2256) : config : frontend 'GLOBAL' has no 'bind' directive. Please declare it as a backend ... intended.
Jul 21 10:25:35 k8s-master01 haproxy-systemd-wrapper[2255]: [ALERT] 201/102535 (2256) : Fatal errors found in configuration.
Jul 21 10:25:35 k8s-master01 systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE
Jul 21 10:25:35 k8s-master01 haproxy-systemd-wrapper[2255]: haproxy-systemd-wrapper: exit, haproxy RC=1
Jul 21 10:25:35 k8s-master01 systemd[1]: Unit haproxy.service entered failed state.
Jul 21 10:25:35 k8s-master01 systemd[1]: haproxy.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

原因分析：配置錯誤。

解決方法：添加option tcp-check即可。

問題15：apiserver啟動失敗？

[root@k8s-master01 ~]# systemctl daemon-reload && systemctl enable --now kube-apiserver
Failed to execute operation: No such file or directory

原因分析：配置問題，etcd不能正常工作原因。

解決方法：修改配置即可。

問題16：kubelet服務啟動失敗？

● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Fri 2021-08-13 12:15:11 CST; 948ms ago
Docs: https://kubernetes.io/docs/
Process: 5600 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
Main PID: 5600 (code=exited, status=1/FAILURE)

Aug 13 12:15:11 k8s-master02 systemd[1]: Unit kubelet.service entered failed state.
Aug 13 12:15:11 k8s-master02 systemd[1]: kubelet.service failed.

原因分析：selinux未關閉，且未進行初始化；

解決方法：關閉selinux並進行初始化。

問題17：k8s主節點初始化失敗？

[root@k8s-master01 ~]# kubeadm init --control-plane-endpoint "k8s-master01-lb:16443" --upload-certs
I0813 13:49:07.164695 10196 version.go:254] remote version is much newer: v1.22.0; falling back to: stable-1.20
[init] Using Kubernetes version: v1.20.10
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.22.0" Control plane version: "1.20.10"
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

原因分析：kubeadm版本不一致；

解決方法：重新安裝kubeadm即可。

yum -y remove kubectl kubelet && yum -y install kubectl-1.20.5-0 kubelet-1.20.5-0 kubeadm-1.20.5-0

問題18：kubeadm安裝k8s，初始化后無法查看node狀態？

[root@k8s-master01 ~]# kubectl get node
The connection to the server localhost:8080 was refused - did you specify the right host or port?

原因分析：kubectl無法與K8S集群通信；

解決方法：.kube目錄是用於存放kubectl與K8S集群交互的緩存文件及相關配置文件；

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

問題20：calico安裝異常？

[root@k8s-master01 calico]# kubectl describe pod calico-node-dtqc8 -n kube-system
Name: calico-node-dtqc8
Namespace: kube-system

......

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned kube-system/calico-node-dtqc8 to k8s-master01
Normal Pulling 10m kubelet Pulling image "registry.cn-beijing.aliyuncs.com/dotbalo/cni:v3.15.3"
Normal Pulled 10m kubelet Successfully pulled image "registry.cn-beijing.aliyuncs.com/dotbalo/cni:v3.15.3" in 44.529354646s
Normal Created 10m kubelet Created container install-cni
Normal Started 10m kubelet Started container install-cni
Normal Pulling 10m kubelet Pulling image "registry.cn-beijing.aliyuncs.com/dotbalo/pod2daemon-flexvol:v3.15.3"
Normal Pulled 9m45s kubelet Successfully pulled image "registry.cn-beijing.aliyuncs.com/dotbalo/pod2daemon-flexvol:v3.15.3" in 27.031642012s
Normal Created 9m45s kubelet Created container flexvol-driver
Normal Started 9m44s kubelet Started container flexvol-driver
Normal Pulling 9m43s kubelet Pulling image "registry.cn-beijing.aliyuncs.com/dotbalo/node:v3.15.3"
Normal Pulled 8m42s kubelet Successfully pulled image "registry.cn-beijing.aliyuncs.com/dotbalo/node:v3.15.3" in 1m1.78614215s
Normal Created 8m41s kubelet Created container calico-node
Normal Started 8m41s kubelet Started container calico-node
Warning Unhealthy 8m39s kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

原因分析：不明。

解決方法：暫無影響，可不理會。

問題21：kubectl工具不能使用？

[root@k8s-master01 ~]# kubectl get nodes
The connection to the server 192.168.1.201:6443 was refused - did you specify the right host or port?

原因分析：未設置環境變量；

解決方法：

cat <<eof>> /root/.bashrc
export KUBECONFIG=/etc/kubernetes/admin.conf
eof
source /root/.bashrc

問題22：第一個master節點初始化失敗？

[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.

Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

原因分析：不明；

解決方法：待解決；

問題23、helm安裝ingress失敗？

[root@k8s-master01 ingress-nginx]# helm install ingress-nginx -n ingress-nginx .
Error: cannot re-use a name that is still in use

原因分析：安裝失敗后，緩存記錄了ingress-nginx；

解決方法：刪除記錄即可。

[root@k8s-master01 ingress-nginx]# helm uninstall ingress-nginx -n ingress-nginx
Error: uninstall: Release not loaded: ingress-nginx: release: not found
[root@k8s-master01 ingress-nginx]# 
[root@k8s-master01 ingress-nginx]# 
[root@k8s-master01 ingress-nginx]# helm install ingress-nginx -n ingress-nginx .
NAME: ingress-nginx
LAST DEPLOYED: Tue Aug 31 16:00:45 2021
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
Get the application URL by running these commands:
  export POD_NAME=$(kubectl --namespace ingress-nginx get pods -o jsonpath="{.items[0].metadata.name}" -l "app=ingress-nginx,component=,release=ingress-nginx")
  kubectl --namespace ingress-nginx port-forward $POD_NAME 8080:80
  echo "Visit http://127.0.0.1:8080 to access your application."

An example Ingress that makes use of the controller:

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

問題24：helm安裝ingress失敗，進入調試模式查看原因？

[root@k8s-master01 ingress-nginx]# helm install ingress-nginx -n ingress-nginx . --debug
install.go:148: [debug] Original chart version: ""
install.go:165: [debug] CHART PATH: /root/ingress-nginx

Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
helm.go:76: [debug] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
could not get apiVersions from Kubernetes
helm.sh/helm/v3/pkg/action.(*Configuration).getCapabilities
/home/circleci/helm.sh/helm/pkg/action/action.go:114
helm.sh/helm/v3/pkg/action.(*Install).Run
/home/circleci/helm.sh/helm/pkg/action/install.go:195
main.runInstall
/home/circleci/helm.sh/helm/cmd/helm/install.go:209
main.newInstallCmd.func1
/home/circleci/helm.sh/helm/cmd/helm/install.go:115
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
/home/circleci/helm.sh/helm/cmd/helm/helm.go:75
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357

原因分析：存在錯誤的apiservice；

解決方法：刪除錯誤的apiservice即可。

問題25：helm安裝ingress失敗，進入調試模式查看原因？

[root@k8s-master01 ingress-nginx]# helm install ingress-nginx -n ingress-nginx . --debug
install.go:148: [debug] Original chart version: ""
install.go:165: [debug] CHART PATH: /root/ingress-nginx

Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: ConfigMap, namespace: ingress-nginx, name: ingress-nginx-controller
helm.go:76: [debug] existing resource conflict: kind: ConfigMap, namespace: ingress-nginx, name: ingress-nginx-controller
rendered manifests contain a resource that already exists. Unable to continue with install
helm.sh/helm/v3/pkg/action.(*Install).Run
/home/circleci/helm.sh/helm/pkg/action/install.go:242
main.runInstall
/home/circleci/helm.sh/helm/cmd/helm/install.go:209
main.newInstallCmd.func1
/home/circleci/helm.sh/helm/cmd/helm/install.go:115
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
main.main
/home/circleci/helm.sh/helm/cmd/helm/helm.go:75
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357

[root@k8s-master01 ingress-nginx]# helm install ingress-nginx -n ingress-nginx . --debug
install.go:148: [debug] Original chart version: ""
install.go:165: [debug] CHART PATH: /root/ingress-nginx

Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: ServiceAccount, namespace: ingress-nginx, name: ingress-nginx
。。。

Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: ClusterRole, namespace: , name: ingress-nginx
helm.go:76: [debug] existing resource conflict: kind: ClusterRole, namespace: , name: ingress-nginx
rendered manifests contain a resource that already exists. Unable to continue with install

。。。

原因分析：已存在configmap等資源；

解決方法：刪除configmap等資源即可；

kubectl delete ConfigMap ingress-nginx-controller -n ingress-nginx

kubectl delete ServiceAccount ingress-nginx -n ingress-nginx

kubectl delete ClusterRole ingress-nginx

kubectl delete ClusterRoleBinding ingress-nginx

kubectl delete Role ingress-nginx -n ingress-nginx

kubectl delete RoleBinding ingress-nginx -n ingress-nginx

kubectl delete svc ingress-nginx-controller ingress-nginx-controller-admission -n ingress-nginx

kubectl delete DaemonSet ingress-nginx-controller -n ingress-nginx

kubectl delete ValidatingWebhookConfiguration ingress-nginx-admission

問題26：etcd集群啟動失敗？

[root@k8s-master01 ~]# systemctl status etcd
● etcd.service - Etcd Service
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: activating (start) since Mon 2021-09-06 20:03:22 CST; 47s ago
Docs: https://coreos.com/etcd/docs/latest/
Main PID: 1497 (etcd)
Tasks: 8
Memory: 23.2M
CGroup: /system.slice/etcd.service
└─1497 /usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml

Sep 06 20:04:07 k8s-master01 etcd[1497]: raft2021/09/06 20:04:07 INFO: c05e87400fe17139 is starting a new election at term 928
Sep 06 20:04:07 k8s-master01 etcd[1497]: raft2021/09/06 20:04:07 INFO: c05e87400fe17139 became candidate at term 929
Sep 06 20:04:07 k8s-master01 etcd[1497]: raft2021/09/06 20:04:07 INFO: c05e87400fe17139 received MsgVoteResp from c05e87400fe17139 at term 929
Sep 06 20:04:07 k8s-master01 etcd[1497]: raft2021/09/06 20:04:07 INFO: c05e87400fe17139 [logterm: 1, index: 3] sent MsgVote request to 208ae89f369427bb at term 929
Sep 06 20:04:07 k8s-master01 etcd[1497]: raft2021/09/06 20:04:07 INFO: c05e87400fe17139 [logterm: 1, index: 3] sent MsgVote request to d2cb62beee1ff93f at term 929
Sep 06 20:04:09 k8s-master01 etcd[1497]: raft2021/09/06 20:04:09 INFO: c05e87400fe17139 is starting a new election at term 929
Sep 06 20:04:09 k8s-master01 etcd[1497]: raft2021/09/06 20:04:09 INFO: c05e87400fe17139 became candidate at term 930
Sep 06 20:04:09 k8s-master01 etcd[1497]: raft2021/09/06 20:04:09 INFO: c05e87400fe17139 received MsgVoteResp from c05e87400fe17139 at term 930
Sep 06 20:04:09 k8s-master01 etcd[1497]: raft2021/09/06 20:04:09 INFO: c05e87400fe17139 [logterm: 1, index: 3] sent MsgVote request to 208ae89f369427bb at term 930
Sep 06 20:04:09 k8s-master01 etcd[1497]: raft2021/09/06 20:04:09 INFO: c05e87400fe17139 [logterm: 1, index: 3] sent MsgVote request to d2cb62beee1ff93f at term 930

[root@k8s-master02 ~]# systemctl status etcd
● etcd.service - Etcd Service
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: timeout) since Mon 2021-09-06 20:06:34 CST; 6s ago
Docs: https://coreos.com/etcd/docs/latest/
Process: 1531 ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml (code=killed, signal=TERM)
Main PID: 1531 (code=killed, signal=TERM)

Sep 06 20:06:34 k8s-master02 systemd[1]: Failed to start Etcd Service.
Sep 06 20:06:34 k8s-master02 systemd[1]: Unit etcd.service entered failed state.
Sep 06 20:06:34 k8s-master02 systemd[1]: etcd.service failed.

[root@k8s-master03 ~]# systemctl status etcd
● etcd.service - Etcd Service
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2021-09-06 20:13:50 CST; 1s ago
Docs: https://coreos.com/etcd/docs/latest/
Process: 1781 ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml (code=killed, signal=TERM)
Main PID: 1781 (code=killed, signal=TERM)

Sep 06 20:13:49 k8s-master03 etcd[1781]: raft2021/09/06 20:13:49 INFO: d2cb62beee1ff93f became candidate at term 1247
Sep 06 20:13:49 k8s-master03 etcd[1781]: raft2021/09/06 20:13:49 INFO: d2cb62beee1ff93f received MsgVoteResp from d2cb62beee1ff93f at term 1247
Sep 06 20:13:49 k8s-master03 etcd[1781]: raft2021/09/06 20:13:49 INFO: d2cb62beee1ff93f [logterm: 1, index: 3] sent MsgVote request to 208ae89f369427bb at term 1247
Sep 06 20:13:49 k8s-master03 etcd[1781]: raft2021/09/06 20:13:49 INFO: d2cb62beee1ff93f [logterm: 1, index: 3] sent MsgVote request to c05e87400fe17139 at term 1247
Sep 06 20:13:49 k8s-master03 etcd[1781]: health check for peer 208ae89f369427bb could not connect: dial tcp 192.168.1.208:2380: i/o timeout
Sep 06 20:13:49 k8s-master03 etcd[1781]: health check for peer c05e87400fe17139 could not connect: dial tcp 192.168.1.207:2380: i/o timeout
Sep 06 20:13:49 k8s-master03 etcd[1781]: health check for peer 208ae89f369427bb could not connect: dial tcp 192.168.1.208:2380: i/o timeout
Sep 06 20:13:49 k8s-master03 etcd[1781]: health check for peer c05e87400fe17139 could not connect: dial tcp 192.168.1.207:2380: i/o timeout
Sep 06 20:13:50 k8s-master03 etcd[1781]: publish error: etcdserver: request timed out

原因分析：證書與配置出現問題，開啟了高可用服務；同時etcd2|etcd3未啟動，導致etcd1啟動失敗。

Sep 6 20:46:33 k8s-master02 Keepalived_vrrp[1036]: /etc/keepalived/check_apiserver.sh exited with status 2
Sep 6 20:46:33 k8s-master02 etcd: raft2021/09/06 20:46:33 INFO: 208ae89f369427bb is starting a new election at term 2231
Sep 6 20:46:33 k8s-master02 etcd: raft2021/09/06 20:46:33 INFO: 208ae89f369427bb became candidate at term 2232
Sep 6 20:46:33 k8s-master02 etcd: raft2021/09/06 20:46:33 INFO: 208ae89f369427bb received MsgVoteResp from 208ae89f369427bb at term 2232
Sep 6 20:46:33 k8s-master02 etcd: raft2021/09/06 20:46:33 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to c05e87400fe17139 at term 2232
Sep 6 20:46:33 k8s-master02 etcd: raft2021/09/06 20:46:33 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to d2cb62beee1ff93f at term 2232
Sep 6 20:46:34 k8s-master02 etcd: raft2021/09/06 20:46:34 INFO: 208ae89f369427bb is starting a new election at term 2232
Sep 6 20:46:34 k8s-master02 etcd: raft2021/09/06 20:46:34 INFO: 208ae89f369427bb became candidate at term 2233
Sep 6 20:46:34 k8s-master02 etcd: raft2021/09/06 20:46:34 INFO: 208ae89f369427bb received MsgVoteResp from 208ae89f369427bb at term 2233
Sep 6 20:46:34 k8s-master02 etcd: raft2021/09/06 20:46:34 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to c05e87400fe17139 at term 2233
Sep 6 20:46:34 k8s-master02 etcd: raft2021/09/06 20:46:34 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to d2cb62beee1ff93f at term 2233
Sep 6 20:46:36 k8s-master02 etcd: raft2021/09/06 20:46:36 INFO: 208ae89f369427bb is starting a new election at term 2233
Sep 6 20:46:36 k8s-master02 etcd: raft2021/09/06 20:46:36 INFO: 208ae89f369427bb became candidate at term 2234
Sep 6 20:46:36 k8s-master02 etcd: raft2021/09/06 20:46:36 INFO: 208ae89f369427bb received MsgVoteResp from 208ae89f369427bb at term 2234
Sep 6 20:46:36 k8s-master02 etcd: raft2021/09/06 20:46:36 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to c05e87400fe17139 at term 2234
Sep 6 20:46:36 k8s-master02 etcd: raft2021/09/06 20:46:36 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to d2cb62beee1ff93f at term 2234
Sep 6 20:46:37 k8s-master02 etcd: raft2021/09/06 20:46:37 INFO: 208ae89f369427bb is starting a new election at term 2234
Sep 6 20:46:37 k8s-master02 etcd: raft2021/09/06 20:46:37 INFO: 208ae89f369427bb became candidate at term 2235
Sep 6 20:46:37 k8s-master02 etcd: raft2021/09/06 20:46:37 INFO: 208ae89f369427bb received MsgVoteResp from 208ae89f369427bb at term 2235
Sep 6 20:46:37 k8s-master02 etcd: raft2021/09/06 20:46:37 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to c05e87400fe17139 at term 2235
Sep 6 20:46:37 k8s-master02 etcd: raft2021/09/06 20:46:37 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to d2cb62beee1ff93f at term 2235
Sep 6 20:46:38 k8s-master02 etcd: health check for peer c05e87400fe17139 could not connect: dial tcp 192.168.1.207:2380: connect: no route to host
Sep 6 20:46:38 k8s-master02 etcd: health check for peer c05e87400fe17139 could not connect: dial tcp 192.168.1.207:2380: connect: no route to host
Sep 6 20:46:38 k8s-master02 etcd: health check for peer d2cb62beee1ff93f could not connect: dial tcp 192.168.1.209:2380: connect: no route to host
Sep 6 20:46:38 k8s-master02 etcd: health check for peer d2cb62beee1ff93f could not connect: dial tcp 192.168.1.209:2380: connect: no route to host
Sep 6 20:46:38 k8s-master02 Keepalived_vrrp[1036]: /etc/keepalived/check_apiserver.sh exited with status 2
Sep 6 20:46:39 k8s-master02 etcd: raft2021/09/06 20:46:39 INFO: 208ae89f369427bb is starting a new election at term 2235
Sep 6 20:46:39 k8s-master02 etcd: raft2021/09/06 20:46:39 INFO: 208ae89f369427bb became candidate at term 2236
Sep 6 20:46:39 k8s-master02 etcd: raft2021/09/06 20:46:39 INFO: 208ae89f369427bb received MsgVoteResp from 208ae89f369427bb at term 2236
Sep 6 20:46:39 k8s-master02 etcd: raft2021/09/06 20:46:39 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to c05e87400fe17139 at term 2236
Sep 6 20:46:39 k8s-master02 etcd: raft2021/09/06 20:46:39 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to d2cb62beee1ff93f at term 2236
Sep 6 20:46:40 k8s-master02 etcd: publish error: etcdserver: request timed out
Sep 6 20:46:41 k8s-master02 etcd: raft2021/09/06 20:46:41 INFO: 208ae89f369427bb is starting a new election at term 2236
Sep 6 20:46:41 k8s-master02 etcd: raft2021/09/06 20:46:41 INFO: 208ae89f369427bb became candidate at term 2237
Sep 6 20:46:41 k8s-master02 etcd: raft2021/09/06 20:46:41 INFO: 208ae89f369427bb received MsgVoteResp from 208ae89f369427bb at term 2237
Sep 6 20:46:41 k8s-master02 etcd: raft2021/09/06 20:46:41 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to c05e87400fe17139 at term 2237
Sep 6 20:46:41 k8s-master02 etcd: raft2021/09/06 20:46:41 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to d2cb62beee1ff93f at term 2237
Sep 6 20:46:42 k8s-master02 etcd: raft2021/09/06 20:46:42 INFO: 208ae89f369427bb is starting a new election at term 2237
Sep 6 20:46:42 k8s-master02 etcd: raft2021/09/06 20:46:42 INFO: 208ae89f369427bb became candidate at term 2238
Sep 6 20:46:42 k8s-master02 etcd: raft2021/09/06 20:46:42 INFO: 208ae89f369427bb received MsgVoteResp from 208ae89f369427bb at term 2238
Sep 6 20:46:42 k8s-master02 etcd: raft2021/09/06 20:46:42 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to c05e87400fe17139 at term 2238
Sep 6 20:46:42 k8s-master02 etcd: raft2021/09/06 20:46:42 INFO: 208ae89f369427bb [logterm: 1, index: 3] sent MsgVote request to d2cb62beee1ff93f at term 2238
Sep 6 20:46:43 k8s-master02 etcd: health check for peer c05e87400fe17139 could not connect: dial tcp 192.168.1.207:2380: i/o timeout
Sep 6 20:46:43 k8s-master02 etcd: health check for peer c05e87400fe17139 could not connect: dial tcp 192.168.1.207:2380: i/o timeout
Sep 6 20:46:43 k8s-master02 etcd: health check for peer d2cb62beee1ff93f could not connect: dial tcp 192.168.1.209:2380: i/o timeout
Sep 6 20:46:43 k8s-master02 etcd: health check for peer d2cb62beee1ff93f could not connect: dial tcp 192.168.1.209:2380: i/o timeout
Sep 6 20:46:43 k8s-master02 Keepalived_vrrp[1036]: /etc/keepalived/check_apiserver.sh exited with status 2
Sep 6 20:46:44 k8s-master02 etcd: raft2021/09/06 20:46:44 INFO: 208ae89f369427bb is starting a new election at term 2238

Sep 6 21:22:49 k8s-master03 etcd: raft2021/09/06 21:22:49 INFO: 9eb1f144b60ad8c6 became candidate at term 64
Sep 6 21:22:49 k8s-master03 etcd: raft2021/09/06 21:22:49 INFO: 9eb1f144b60ad8c6 received MsgVoteResp from 9eb1f144b60ad8c6 at term 64
Sep 6 21:22:49 k8s-master03 etcd: raft2021/09/06 21:22:49 INFO: 9eb1f144b60ad8c6 [logterm: 1, index: 3] sent MsgVote request to 7f890ac3cab00125 at term 64
Sep 6 21:22:49 k8s-master03 etcd: raft2021/09/06 21:22:49 INFO: 9eb1f144b60ad8c6 [logterm: 1, index: 3] sent MsgVote request to ef155f75af77a2e6 at term 64
Sep 6 21:22:49 k8s-master03 etcd: rejected connection from "192.168.1.202:37322" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:49 k8s-master03 etcd: rejected connection from "192.168.1.202:37324" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:49 k8s-master03 etcd: rejected connection from "192.168.1.202:37330" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:49 k8s-master03 etcd: rejected connection from "192.168.1.202:37332" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:49 k8s-master03 etcd: rejected connection from "192.168.1.202:37338" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:49 k8s-master03 etcd: rejected connection from "192.168.1.202:37340" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:49 k8s-master03 etcd: rejected connection from "192.168.1.202:37346" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:49 k8s-master03 etcd: rejected connection from "192.168.1.202:37348" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37354" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37356" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37364" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37362" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37370" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37372" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37380" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37382" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: health check for peer 7f890ac3cab00125 could not connect: dial tcp 192.168.1.201:2380: connect: connection refused
Sep 6 21:22:50 k8s-master03 etcd: health check for peer ef155f75af77a2e6 could not connect: x509: certificate is valid for 127.0.0.1, 192.168.1.206, 192.168.1.207, 192.168.1.208, not 192.168.1.202
Sep 6 21:22:50 k8s-master03 etcd: health check for peer 7f890ac3cab00125 could not connect: dial tcp 192.168.1.201:2380: connect: connection refused
Sep 6 21:22:50 k8s-master03 etcd: health check for peer ef155f75af77a2e6 could not connect: x509: certificate is valid for 127.0.0.1, 192.168.1.206, 192.168.1.207, 192.168.1.208, not 192.168.1.202
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37388" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 etcd: rejected connection from "192.168.1.202:37390" (error "remote error: tls: bad certificate", ServerName "")
Sep 6 21:22:50 k8s-master03 systemd: etcd.service start operation timed out. Terminating.
Sep 6 21:22:50 k8s-master03 systemd: Failed to start Etcd Service.
Sep 6 21:22:50 k8s-master03 systemd: Unit etcd.service entered failed state.
Sep 6 21:22:50 k8s-master03 systemd: etcd.service failed.

解決方法：

重新生成證書，如下圖：

cfssl gencert \
-ca=/etc/etcd/ssl/etcd-ca.pem \
-ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
-config=ca-config.json \
-hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.1.201,192.168.1.202,192.168.1.203,192.168.1.206,192.168.1.207,192.168.1.208,192.168.1.209 \
-profile=kubernetes \
etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd #ip與域名對應，都必須寫；

systemctl stop etcd && rm -rf /var/lib/etcd/* && ll /var/lib/etcd/ #停止etcd服務，刪除etcd運行時產生的文件；同時嚴禁高可用服務開啟；

tail /var/log/messages

Sep 6 22:21:41 k8s-master01 systemd: Started Etcd Service.
Sep 6 22:21:41 k8s-master01 etcd: serving client requests on 127.0.0.1:2379
Sep 6 22:21:41 k8s-master01 etcd: ready to serve client requests
Sep 6 22:21:41 k8s-master01 etcd: serving client requests on 192.168.1.201:2379
Sep 6 22:21:41 k8s-master01 etcd: established a TCP streaming connection with peer ef155f75af77a2e6 (stream Message writer)
Sep 6 22:21:41 k8s-master01 etcd: established a TCP streaming connection with peer ef155f75af77a2e6 (stream MsgApp v2 writer)
Sep 6 22:21:44 k8s-master01 etcd: updated the cluster version from 3.0 to 3.4
Sep 6 22:21:44 k8s-master01 etcd: enabled capabilities for version 3.4
Sep 6 22:30:01 k8s-master01 systemd: Started Session 16 of user root.

問題27：某個node的etcd服務正常運行，但提示rpc error？

[root@k8s-master01 pki]# ETCDCTL_API=3 etcdctl member list
{"level":"warn","ts":"2021-09-06T22:58:58.455+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-40bd4115-87ef-4e13-8d82-28ede2ed9ff3/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection closed"}
Error: context deadline exceeded

原因分析：127.0.0.12379遠程過程調用error不影響etcd集群使用，但會降低性能。

解決方法：可不理會。

問題28：kube-apiserver無法啟動？

[root@k8s-master01 ~]# systemctl enable --now kube-apiserver
Failed to execute operation: No such file or directory

原因分析：service文件放錯位置。

I0907 10:13:14.991428 6008 store.go:1376] Monitoring limitranges count at <storage-prefix>//limitranges
I0907 10:13:14.997097 6008 cacher.go:405] cacher (*core.LimitRange): initialized
I0907 10:13:14.999869 6008 client.go:360] parsed scheme: "endpoint"
I0907 10:13:14.999947 6008 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 10:13:15.008518 6008 store.go:1376] Monitoring resourcequotas count at <storage-prefix>//resourcequotas
I0907 10:13:15.009517 6008 client.go:360] parsed scheme: "endpoint"
I0907 10:13:15.009542 6008 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 10:13:15.027691 6008 store.go:1376] Monitoring secrets count at <storage-prefix>//secrets
I0907 10:13:15.028338 6008 client.go:360] parsed scheme: "endpoint"
I0907 10:13:15.028366 6008 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 10:13:15.031897 6008 cacher.go:405] cacher (*core.ResourceQuota): initialized
I0907 10:13:15.032423 6008 cacher.go:405] cacher (*core.Secret): initialized
I0907 10:13:15.040312 6008 store.go:1376] Monitoring persistentvolumes count at <storage-prefix>//persistentvolumes
I0907 10:13:15.041293 6008 client.go:360] parsed scheme: "endpoint"
I0907 10:13:15.041340 6008 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 10:13:15.049139 6008 cacher.go:405] cacher (*core.PersistentVolume): initialized
I0907 10:13:15.049286 6008 store.go:1376] Monitoring persistentvolumeclaims count at <storage-prefix>//persistentvolumeclaims
I0907 10:13:15.050411 6008 client.go:360] parsed scheme: "endpoint"
I0907 10:13:15.050509 6008 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 10:13:15.052529 6008 cacher.go:405] cacher (*core.PersistentVolumeClaim): initialized
I0907 10:13:15.057764 6008 store.go:1376] Monitoring configmaps count at <storage-prefix>//configmaps
I0907 10:13:15.058496 6008 client.go:360] parsed scheme: "endpoint"
I0907 10:13:15.058550 6008 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 10:13:15.065377 6008 cacher.go:405] cacher (*core.ConfigMap): initialized
I0907 10:13:15.067289 6008 store.go:1376] Monitoring namespaces count at <storage-prefix>//namespaces
I0907 10:13:15.068480 6008 client.go:360] parsed scheme: "endpoint"
I0907 10:13:15.068510 6008 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 10:13:15.077229 6008 cacher.go:405] cacher (*core.Namespace): initialized
I0907 10:13:15.077465 6008 store.go:1376] Monitoring endpoints count at <storage-prefix>//services/endpoints
I0907 10:13:15.078124 6008 client.go:360] parsed scheme: "endpoint"
I0907 10:13:15.078166 6008 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 10:13:15.084988 6008 cacher.go:405] cacher (*core.Endpoints): initialized
I0907 10:13:15.087716 6008 store.go:1376] Monitoring nodes count at <storage-prefix>//minions
Error: error building core storage: open /etc/kubenetes/pki/apiserver-key.pem: no such file or directory

解決方法：修改配置文件位置即可。

問題29：kube-apiserver無法啟動？

I0907 11:10:10.104401 7577 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/etc/kubernetes/pki/ca.pem"
I0907 11:10:10.104462 7577 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "request-header::/etc/kubernetes/pki/front-proxy-ca.pem"
I0907 11:10:10.105150 7577 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0907 11:10:10.105167 7577 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0907 11:10:10.108787 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.108840 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.118716 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.118776 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.127421 7577 store.go:1376] Monitoring customresourcedefinitions.apiextensions.k8s.io count at <storage-prefix>//apiextensions.k8s.io/customresourcedefinitions
I0907 11:10:10.128585 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.128648 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.128859 7577 client.go:360] parsed scheme: "passthrough"
I0907 11:10:10.128920 7577 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.1.201:2379 <nil> 0 <nil>}] <nil> <nil>}
I0907 11:10:10.129014 7577 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0907 11:10:10.129173 7577 client.go:360] parsed scheme: "passthrough"
I0907 11:10:10.129187 7577 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.1.202:2379 <nil> 0 <nil>}] <nil> <nil>}
I0907 11:10:10.129191 7577 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0907 11:10:10.129357 7577 client.go:360] parsed scheme: "passthrough"
I0907 11:10:10.129378 7577 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.1.203:2379 <nil> 0 <nil>}] <nil> <nil>}
I0907 11:10:10.129384 7577 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0907 11:10:10.130065 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000609350, {CONNECTING <nil>}
I0907 11:10:10.130106 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0006095c0, {CONNECTING <nil>}
I0907 11:10:10.130158 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000609810, {CONNECTING <nil>}
I0907 11:10:10.141244 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0006095c0, {READY <nil>}
I0907 11:10:10.141914 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000609810, {READY <nil>}
I0907 11:10:10.142132 7577 store.go:1376] Monitoring customresourcedefinitions.apiextensions.k8s.io count at <storage-prefix>//apiextensions.k8s.io/customresourcedefinitions
I0907 11:10:10.143041 7577 cacher.go:405] cacher (*apiextensions.CustomResourceDefinition): initialized
I0907 11:10:10.144261 7577 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0907 11:10:10.145268 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000609350, {READY <nil>}
I0907 11:10:10.145679 7577 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0907 11:10:10.159351 7577 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0907 11:10:10.159498 7577 cacher.go:405] cacher (*apiextensions.CustomResourceDefinition): initialized
I0907 11:10:10.173307 7577 instance.go:289] Using reconciler: lease
I0907 11:10:10.173948 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.173992 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.184871 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.184916 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.193173 7577 store.go:1376] Monitoring podtemplates count at <storage-prefix>//podtemplates
I0907 11:10:10.193829 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.193864 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.200520 7577 cacher.go:405] cacher (*core.PodTemplate): initialized
I0907 11:10:10.204003 7577 store.go:1376] Monitoring events count at <storage-prefix>//events
I0907 11:10:10.204885 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.204931 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.211519 7577 store.go:1376] Monitoring limitranges count at <storage-prefix>//limitranges
I0907 11:10:10.214437 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.214499 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.217852 7577 cacher.go:405] cacher (*core.LimitRange): initialized
I0907 11:10:10.225868 7577 store.go:1376] Monitoring resourcequotas count at <storage-prefix>//resourcequotas
I0907 11:10:10.228182 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.228286 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.232981 7577 cacher.go:405] cacher (*core.ResourceQuota): initialized
I0907 11:10:10.240524 7577 store.go:1376] Monitoring secrets count at <storage-prefix>//secrets
I0907 11:10:10.241317 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.241338 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.247825 7577 store.go:1376] Monitoring persistentvolumes count at <storage-prefix>//persistentvolumes
I0907 11:10:10.248290 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.248314 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.250441 7577 cacher.go:405] cacher (*core.Secret): initialized
I0907 11:10:10.258956 7577 cacher.go:405] cacher (*core.PersistentVolume): initialized
I0907 11:10:10.261346 7577 store.go:1376] Monitoring persistentvolumeclaims count at <storage-prefix>//persistentvolumeclaims
I0907 11:10:10.261934 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.261958 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.264535 7577 cacher.go:405] cacher (*core.PersistentVolumeClaim): initialized
I0907 11:10:10.268862 7577 store.go:1376] Monitoring configmaps count at <storage-prefix>//configmaps
I0907 11:10:10.269788 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.269837 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.273873 7577 cacher.go:405] cacher (*core.ConfigMap): initialized
I0907 11:10:10.276406 7577 store.go:1376] Monitoring namespaces count at <storage-prefix>//namespaces
I0907 11:10:10.277404 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.277433 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.282949 7577 cacher.go:405] cacher (*core.Namespace): initialized
I0907 11:10:10.286760 7577 store.go:1376] Monitoring endpoints count at <storage-prefix>//services/endpoints
I0907 11:10:10.287791 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.287876 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.292339 7577 cacher.go:405] cacher (*core.Endpoints): initialized
I0907 11:10:10.297625 7577 store.go:1376] Monitoring nodes count at <storage-prefix>//minions
I0907 11:10:10.298802 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.298853 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.309092 7577 cacher.go:405] cacher (*core.Node): initialized
I0907 11:10:10.309260 7577 store.go:1376] Monitoring pods count at <storage-prefix>//pods
I0907 11:10:10.310157 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.310201 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.318518 7577 cacher.go:405] cacher (*core.Pod): initialized
I0907 11:10:10.319040 7577 store.go:1376] Monitoring serviceaccounts count at <storage-prefix>//serviceaccounts
I0907 11:10:10.322302 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.322364 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.328407 7577 cacher.go:405] cacher (*core.ServiceAccount): initialized
I0907 11:10:10.329588 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.329617 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.341627 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.341712 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.349861 7577 store.go:1376] Monitoring replicationcontrollers count at <storage-prefix>//controllers
I0907 11:10:10.350976 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.351033 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.359374 7577 cacher.go:405] cacher (*core.ReplicationController): initialized
I0907 11:10:10.362334 7577 store.go:1376] Monitoring services count at <storage-prefix>//services/specs
I0907 11:10:10.362421 7577 rest.go:131] the default service ipfamily for this cluster is: IPv4
I0907 11:10:10.369497 7577 cacher.go:405] cacher (*core.Service): initialized
I0907 11:10:10.433239 7577 instance.go:594] Skipping disabled API group "internal.apiserver.k8s.io".
I0907 11:10:10.433337 7577 instance.go:615] Enabling API group "authentication.k8s.io".
I0907 11:10:10.433365 7577 instance.go:615] Enabling API group "authorization.k8s.io".
I0907 11:10:10.434320 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.434415 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.442103 7577 store.go:1376] Monitoring horizontalpodautoscalers.autoscaling count at <storage-prefix>//horizontalpodautoscalers
I0907 11:10:10.443432 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.443525 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.446393 7577 cacher.go:405] cacher (*autoscaling.HorizontalPodAutoscaler): initialized
I0907 11:10:10.451431 7577 store.go:1376] Monitoring horizontalpodautoscalers.autoscaling count at <storage-prefix>//horizontalpodautoscalers
I0907 11:10:10.452156 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.452192 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.455263 7577 cacher.go:405] cacher (*autoscaling.HorizontalPodAutoscaler): initialized
I0907 11:10:10.459332 7577 store.go:1376] Monitoring horizontalpodautoscalers.autoscaling count at <storage-prefix>//horizontalpodautoscalers
I0907 11:10:10.459400 7577 instance.go:615] Enabling API group "autoscaling".
I0907 11:10:10.460460 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.460512 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.463882 7577 cacher.go:405] cacher (*autoscaling.HorizontalPodAutoscaler): initialized
I0907 11:10:10.467647 7577 store.go:1376] Monitoring jobs.batch count at <storage-prefix>//jobs
I0907 11:10:10.468669 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.468755 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.476015 7577 store.go:1376] Monitoring cronjobs.batch count at <storage-prefix>//cronjobs
I0907 11:10:10.476054 7577 instance.go:615] Enabling API group "batch".
I0907 11:10:10.476619 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.476644 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.478226 7577 cacher.go:405] cacher (*batch.Job): initialized
I0907 11:10:10.486063 7577 store.go:1376] Monitoring certificatesigningrequests.certificates.k8s.io count at <storage-prefix>//certificatesigningrequests
I0907 11:10:10.486597 7577 cacher.go:405] cacher (*batch.CronJob): initialized
I0907 11:10:10.487418 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.487473 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.490073 7577 cacher.go:405] cacher (*certificates.CertificateSigningRequest): initialized
I0907 11:10:10.495265 7577 store.go:1376] Monitoring certificatesigningrequests.certificates.k8s.io count at <storage-prefix>//certificatesigningrequests
I0907 11:10:10.495400 7577 instance.go:615] Enabling API group "certificates.k8s.io".
I0907 11:10:10.495926 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.495961 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.505839 7577 cacher.go:405] cacher (*certificates.CertificateSigningRequest): initialized
I0907 11:10:10.506269 7577 store.go:1376] Monitoring leases.coordination.k8s.io count at <storage-prefix>//leases
I0907 11:10:10.506752 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.506779 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.515924 7577 cacher.go:405] cacher (*coordination.Lease): initialized
I0907 11:10:10.518790 7577 store.go:1376] Monitoring leases.coordination.k8s.io count at <storage-prefix>//leases
I0907 11:10:10.518856 7577 instance.go:615] Enabling API group "coordination.k8s.io".
I0907 11:10:10.519538 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.519581 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.529463 7577 store.go:1376] Monitoring endpointslices.discovery.k8s.io count at <storage-prefix>//endpointslices
I0907 11:10:10.529610 7577 instance.go:615] Enabling API group "discovery.k8s.io".
I0907 11:10:10.530428 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.530480 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.534466 7577 cacher.go:405] cacher (*coordination.Lease): initialized
I0907 11:10:10.540060 7577 cacher.go:405] cacher (*discovery.EndpointSlice): initialized
I0907 11:10:10.540688 7577 store.go:1376] Monitoring ingresses.networking.k8s.io count at <storage-prefix>//ingress
I0907 11:10:10.540744 7577 instance.go:615] Enabling API group "extensions".
I0907 11:10:10.543053 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.543091 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.544632 7577 cacher.go:405] cacher (*networking.Ingress): initialized
I0907 11:10:10.552559 7577 store.go:1376] Monitoring networkpolicies.networking.k8s.io count at <storage-prefix>//networkpolicies
I0907 11:10:10.553464 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.553588 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.558500 7577 cacher.go:405] cacher (*networking.NetworkPolicy): initialized
I0907 11:10:10.563785 7577 store.go:1376] Monitoring ingresses.networking.k8s.io count at <storage-prefix>//ingress
I0907 11:10:10.564430 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.564455 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.571907 7577 cacher.go:405] cacher (*networking.Ingress): initialized
I0907 11:10:10.575443 7577 store.go:1376] Monitoring ingressclasses.networking.k8s.io count at <storage-prefix>//ingressclasses
I0907 11:10:10.576185 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.576222 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.585044 7577 cacher.go:405] cacher (*networking.IngressClass): initialized
I0907 11:10:10.585767 7577 store.go:1376] Monitoring ingresses.networking.k8s.io count at <storage-prefix>//ingress
I0907 11:10:10.586920 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.586983 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.594465 7577 store.go:1376] Monitoring ingressclasses.networking.k8s.io count at <storage-prefix>//ingressclasses
I0907 11:10:10.594507 7577 instance.go:615] Enabling API group "networking.k8s.io".
I0907 11:10:10.595218 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.595254 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.595676 7577 cacher.go:405] cacher (*networking.Ingress): initialized
I0907 11:10:10.605089 7577 cacher.go:405] cacher (*networking.IngressClass): initialized
I0907 11:10:10.605463 7577 store.go:1376] Monitoring runtimeclasses.node.k8s.io count at <storage-prefix>//runtimeclasses
I0907 11:10:10.606735 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.606793 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.608653 7577 cacher.go:405] cacher (*node.RuntimeClass): initialized
I0907 11:10:10.614156 7577 store.go:1376] Monitoring runtimeclasses.node.k8s.io count at <storage-prefix>//runtimeclasses
I0907 11:10:10.614220 7577 instance.go:615] Enabling API group "node.k8s.io".
I0907 11:10:10.614835 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.614868 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.624391 7577 cacher.go:405] cacher (*node.RuntimeClass): initialized
I0907 11:10:10.628409 7577 store.go:1376] Monitoring poddisruptionbudgets.policy count at <storage-prefix>//poddisruptionbudgets
I0907 11:10:10.629370 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.629407 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.640131 7577 cacher.go:405] cacher (*policy.PodDisruptionBudget): initialized
I0907 11:10:10.640673 7577 store.go:1376] Monitoring podsecuritypolicies.policy count at <storage-prefix>//podsecuritypolicy
I0907 11:10:10.640700 7577 instance.go:615] Enabling API group "policy".
I0907 11:10:10.641229 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.641272 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.645399 7577 cacher.go:405] cacher (*policy.PodSecurityPolicy): initialized
I0907 11:10:10.655313 7577 store.go:1376] Monitoring roles.rbac.authorization.k8s.io count at <storage-prefix>//roles
I0907 11:10:10.656420 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.656452 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.665228 7577 store.go:1376] Monitoring rolebindings.rbac.authorization.k8s.io count at <storage-prefix>//rolebindings
I0907 11:10:10.665567 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.665584 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.671995 7577 cacher.go:405] cacher (*rbac.Role): initialized
I0907 11:10:10.673596 7577 cacher.go:405] cacher (*rbac.RoleBinding): initialized
I0907 11:10:10.675409 7577 store.go:1376] Monitoring clusterroles.rbac.authorization.k8s.io count at <storage-prefix>//clusterroles
I0907 11:10:10.676170 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.676204 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.681365 7577 cacher.go:405] cacher (*rbac.ClusterRole): initialized
I0907 11:10:10.685026 7577 store.go:1376] Monitoring clusterrolebindings.rbac.authorization.k8s.io count at <storage-prefix>//clusterrolebindings
I0907 11:10:10.685852 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.685915 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.694700 7577 store.go:1376] Monitoring roles.rbac.authorization.k8s.io count at <storage-prefix>//roles
I0907 11:10:10.695418 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.695452 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.697954 7577 cacher.go:405] cacher (*rbac.ClusterRoleBinding): initialized
I0907 11:10:10.704331 7577 store.go:1376] Monitoring rolebindings.rbac.authorization.k8s.io count at <storage-prefix>//rolebindings
I0907 11:10:10.705273 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.705328 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.707115 7577 cacher.go:405] cacher (*rbac.Role): initialized
I0907 11:10:10.714130 7577 store.go:1376] Monitoring clusterroles.rbac.authorization.k8s.io count at <storage-prefix>//clusterroles
I0907 11:10:10.714280 7577 cacher.go:405] cacher (*rbac.RoleBinding): initialized
I0907 11:10:10.715533 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.715568 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.722210 7577 cacher.go:405] cacher (*rbac.ClusterRole): initialized
I0907 11:10:10.724814 7577 store.go:1376] Monitoring clusterrolebindings.rbac.authorization.k8s.io count at <storage-prefix>//clusterrolebindings
I0907 11:10:10.724868 7577 instance.go:615] Enabling API group "rbac.authorization.k8s.io".
I0907 11:10:10.727455 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.727504 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.734196 7577 cacher.go:405] cacher (*rbac.ClusterRoleBinding): initialized
I0907 11:10:10.734678 7577 store.go:1376] Monitoring priorityclasses.scheduling.k8s.io count at <storage-prefix>//priorityclasses
I0907 11:10:10.735336 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.735367 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.746599 7577 cacher.go:405] cacher (*scheduling.PriorityClass): initialized
I0907 11:10:10.747241 7577 store.go:1376] Monitoring priorityclasses.scheduling.k8s.io count at <storage-prefix>//priorityclasses
I0907 11:10:10.747274 7577 instance.go:615] Enabling API group "scheduling.k8s.io".
I0907 11:10:10.748135 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.749535 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.756577 7577 store.go:1376] Monitoring storageclasses.storage.k8s.io count at <storage-prefix>//storageclasses
I0907 11:10:10.757360 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.757411 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.760090 7577 cacher.go:405] cacher (*scheduling.PriorityClass): initialized
I0907 11:10:10.761445 7577 cacher.go:405] cacher (*storage.StorageClass): initialized
I0907 11:10:10.767222 7577 store.go:1376] Monitoring volumeattachments.storage.k8s.io count at <storage-prefix>//volumeattachments
I0907 11:10:10.768289 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.768357 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.774868 7577 cacher.go:405] cacher (*storage.VolumeAttachment): initialized
I0907 11:10:10.776209 7577 store.go:1376] Monitoring csinodes.storage.k8s.io count at <storage-prefix>//csinodes
I0907 11:10:10.776692 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.776708 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.784385 7577 cacher.go:405] cacher (*storage.CSINode): initialized
I0907 11:10:10.785952 7577 store.go:1376] Monitoring csidrivers.storage.k8s.io count at <storage-prefix>//csidrivers
I0907 11:10:10.786543 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.786571 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.795441 7577 cacher.go:405] cacher (*storage.CSIDriver): initialized
I0907 11:10:10.795719 7577 store.go:1376] Monitoring storageclasses.storage.k8s.io count at <storage-prefix>//storageclasses
I0907 11:10:10.796371 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.796406 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.800397 7577 cacher.go:405] cacher (*storage.StorageClass): initialized
I0907 11:10:10.804087 7577 store.go:1376] Monitoring volumeattachments.storage.k8s.io count at <storage-prefix>//volumeattachments
I0907 11:10:10.804770 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.804813 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.814822 7577 cacher.go:405] cacher (*storage.VolumeAttachment): initialized
I0907 11:10:10.815314 7577 store.go:1376] Monitoring csinodes.storage.k8s.io count at <storage-prefix>//csinodes
I0907 11:10:10.815972 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.816002 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.823372 7577 store.go:1376] Monitoring csidrivers.storage.k8s.io count at <storage-prefix>//csidrivers
I0907 11:10:10.823449 7577 instance.go:615] Enabling API group "storage.k8s.io".
I0907 11:10:10.824268 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.824286 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.824776 7577 cacher.go:405] cacher (*storage.CSINode): initialized
I0907 11:10:10.835613 7577 cacher.go:405] cacher (*storage.CSIDriver): initialized
I0907 11:10:10.836114 7577 store.go:1376] Monitoring flowschemas.flowcontrol.apiserver.k8s.io count at <storage-prefix>//flowschemas
I0907 11:10:10.836751 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.838994 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.848549 7577 store.go:1376] Monitoring prioritylevelconfigurations.flowcontrol.apiserver.k8s.io count at <storage-prefix>//prioritylevelconfigurations
I0907 11:10:10.848594 7577 instance.go:615] Enabling API group "flowcontrol.apiserver.k8s.io".
I0907 11:10:10.849819 7577 cacher.go:405] cacher (*flowcontrol.FlowSchema): initialized
I0907 11:10:10.849904 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.849940 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.857339 7577 store.go:1376] Monitoring deployments.apps count at <storage-prefix>//deployments
I0907 11:10:10.857508 7577 cacher.go:405] cacher (*flowcontrol.PriorityLevelConfiguration): initialized
I0907 11:10:10.858030 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.858053 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.864831 7577 store.go:1376] Monitoring statefulsets.apps count at <storage-prefix>//statefulsets
I0907 11:10:10.865201 7577 cacher.go:405] cacher (*apps.Deployment): initialized
I0907 11:10:10.865754 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.865792 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.876153 7577 cacher.go:405] cacher (*apps.StatefulSet): initialized
I0907 11:10:10.876223 7577 store.go:1376] Monitoring daemonsets.apps count at <storage-prefix>//daemonsets
I0907 11:10:10.877053 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.877083 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.888313 7577 store.go:1376] Monitoring replicasets.apps count at <storage-prefix>//replicasets
I0907 11:10:10.889675 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.889731 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.893058 7577 cacher.go:405] cacher (*apps.DaemonSet): initialized
I0907 11:10:10.900221 7577 cacher.go:405] cacher (*apps.ReplicaSet): initialized
I0907 11:10:10.900235 7577 store.go:1376] Monitoring controllerrevisions.apps count at <storage-prefix>//controllerrevisions
I0907 11:10:10.900316 7577 instance.go:615] Enabling API group "apps".
I0907 11:10:10.901237 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.901307 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.905216 7577 cacher.go:405] cacher (*apps.ControllerRevision): initialized
I0907 11:10:10.910966 7577 store.go:1376] Monitoring validatingwebhookconfigurations.admissionregistration.k8s.io count at <storage-prefix>//validatingwebhookconfigurations
I0907 11:10:10.911510 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.911540 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.921253 7577 cacher.go:405] cacher (*admissionregistration.ValidatingWebhookConfiguration): initialized
I0907 11:10:10.921432 7577 store.go:1376] Monitoring mutatingwebhookconfigurations.admissionregistration.k8s.io count at <storage-prefix>//mutatingwebhookconfigurations
I0907 11:10:10.922011 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.922098 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.931306 7577 cacher.go:405] cacher (*admissionregistration.MutatingWebhookConfiguration): initialized
I0907 11:10:10.931434 7577 store.go:1376] Monitoring validatingwebhookconfigurations.admissionregistration.k8s.io count at <storage-prefix>//validatingwebhookconfigurations
I0907 11:10:10.932370 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.932417 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.938320 7577 cacher.go:405] cacher (*admissionregistration.ValidatingWebhookConfiguration): initialized
I0907 11:10:10.940181 7577 store.go:1376] Monitoring mutatingwebhookconfigurations.admissionregistration.k8s.io count at <storage-prefix>//mutatingwebhookconfigurations
I0907 11:10:10.940215 7577 instance.go:615] Enabling API group "admissionregistration.k8s.io".
I0907 11:10:10.940809 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.940853 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.944728 7577 cacher.go:405] cacher (*admissionregistration.MutatingWebhookConfiguration): initialized
I0907 11:10:10.953665 7577 store.go:1376] Monitoring events count at <storage-prefix>//events
I0907 11:10:10.956627 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:10.956676 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:10.971189 7577 store.go:1376] Monitoring events count at <storage-prefix>//events
I0907 11:10:10.971225 7577 instance.go:615] Enabling API group "events.k8s.io".
W0907 11:10:11.073103 7577 genericapiserver.go:419] Skipping API batch/v2alpha1 because it has no resources.
W0907 11:10:11.082739 7577 genericapiserver.go:419] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0907 11:10:11.093612 7577 genericapiserver.go:419] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0907 11:10:11.101685 7577 genericapiserver.go:419] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
I0907 11:10:11.104788 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:11.104939 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
W0907 11:10:11.107101 7577 genericapiserver.go:419] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0907 11:10:11.114524 7577 genericapiserver.go:419] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0907 11:10:11.118164 7577 genericapiserver.go:419] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0907 11:10:11.124193 7577 genericapiserver.go:419] Skipping API apps/v1beta2 because it has no resources.
W0907 11:10:11.124281 7577 genericapiserver.go:419] Skipping API apps/v1beta1 because it has no resources.
I0907 11:10:11.142748 7577 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0907 11:10:11.142776 7577 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0907 11:10:11.151123 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:11.151160 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:11.163588 7577 store.go:1376] Monitoring apiservices.apiregistration.k8s.io count at <storage-prefix>//apiregistration.k8s.io/apiservices
I0907 11:10:11.164601 7577 client.go:360] parsed scheme: "endpoint"
I0907 11:10:11.164626 7577 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://192.168.1.201:2379 <nil> 0 <nil>} {https://192.168.1.202:2379 <nil> 0 <nil>} {https://192.168.1.203:2379 <nil> 0 <nil>}]
I0907 11:10:11.179263 7577 store.go:1376] Monitoring apiservices.apiregistration.k8s.io count at <storage-prefix>//apiregistration.k8s.io/apiservices
I0907 11:10:11.181001 7577 dynamic_serving_content.go:111] Loaded a new cert/key pair for "aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.pem::/etc/kubernetes/pki/front-proxy-client-key.pem"
I0907 11:10:11.181300 7577 cacher.go:405] cacher (*apiregistration.APIService): initialized
I0907 11:10:11.188700 7577 cacher.go:405] cacher (*apiregistration.APIService): initialized
I0907 11:10:12.892235 7577 aggregator.go:109] Building initial OpenAPI spec
I0907 11:10:13.528646 7577 aggregator.go:112] Finished initial OpenAPI spec generation after 636.366725ms
I0907 11:10:13.529062 7577 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/pki/front-proxy-ca.pem
I0907 11:10:13.529096 7577 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.pem
I0907 11:10:13.529147 7577 tlsconfig.go:178] loaded client CA [0/"client-ca-bundle::/etc/kubernetes/pki/ca.pem,request-header::/etc/kubernetes/pki/front-proxy-ca.pem"]: "kubernetes" [] groups=[Kubernetes] issuer="<self>" (2021-09-06 05:53:00 +0000 UTC to 2121-08-13 05:53:00 +0000 UTC (now=2021-09-07 03:10:13.529132692 +0000 UTC))
I0907 11:10:13.529162 7577 tlsconfig.go:178] loaded client CA [1/"client-ca-bundle::/etc/kubernetes/pki/ca.pem,request-header::/etc/kubernetes/pki/front-proxy-ca.pem"]: "kubernetes" [] issuer="<self>" (2021-09-06 06:11:00 +0000 UTC to 2026-09-05 06:11:00 +0000 UTC (now=2021-09-07 03:10:13.5291582 +0000 UTC))
I0907 11:10:13.529378 7577 tlsconfig.go:200] loaded serving cert ["serving-cert::/etc/kubernetes/pki/apiserver.pem::/etc/kubernetes/pki/apiserver-key.pem"]: "kube-apiserver" [serving,client] groups=[Kubernetes] validServingFor=[127.0.0.1,10.96.0.1,192.168.0.211,192.168.1.201,192.168.1.202,192.168.1.203,192.168.1.206,192.168.1.207,192.168.1.208,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local] issuer="kubernetes" (2021-09-06 05:59:00 +0000 UTC to 2121-08-13 05:59:00 +0000 UTC (now=2021-09-07 03:10:13.529368866 +0000 UTC))
I0907 11:10:13.529457 7577 dynamic_serving_content.go:130] Starting serving-cert::/etc/kubernetes/pki/apiserver.pem::/etc/kubernetes/pki/apiserver-key.pem
I0907 11:10:13.529566 7577 named_certificates.go:53] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1630984210" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1630984209" (2021-09-07 02:10:09 +0000 UTC to 2022-09-07 02:10:09 +0000 UTC (now=2021-09-07 03:10:13.529559333 +0000 UTC))
I0907 11:10:13.529585 7577 secure_serving.go:197] Serving securely on [::]:6443
I0907 11:10:13.529614 7577 controller.go:83] Starting OpenAPI AggregationController
I0907 11:10:13.529627 7577 tlsconfig.go:240] Starting DynamicServingCertificateController
I0907 11:10:13.529692 7577 available_controller.go:475] Starting AvailableConditionController
I0907 11:10:13.529697 7577 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0907 11:10:13.529709 7577 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.pem::/etc/kubernetes/pki/front-proxy-client-key.pem
I0907 11:10:13.529739 7577 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0907 11:10:13.529741 7577 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0907 11:10:13.529772 7577 customresource_discovery_controller.go:209] Starting DiscoveryController
I0907 11:10:13.529799 7577 apf_controller.go:249] Starting API Priority and Fairness config controller
I0907 11:10:13.529936 7577 reflector.go:219] Starting reflector *v1.CustomResourceDefinition (5m0s) from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117
I0907 11:10:13.530485 7577 controller.go:86] Starting OpenAPI controller
I0907 11:10:13.530500 7577 naming_controller.go:291] Starting NamingConditionController
I0907 11:10:13.530508 7577 establishing_controller.go:76] Starting EstablishingController
I0907 11:10:13.530519 7577 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0907 11:10:13.530529 7577 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0907 11:10:13.530540 7577 crd_finalizer.go:266] Starting CRDFinalizer
I0907 11:10:13.534971 7577 autoregister_controller.go:141] Starting autoregister controller
I0907 11:10:13.534989 7577 cache.go:32] Waiting for caches to sync for autoregister controller
I0907 11:10:13.535796 7577 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0907 11:10:13.535822 7577 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
I0907 11:10:13.536050 7577 reflector.go:219] Starting reflector *v1beta1.PriorityLevelConfiguration (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.536667 7577 reflector.go:219] Starting reflector *v1.APIService (30s) from k8s.io/kube-aggregator/pkg/client/informers/externalversions/factory.go:117
I0907 11:10:13.542294 7577 crdregistration_controller.go:111] Starting crd-autoregister controller
I0907 11:10:13.542345 7577 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
I0907 11:10:13.542439 7577 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.pem
I0907 11:10:13.542468 7577 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/pki/front-proxy-ca.pem
I0907 11:10:13.542661 7577 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/kubernetes/pkg/controlplane/controller/clusterauthenticationtrust/cluster_authentication_trust_controller.go:444
I0907 11:10:13.543102 7577 reflector.go:219] Starting reflector *v1.Secret (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.543321 7577 reflector.go:219] Starting reflector *v1.Pod (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.543472 7577 reflector.go:219] Starting reflector *v1.VolumeAttachment (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.543619 7577 reflector.go:219] Starting reflector *v1.Role (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.543761 7577 reflector.go:219] Starting reflector *v1.RoleBinding (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.543925 7577 reflector.go:219] Starting reflector *v1.Endpoints (10m0s) from k8s.io/client-go/informers/factory.go:134
E0907 11:10:13.544000 7577 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.1.201, ResourceVersion: 0, AdditionalErrorMsg:
I0907 11:10:13.544079 7577 reflector.go:219] Starting reflector *v1.ServiceAccount (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.544255 7577 reflector.go:219] Starting reflector *v1.PersistentVolume (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.544458 7577 reflector.go:219] Starting reflector *v1.ClusterRole (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.544634 7577 reflector.go:219] Starting reflector *v1.Service (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.544821 7577 reflector.go:219] Starting reflector *v1.StorageClass (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.544976 7577 reflector.go:219] Starting reflector *v1.RuntimeClass (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.545182 7577 reflector.go:219] Starting reflector *v1.Node (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.545330 7577 reflector.go:219] Starting reflector *v1.ClusterRoleBinding (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.545479 7577 reflector.go:219] Starting reflector *v1.Namespace (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.545648 7577 reflector.go:219] Starting reflector *v1.MutatingWebhookConfiguration (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.545817 7577 reflector.go:219] Starting reflector *v1beta1.FlowSchema (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.545972 7577 reflector.go:219] Starting reflector *v1.LimitRange (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.546139 7577 reflector.go:219] Starting reflector *v1.PriorityClass (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.546358 7577 reflector.go:219] Starting reflector *v1.IngressClass (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.546515 7577 reflector.go:219] Starting reflector *v1.ValidatingWebhookConfiguration (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.546660 7577 reflector.go:219] Starting reflector *v1.ResourceQuota (10m0s) from k8s.io/client-go/informers/factory.go:134
I0907 11:10:13.583371 7577 healthz.go:244] poststarthook/start-apiextensions-controllers,poststarthook/crd-informer-synced,poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes,poststarthook/apiservice-registration-controller check failed: readyz
[-]poststarthook/start-apiextensions-controllers failed: not finished
[-]poststarthook/crd-informer-synced failed: not finished
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
[-]poststarthook/apiservice-registration-controller failed: not finished
I0907 11:10:13.604242 7577 shared_informer.go:247] Caches are synced for node_authorizer
I0907 11:10:13.629834 7577 apf_controller.go:253] Running API Priority and Fairness config worker
I0907 11:10:13.629839 7577 cache.go:39] Caches are synced for AvailableConditionController controller
I0907 11:10:13.629855 7577 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0907 11:10:13.638389 7577 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller
I0907 11:10:13.638653 7577 cache.go:39] Caches are synced for autoregister controller
I0907 11:10:13.642879 7577 shared_informer.go:247] Caches are synced for crd-autoregister
I0907 11:10:13.688937 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I0907 11:10:13.787157 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I0907 11:10:13.889949 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I0907 11:10:13.988980 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I0907 11:10:14.087720 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I0907 11:10:14.188777 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I0907 11:10:14.293555 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I0907 11:10:14.387192 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I0907 11:10:14.487020 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles,poststarthook/scheduling/bootstrap-system-priority-classes check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I0907 11:10:14.528928 7577 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0907 11:10:14.529050 7577 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0907 11:10:14.541140 7577 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
I0907 11:10:14.589159 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
I0907 11:10:14.687935 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
I0907 11:10:14.787275 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
I0907 11:10:14.890847 7577 healthz.go:244] poststarthook/rbac/bootstrap-roles check failed: readyz
[-]poststarthook/rbac/bootstrap-roles failed: not finished
I0907 11:10:42.761046 7577 client.go:360] parsed scheme: "passthrough"
I0907 11:10:42.761119 7577 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.1.201:2379 <nil> 0 <nil>}] <nil> <nil>}
I0907 11:10:42.761132 7577 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0907 11:10:42.761347 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00ec66420, {CONNECTING <nil>}
I0907 11:10:42.768677 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00ec66420, {READY <nil>}
I0907 11:10:42.769865 7577 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0907 11:10:52.208425 7577 client.go:360] parsed scheme: "passthrough"
I0907 11:10:52.208522 7577 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.1.203:2379 <nil> 0 <nil>}] <nil> <nil>}
I0907 11:10:52.208542 7577 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0907 11:10:52.208728 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00ed6daa0, {CONNECTING <nil>}
I0907 11:10:52.216190 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00ed6daa0, {READY <nil>}
I0907 11:10:52.217462 7577 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0907 11:10:52.279861 7577 client.go:360] parsed scheme: "passthrough"
I0907 11:10:52.279938 7577 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.1.202:2379 <nil> 0 <nil>}] <nil> <nil>}
I0907 11:10:52.279952 7577 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0907 11:10:52.280099 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00ee5cca0, {CONNECTING <nil>}
I0907 11:10:52.290254 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00ee5cca0, {READY <nil>}
I0907 11:10:52.291364 7577 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0907 11:11:25.425875 7577 client.go:360] parsed scheme: "passthrough"
I0907 11:11:25.426010 7577 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.1.201:2379 <nil> 0 <nil>}] <nil> <nil>}
I0907 11:11:25.426026 7577 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0907 11:11:25.426411 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00f0c2bd0, {CONNECTING <nil>}
I0907 11:11:25.432251 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00f0c2bd0, {READY <nil>}
I0907 11:11:25.433261 7577 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
I0907 11:11:30.467306 7577 client.go:360] parsed scheme: "passthrough"
I0907 11:11:30.467362 7577 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.1.202:2379 <nil> 0 <nil>}] <nil> <nil>}
I0907 11:11:30.467371 7577 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0907 11:11:30.467490 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00f18cb10, {CONNECTING <nil>}
I0907 11:11:30.477647 7577 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00f18cb10, {READY <nil>}
I0907 11:11:30.479113 7577 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"

原因分析：I0907表示為imformation，不是報錯，error: desc = "transport is closing"不會影響使用，但會影響性能；只有E開頭（error）的才是報錯；

解決方法：服務正常運行，暫不處理。

問題30：bootstrap.secret.yaml創建secret等資源失敗？

[root@k8s-master01 bootstrap]# kubectl create -f bootstrap.secret.yaml
Unable to connect to the server: dial tcp 192.168.1.211:8443: connect: no route to host

原因分析：ip無法訪問；

解決方法：修改證書的apiserver地址即可。

mkdir -p /root/.kube;cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

問題31、telnet高可用VIP:8443端口，立即異常退出，RS IP需按ctrl+]才會退出端口測試？

[root@k8s-master02 ~]# telnet 192.168.1.236 8443
Trying 192.168.1.236...
Connected to 192.168.1.236.
Escape character is '^]'.
Connection closed by foreign host.
[root@k8s-master02 ~]# telnet 192.168.1.201 6443
Trying 192.168.1.201...
Connected to 192.168.1.201.
Escape character is '^]'.

原因分析：VIP:8443不提供具體服務，而是由RS:6443提供具體服務；

解決方法：無需處理。

問題32、K8S集群無法查詢node資源？

[root@k8s-master01 bootstrap]# kubectl get node
No resources found

原因分析：kubelet未安裝，所以無法查詢node信息；

解決方法：無需處理。

問題33、kubelet啟動失敗？

[root@k8s-master01 kubernetes]# systemctl status kubelet.service
● kubelet.service - kubernetes kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─kubelet.conf
Active: activating (auto-restart) (Result: exit-code) since Wed 2021-09-08 15:47:44 CST; 435ms ago
Docs: https://github.com/kubernetes/kubernetes
Process: 18448 ExecStart=/usr/local/bin/kubelet (code=exited, status=255)
Main PID: 18448 (code=exited, status=255)

Sep 08 15:47:44 k8s-master01 systemd[1]: Unit kubelet.service entered failed state.
Sep 08 15:47:44 k8s-master01 systemd[1]: kubelet.service failed.

原因分析：日志顯示，配置錯誤。

Sep  8 15:56:56 k8s-master01 kubelet: I0908 15:56:56.003430   22603 docker_service.go:240] Hairpin mode set to "hairpin-veth"
Sep  8 15:56:56 k8s-master01 kubelet: W0908 15:56:56.003518   22603 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
Sep  8 15:56:56 k8s-master01 kubelet: I0908 15:56:56.009817   22603 docker_service.go:255] Docker cri networking managed by kubernetes.io/no-op
Sep  8 15:56:56 k8s-master01 kubelet: I0908 15:56:56.021591   22603 docker_service.go:260] Docker Info: &{ID:BKPM:2C3W:J2KC:TFPT:DS5J:ANQ3:UVLF:KYLC:6WL6:32YK:3HTR:ZAWJ Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:0 Driver:overlay2 DriverStatus:[[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:22 OomKillDisable:true NGoroutines:35 SystemTime:2021-09-08T15:56:56.011370377+08:00 LoggingDriver:json-file CgroupDriver:systemd NEventsListener:0 KernelVersion:4.19.12-1.el7.elrepo.x86_64 OperatingSystem:CentOS Linux 7 (Core) OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc000972380 NCPU:2 MemTotal:4111605760 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:k8s-master01 Labels:[] ExperimentalBuild:false ServerVersion:19.03.11 ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:e25210fe30a0a703442421b0f60afac609f950a3 Expected:e25210fe30a0a703442421b0f60afac609f950a3} RuncCommit:{ID:v1.0.1-0-g4144b63 Expected:v1.0.1-0-g4144b63} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=seccomp,profile=default] ProductLicense: Warnings:[]}
Sep  8 15:56:56 k8s-master01 kubelet: F0908 15:56:56.021746   22603 server.go:269] failed to run Kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
Sep  8 15:56:56 k8s-master01 kubelet: goroutine 1 [running]:
Sep  8 15:56:56 k8s-master01 kubelet: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc000010001, 0xc00000bf00, 0xaa, 0xfc)
Sep  8 15:56:56 k8s-master01 kubelet: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
Sep  8 15:56:56 k8s-master01 kubelet: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x70c7460, 0xc000000003, 0x0, 0x0, 0xc0009723f0, 0x6f32a12, 0x9, 0x10d, 0x411b00)
Sep  8 15:56:56 k8s-master01 kubelet: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x19b
Sep  8 15:56:56 k8s-master01 kubelet: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x70c7460, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc00092f2a0, 0x1, 0x1)
Sep  8 15:56:56 k8s-master01 kubelet: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:732 +0x16f
Sep  8 15:56:56 k8s-master01 kubelet: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).print(...)
Sep  8 15:56:56 k8s-master01 kubelet: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:714
Sep  8 15:56:56 k8s-master01 kubelet: k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatal(...)
Sep  8 15:56:56 k8s-master01 kubelet: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1482
...
Sep  8 15:56:56 k8s-master01 systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/net.go:182 +0x8e
Sep  8 15:56:56 k8s-master01 kubelet: net/http.(*persistConn).Read(0xc0007f8fc0, 0xc000873000, 0x1000, 0x1000, 0xc00009c420, 0xc00007dc58, 0x409095)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:1887 +0x77
Sep  8 15:56:56 k8s-master01 kubelet: bufio.(*Reader).fill(0xc0007e9da0)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/bufio/bufio.go:101 +0x105
Sep  8 15:56:56 k8s-master01 kubelet: bufio.(*Reader).Peek(0xc0007e9da0, 0x1, 0x0, 0x0, 0x1, 0x0, 0xc000142720)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/bufio/bufio.go:139 +0x4f
Sep  8 15:56:56 k8s-master01 kubelet: net/http.(*persistConn).readLoop(0xc0007f8fc0)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:2040 +0x1a8
Sep  8 15:56:56 k8s-master01 kubelet: created by net/http.(*Transport).dialConn
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:1708 +0xcb7
Sep  8 15:56:56 k8s-master01 kubelet: goroutine 104 [select]:
Sep  8 15:56:56 k8s-master01 kubelet: net/http.(*persistConn).writeLoop(0xc0007f8fc0)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:2340 +0x11c
Sep  8 15:56:56 k8s-master01 kubelet: created by net/http.(*Transport).dialConn
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:1709 +0xcdc
Sep  8 15:56:56 k8s-master01 kubelet: goroutine 106 [chan receive]:
Sep  8 15:56:56 k8s-master01 kubelet: k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x4f1b800, 0xc00090ea20, 0xc0003f8b70)
Sep  8 15:56:56 k8s-master01 kubelet: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:301 +0xaa
Sep  8 15:56:56 k8s-master01 kubelet: created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
Sep  8 15:56:56 k8s-master01 kubelet: /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:299 +0x6e
Sep  8 15:56:56 k8s-master01 kubelet: goroutine 108 [IO wait]:
Sep  8 15:56:56 k8s-master01 kubelet: internal/poll.runtime_pollWait(0x7f8f9916cd58, 0x72, 0x4f0ee60)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/runtime/netpoll.go:222 +0x55
Sep  8 15:56:56 k8s-master01 kubelet: internal/poll.(*pollDesc).wait(0xc00087c498, 0x72, 0x4f0ee00, 0x6fd5e28, 0x0)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
Sep  8 15:56:56 k8s-master01 kubelet: internal/poll.(*pollDesc).waitRead(...)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
Sep  8 15:56:56 k8s-master01 kubelet: internal/poll.(*FD).Read(0xc00087c480, 0xc000705000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
Sep  8 15:56:56 k8s-master01 kubelet: net.(*netFD).Read(0xc00087c480, 0xc000705000, 0x1000, 0x1000, 0x43de5c, 0xc000079b58, 0x46b2e0)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/fd_posix.go:55 +0x4f
Sep  8 15:56:56 k8s-master01 systemd: Unit kubelet.service entered failed state.
Sep  8 15:56:56 k8s-master01 kubelet: net.(*conn).Read(0xc00014e248, 0xc000705000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/net.go:182 +0x8e
Sep  8 15:56:56 k8s-master01 kubelet: net/http.(*persistConn).Read(0xc000a44c60, 0xc000705000, 0x1000, 0x1000, 0xc00009cd20, 0xc000079c58, 0x409095)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:1887 +0x77
Sep  8 15:56:56 k8s-master01 kubelet: bufio.(*Reader).fill(0xc000019440)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/bufio/bufio.go:101 +0x105
Sep  8 15:56:56 k8s-master01 kubelet: bufio.(*Reader).Peek(0xc000019440, 0x1, 0x0, 0x0, 0x1, 0x0, 0xc000143380)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/bufio/bufio.go:139 +0x4f
Sep  8 15:56:56 k8s-master01 kubelet: net/http.(*persistConn).readLoop(0xc000a44c60)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:2040 +0x1a8
Sep  8 15:56:56 k8s-master01 kubelet: created by net/http.(*Transport).dialConn
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:1708 +0xcb7
Sep  8 15:56:56 k8s-master01 kubelet: goroutine 109 [select]:
Sep  8 15:56:56 k8s-master01 kubelet: net/http.(*persistConn).writeLoop(0xc000a44c60)
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:2340 +0x11c
Sep  8 15:56:56 k8s-master01 kubelet: created by net/http.(*Transport).dialConn
Sep  8 15:56:56 k8s-master01 kubelet: /usr/local/go/src/net/http/transport.go:1709 +0xcdc
Sep  8 15:56:56 k8s-master01 systemd: kubelet.service failed.

解決方法：修改配置即可。

cat <<eof> /etc/docker/daemon.json
{
"exec-opts":["native.cgroupdriver=cgroupfs"],
"log-driver":"json-file",
"log-opts":{
"max-size":"100m"
}
}
eof
systemctl restart docker
systemctl status docker

問題34、calico-etcd部署失敗？

[root@k8s-master01 calico]# kubectl apply -f calico-etcd.yaml
secret/calico-etcd-secrets created
error: error parsing calico-etcd.yaml: error converting YAML to JSON: yaml: line 12: did not find expected key

原因分析：查看12行，沒有錯誤。實際錯誤發生在18行。

解決方法：修改配置即可。

問題34、創建pod失敗？

[root@k8s-master01 ~]# cat myapp.yaml
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh','-c','echo The app is running! && sleep 3600']
[root@k8s-master01 ~]#
[root@k8s-master01 ~]# kubectl apply -f myapp.yaml
Error from server (Forbidden): error when creating "myapp.yaml": pods "myapp-pod" is forbidden: error looking up service account default/default: serviceaccount "default" not found
原因分析：沒有default的SA；

解決方法：創建default/default的sa即可。

問題35、創建pod失敗？

[root@k8s-master01 ~]# kubectl apply -f myapp.yaml
Error from server (ServerTimeout): error when creating "myapp.yaml": No API token found for service account "default", retry after the token is automatically created and added to the service account

原因分析：SA沒有API token；

[root@k8s-master01 ~]# kubectl get sa
NAME SECRETS AGE
default 0 4s

解決方法：待處理；

問題36、kubelet.service啟動出現異常？

[root@k8s-master01 ~]# systemctl status kubelet.service
● kubelet.service - kubernetes kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubelet.conf
Active: active (running) since Fri 2021-09-10 10:08:26 CST; 19min ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 21876 (kubelet)
CGroup: /system.slice/kubelet.service
└─21876 /usr/local/bin/kubelet

Sep 10 10:08:27 k8s-master01 kubelet[21876]: I0910 10:08:27.353822 21876 policy_none.go:43] [cpumanager] none policy: Start
Sep 10 10:08:27 k8s-master01 kubelet[21876]: W0910 10:08:27.354453 21876 manager.go:594] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
Sep 10 10:08:27 k8s-master01 kubelet[21876]: I0910 10:08:27.354919 21876 plugin_manager.go:114] Starting Kubelet Plugin Manager
Sep 10 10:08:27 k8s-master01 kubelet[21876]: I0910 10:08:27.444279 21876 reconciler.go:157] Reconciler: start to sync state
Sep 10 10:10:21 k8s-master01 systemd[1]: [/etc/systemd/system/kubelet.service.d/10-kubelet.conf:5] Trailing garbage, ignoring.
Sep 10 10:10:21 k8s-master01 systemd[1]: [/etc/systemd/system/kubelet.service.d/10-kubelet.conf:5] Trailing garbage, ignoring.
Sep 10 10:10:51 k8s-master01 systemd[1]: [/etc/systemd/system/kubelet.service.d/10-kubelet.conf:5] Trailing garbage, ignoring.
Sep 10 10:10:51 k8s-master01 systemd[1]: [/etc/systemd/system/kubelet.service.d/10-kubelet.conf:5] Trailing garbage, ignoring.
Sep 10 10:10:51 k8s-master01 systemd[1]: [/etc/systemd/system/kubelet.service.d/10-kubelet.conf:5] Trailing garbage, ignoring.
Sep 10 10:10:51 k8s-master01 systemd[1]: [/etc/systemd/system/kubelet.service.d/10-kubelet.conf:5] Trailing garbage, ignoring.

[root@k8s-master01 ~]# kubectl get nodes
No resources found
[root@k8s-master01 ~]#

原因分析：配置產生垃圾；

解決方法：清除垃圾即可；

[root@k8s-master01 ~]# systemctl unmask kubelet.service
[root@k8s-master01 ~]#
[root@k8s-master01 ~]# systemctl restart kubelet.service
systemctl status kubelet.service
[root@k8s-master01 ~]# systemctl status kubelet.service
● kubelet.service - kubernetes kubelet
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubelet.conf
Active: active (running) since Fri 2021-09-10 10:31:42 CST; 31ms ago
Docs: https://github.com/kubernetes/kubernetes
Main PID: 27518 (kubelet)
Tasks: 6
Memory: 7.1M
CGroup: /system.slice/kubelet.service
└─27518 /usr/local/bin/kubelet

Sep 10 10:31:42 k8s-master01 systemd[1]: Started kubernetes kubelet.

問題37、kubelet.service啟動出現異常？

[root@k8s-master02 ~]# kubeadm config images pull --config /root/new-kubeadm-config.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.22.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.22.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.22.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.22.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.5
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.0-0
failed to pull image "registry.aliyuncs.com/google_containers/coredns:v1.8.4": output: Error response from daemon: manifest for registry.aliyuncs.com/google_containers/coredns:v1.8.4 not found: manifest unknown: manifest unknown
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

原因分析：源站鏡像tag格式不同。

解決方法：使用docker拉取鏡像就行；

docker pull registry.aliyuncs.com/google_containers/coredns:1.8.4

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 K8S使用---故障處理部署k8s（一） NFS故障處理 K8S集群安裝故障排查 k8s常見故障匯總 k8s集群部署 K8S部署之kubeadm k8s部署mysql k8s集群部署 kubeadm部署K8S