1、kube-apiserver、kube-controller-manager啟動文件里面的公鑰文件、私鑰文件竟然是一樣的，還怎么配對，這是源文檔的大坑 查看源文檔kube-apiserver啟動文件中公鑰 cat etc/systemd/system/kube-apiserver.service --service-account-key-file=/etc/kubernetes/cert/ca-key.pem 查看源文檔kube-controller-manager啟動文件中私鑰 cat /etc/systemd/system/kube-controller-manager.service --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem 發現公鑰和私鑰都是用的一個私鑰，這是不可能配對的，而且原文中已經說明需要配對，說明如下： 原版文件在apiserver啟動文字中已經說明 > --service-account-key-file：簽名 ServiceAccount Token 的公鑰文件，kube-controller-manager 的 --service-account-private-key-file 指定私鑰文件，兩者配對使用； 但是原版中apiserver、controller-manager這兩個啟動文字里面用的私有、公鑰都是一個，所以懷疑出錯， 按照另一篇對照修改的：https://www.cnblogs.com/effortsing/p/10312081.html，需要修改如下： 生成 service account key cd /etc/kubernetes/ openssl genrsa -out /etc/kubernetes/sa.key 2048 openssl rsa -in /etc/kubernetes/cert/sa.key -pubout -out /etc/kubernetes/cert/sa.pub ls /etc/kubernetes/pki/sa.* cd $HOME 分發service account key到所有master節點 subprocess.call(["ansible k8s -m copy -a 'src=/etc/kubernetes/sa.key dest=/etc/kubernetes/cert/ force=yes'"], shell=True) subprocess.call(["ansible k8s -m copy -a 'src=/etc/kubernetes/sa.pub dest=/etc/kubernetes/cert/ force=yes'"], shell=True) 修改kube-apiserver啟動文件中公鑰為sa.pub cat etc/systemd/system/kube-apiserver.service --service-account-private-key-file=/etc/kubernetes/cert/sa.pub 修改kube-controller-manager啟動文件中私鑰為sa.key cat /etc/systemd/system/kube-controller-manager.service --service-account-private-key-file=/etc/kubernetes/cert/sa.key 2、源文檔kube-controller-manager啟動文件里面少兩個致命參數，導致flannel啟動失敗，缺少 --allocate-node-cidrs=true --cluster-cidr=172.30.0.0/16 flannel啟動失敗，報錯如下 Error registering network: failed to acquire lease: node "test4" pod cidr not assigned 查看pod [root@test4 profile]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE kube-flannel-ds-gzvrh 0/1 Error 0 <invalid> 通過docker查看flannel日志 [root@test4 profile]# docker ps -l CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f7be3ebe77fd b949a39093d6 "/opt/bin/flanneld -…" 1 second ago Created k8s_kube-flannel_kube-flannel-ds-7cqww_kube-system_26fab004-2b88-11e9-9085-000c2935f634_0 [root@test4 profile]# docker logs f7be3ebe77fd I0208 09:58:34.068723 1 main.go:488] Using interface with name ens33 and address 192.168.0.94 I0208 09:58:34.069094 1 main.go:505] Defaulting external address to interface address (192.168.0.94) I0208 09:58:34.376952 1 kube.go:131] Waiting 10m0s for node controller to sync I0208 09:58:34.466001 1 kube.go:294] Starting kube subnet manager I0208 09:58:35.481478 1 kube.go:138] Node controller sync successful I0208 09:58:35.481666 1 main.go:235] Created subnet manager: Kubernetes Subnet Manager - test4 I0208 09:58:35.481694 1 main.go:238] Installing signal handlers I0208 09:58:35.482001 1 main.go:353] Found network config - Backend type: vxlan I0208 09:58:35.482255 1 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false E0208 09:58:35.483159 1 main.go:280] Error registering network: failed to acquire lease: node "test4" pod cidr not assigned I0208 09:58:35.483433 1 main.go:333] Stopping shutdownHandler... 看到：Error registering network: failed to acquire lease: node "test4" pod cidr not assigned 原因：是因為按照二進制高可用文檔安裝配置kube-controller-manager啟動文件的時候，有坑，當時沒有發現， 解決： 啟動文件加上下面兩句話，那篇文檔沒有加，所以報錯；下面這個cluster-cidr要和kube-flannel.yml里面的地址一致，要和kube-proxy.config.yaml里面的clusterCIDR一致 --allocate-node-cidrs=true \ --cluster-cidr=172.30.0.0/16 \ 4、源文檔中的kubelet啟動參數中都沒有帶cadvisor監控服務參數，源文檔中就可以訪問cadvisor監控了。明擺着安裝完把cadvisor參數去掉了，這是大坑 5、執行查看資源報錯： unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy) [root@test4 ~]# kubectl exec -it http-test-dm2-6dbd76c7dd-cv9qf sh error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy) 解決：創建apiserver到kubelet的權限 注意：user=kubernetes ，這個user要替換掉下面yaml文件里面的用戶名 cat > apiserver-to-kubelet.yaml <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kubernetes-to-kubelet rules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kubernetes namespace: "" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kubernetes-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubernetes EOF 創建授權: kubectl create -f apiserver-to-kubelet.yaml [root@test4 ~]# kubectl create -f apiserver-to-kubelet.yaml clusterrole.rbac.authorization.k8s.io/system:kubernetes-to-kubelet created clusterrolebinding.rbac.authorization.k8s.io/system:kubernetes created 重新進到容器查看資源 [root@test4 ~]# kubectl exec -it http-test-dm2-6dbd76c7dd-cv9qf sh / # exit 現在可以進到容器里面查看資源了 參照文檔：https://www.jianshu.com/p/b3d8e8b8fd7e 6、源文檔中kube-apiserver啟動參數中沒有這句話 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname 這句話是必須要加的否則用kubectl查看資源、創建dnstools工具時候會報如下錯誤： [root@test4 profile]# kubectl run -it --rm --image=infoblox/dnstools dns-client kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. If you don't see a command prompt, try pressing enter. Error attaching, falling back to logs: error dialing backend: dial tcp 0.0.0.0:10250: connect: connection refused deployment.apps "dns-client" deleted Error from server: Get https://test4:10250/containerLogs/default/dns-client-86c6d59f7-tzh5c/dns-client: dial tcp 0.0.0.0:10250: connect: connection refused [root@test4 ~]# kubectl exec -it http-test-dm2-6dbd76c7dd-cv9qf sh error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy) 7、源文檔中kube-apiserver啟動參數中--enable-admission-plugins= 這個選項里面的參數很少，導致各種錯誤，必須填寫全，如下： --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \