k8s 二進制部署-啟動api-server 報錯 Error: unknown flag: --etcdservers ,啟動 kubelet Kubelet: cannot create certificate signing request

 

 

 systemctl status kube-apiserver  啟動失敗

查看錯誤日志

cat /var/log/messages|grep kube-apiserver|grep -i error

Jan 11 11:22:44 m1 kube-apiserver: --logtostderr                      log to standard error instead of files
Jan 11 11:25:16 m1 kube-apiserver: Error: unknown flag: --etcdservers
Jan 11 11:25:16 m1 kube-apiserver: --alsologtostderr                  log to standard error as well as files
Jan 11 11:25:16 m1 kube-apiserver: --logtostderr    

【Error: unknown flag: --etcdservers】 說明我的字符串寫錯了；

我復制教材的pdf內容； --etcdservers 復制出來粘貼到記事本,發現少了【-】符號；

chrome瀏覽器復制pdf -換行 內容到notepad++ 少了【-】符號

所以注意復制內容的差異；

 

 

 

 

修正后啟動成功

 

 https://www.jianshu.com/p/19e2d7cc94d6

作者辛苦了；文章步驟很清晰，但是有一句錯誤，會引發kubelet在master上啟動不了；

啟動kubelet報錯誤：【kubelet  failed to run Kubelet: cannot create certificate signing request】

CA證書和token對於kubelet啟動的重要性，注意這三步正確就能夠啟動kubelet成功；
1、token.csv
cat > /opt/kubernetes/cfg/token.csv << EOF
b1dc586d69159ff4e3ef7efa9db60e48,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

2、創建用戶
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
3、生成bootstrap.kubeconfig文件 （ca.pem填充不上，手動修改）
==========================
##設置環境變量
KUBE_APISERVER="https://172.16.210.53:6443" # apiserver IP:PORT
TOKEN="b1dc586d69159ff4e3ef7efa9db60e48" # 與token.csv里保持一致

# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user="kubelet-bootstrap" \
  --kubeconfig=bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

======================
上邊的命令：-certificate-authority在bootstrap.kubeconfig文件中會變成certificate-authority-data: AFSADFADFADFADFD（一堆字符碼）
###
可以直接vi編輯（如下模板可用），就沒有certificate-authority-data字符碼的問題；
#######################
[root@localhost bin]# cat bootstrap.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    server: https://192.168.79.140:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubelet-bootstrap
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    token: b1dc586d69159ff4e3ef7efa9db60e48
###########################

============================

執行如下命令生成kube-proxy.kubeconfig文件， 
kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

報錯如下；原因是*.pem文件沒有設置成功；授權失敗；解決方法參考【二進制部署k8s完整步驟和修正步驟】

[root@localhost logs]# more kube-proxy.FATAL
Log file created at: 2022/01/15 18:08:18
Running on machine: localhost
Binary: Built with gc go1.13.9 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0115 18:08:18.940662 12704 server.go:497] error loading config file "/opt/kubernetes/cfg/kube-proxy.kubeconfig": v1.Config.AuthInfos: []v1.NamedAuthInfo: v1.Nam
edAuthInfo.AuthInfo: v1.AuthInfo.ClientKeyData: ClientCertificateData: decode base64: illegal base64 data at input byte 24, error found in #10 byte of ...|proxy.pe
m","client-k|..., bigger context ...|ificate-data":"/opt/kubernetes/ssl/kube-proxy.pem","client-key-data":"/opt/kubernetes/ssl/kube-proxy|

https://blog.csdn.net/weixin_39608791/article/details/108881130

https://blog.csdn.net/IvyXYW/article/details/115710665

【failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is fo
rbidden: User "10001" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
】

cat > /opt/kubernetes/cfg/token.csv << EOF
b1dc586d69159ff4e3ef7efa9db60e48,10001,"system:node-bootstrapper"   ## 錯誤在與沒有指定用戶
EOF
改為：
cat > /opt/kubernetes/cfg/token.csv << EOF
b1dc586d69159ff4e3ef7efa9db60e48,kubelet-bootstrap,10001,"system:nodebootstrapper"
EOF


重新生成授權[kubelet-bootstrap]
【kubectl delete clusterrolebindings kubelet-bootstrap】 刪除舊的；

【kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap】

 

忽然斷電后，etcd啟動失敗：修復
https://blog.csdn.net/u013958257/article/details/106978416

備份 cp -r  /var/lib/etcd/default.etcd /var/lib/etcd/default.etcd.bak
刪除 rm -rf /var/lib/etcd/default.etcd/*
kubectl describe pod web-65b7447c7-cxnqn
 kubectl get pods -n kube-system
  systemctl status flanneld
 kubectl apply -f kube-flannel.yml
 kubectl get pods -n kube-system
 kubectl get nodes
 kubectl get pods
 kubectl describe pod web-65b7447c7-cxnqn

　　

[root@master ~]# kubectl logs pi-nb5ds
Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log pi-nb5ds)
[root@master ~]# kubectl create clusterrolebinding kubernetes --clusterrole=cluster-admin --user=kubernetes
clusterrolebinding.rbac.authorization.k8s.io/kubernetes created
[root@master ~]# kubectl logs pi-nb5ds
3.1415926535897932384626433832795028841971693993

　　

 

二進制部署k8s完整步驟和修正步驟
https://www.jianshu.com/p/19e2d7cc94d6

3. 啟用 TLS Bootstrapping 機制
   創建上述配置文件中token文件：
  修正為：
cat > /opt/kubernetes/cfg/token.csv << EOF
b1dc586d69159ff4e3ef7efa9db60e48,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

五.3. 生成bootstrap.kubeconfig文件
不用執行kubectl命令，命令執行不會自動填充ca.pem內容；
改為直接使用如內容；

[root@localhost bin]# cat bootstrap.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    server: https://192.168.79.140:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubelet-bootstrap
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    token: b1dc586d69159ff4e3ef7efa9db60e48


5.4.3  生成kube-proxy.kubeconfig文件  
不執行kubectl命令，因為會導致.pem不能自動填寫；
改為
[root@localhost k8s]# cat kube-proxy.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /opt/kubernetes/ssl/ca.pem
    server: https://192.168.79.140:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kube-proxy
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-proxy
  user:
    client-certificate: /opt/kubernetes/ssl/kube-proxy.pem
    client-key: /opt/kubernetes/ssl/kube-proxy-key.pem

部署好部署CNI網絡后，kube-flannel.yml；部署nginx測試一下

[root@localhost k8s]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@localhost k8s]# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
[root@localhost k8s]# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-f89759699-fbdw9 1/1 Running 0 40s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 3h54m
service/nginx NodePort 10.0.0.4 <none> 80:31254/TCP 13s
瀏覽器：
http://192.168.79.140:31254/

　5.5部署CNI網絡

關於cni和flanel安裝規划
cni安裝到每台node節點
flannel安裝到master節點
https://www.cnblogs.com/TSir/p/12240825.html

　在master上執行kubectl get nodes查看結果是否READY

5.5部署CNI網絡完成后即為完成；