證書更新
- 默認證書一年有效期
- 一旦證書過期，使用kubectl時會出現如下提示：`Unable to connect to the server: x509: certificate has expired or is not yet valid`
查看證書過期情況
1 [root@k8s-test-master-1 ~]# kubeadm alpha certs check-expiration 2 CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED 3 admin.conf Jul 28, 2020 05:41 UTC 364d no 4 apiserver Jul 28, 2020 05:41 UTC 364d no 5 apiserver-etcd-client Jul 28, 2020 05:41 UTC 364d no 6 apiserver-kubelet-client Jul 28, 2020 05:41 UTC 364d no 7 controller-manager.conf Jul 28, 2020 05:41 UTC 364d no 8 etcd-healthcheck-client Jul 28, 2020 05:41 UTC 364d no 9 etcd-peer Jul 28, 2020 05:41 UTC 364d no 10 etcd-server Jul 28, 2020 05:41 UTC 364d no 11 front-proxy-client Jul 28, 2020 05:41 UTC 364d no 12 scheduler.conf Jul 28, 2020 05:41 UTC 364d no 13 14 # 查看根CA證書的有效期（十年） 15 [root@k8s-test-master-1 pki]# cd /etc/kubernetes/pki 16 [root@k8s-test-master-1 pki]# ls | grep ca.crt | xargs -I {} openssl x509 -text -in {} | grep "Not After" 17 Not After : Jul 26 05:41:23 2029 GMT 18 Not After : Jul 26 05:41:23 2029 GMT
證書目錄結構
[root@k8s-test-master-1 pki]# pwd /etc/kubernetes/pki [root@k8s-test-master-1 pki]# tree . . ├── apiserver.crt ├── apiserver-etcd-client.crt ├── apiserver-etcd-client.key ├── apiserver.key ├── apiserver-kubelet-client.crt ├── apiserver-kubelet-client.key ├── ca.crt ├── ca.key ├── etcd │ ├── ca.crt │ ├── ca.key │ ├── healthcheck-client.crt │ ├── healthcheck-client.key │ ├── peer.crt │ ├── peer.key │ ├── server.crt │ └── server.key ├── front-proxy-ca.crt ├── front-proxy-ca.key ├── front-proxy-client.crt ├── front-proxy-client.key ├── sa.key └── sa.pub 1 directory, 22 files
Kubernetes 集群根證書
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
由此根證書簽發的證書有:
- 1,kube-apiserver 組件持有的服務端證書
/etc/kubernetes/pki/apiserver.crt
/etc/kubernetes/pki/apiserver.key
- 2,kubelet 組件持有的客戶端證書
/etc/kubernetes/pki/apiserver-kubelet-client.crt
/etc/kubernetes/pki/apiserver-kubelet-client.key
kubelet 上一般不會明確指定服務端證書, 而是只指定 ca 根證書, 讓 kubelet 根據本地主機信息自動生成服務端證書並保存到配置的cert-dir文件夾中。
匯聚層(aggregator)證書
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
由此根證書簽發的證書只有一組:
- 1,代理端使用的客戶端證書, 用作代用戶與 kube-apiserver 認證
/etc/kubernetes/pki/front-proxy-client.crt
/etc/kubernetes/pki/front-proxy-client.key
etcd 集群根證書
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key
由此根證書簽發機構簽發的證書有:
- 1,etcd server 持有的服務端證書
/etc/kubernetes/pki/etcd/server.crt
/etc/kubernetes/pki/etcd/server.key
- 2,peer 集群中節點互相通信使用的客戶端證書
/etc/kubernetes/pki/etcd/peer.crt
/etc/kubernetes/pki/etcd/peer.key
- 3,pod 中定義 Liveness 探針使用的客戶端證書
/etc/kubernetes/pki/etcd/healthcheck-client.crt
/etc/kubernetes/pki/etcd/healthcheck-client.key
- 4,配置在 kube-apiserver 中用來與 etcd server 做雙向認證的客戶端證書
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-etcd-client.key
Serveice Account秘鑰
這組的密鑰對兒僅提供給 kube-controller-manager 使用. kube-controller-manager 通過 sa.key 對 token 進行簽名, master 節點通過公鑰 sa.pub 進行簽名的驗證.
API Server的authenticating環節支持多種身份校驗方式：client cert、bearer token、static password auth等，這些方式中有一種方式通過authenticating（Kubernetes API Server會逐個方式嘗試），那么身份校驗就會通過。一旦API Server發現client發起的request使用的是service account token的方式，API Server就會自動采用signed bearer token方式進行身份校驗。而request就會使用攜帶的service account token參與驗證。該token是API Server在創建service account時用API server啟動參數：–service-account-key-file的值簽署(sign)生成的。如果–service-account-key-file未傳入任何值，那么將默認使用–tls-private-key-file的值，即API Server的私鑰（server.key）。
通過authenticating后，API Server將根據Pod username所在的group：system:serviceaccounts和system:serviceaccounts:(NAMESPACE)的權限對其進行authority 和admission control兩個環節的處理。在這兩個環節中，cluster管理員可以對service account的權限進行細化設置。
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
kubeadm 創建的集群, kube-proxy ,flannel,coreDNS是以 pod 形式運行的, 在 pod 中, 直接使用 service account 與 kube-apiserver 進行認證, 此時就不需要再單獨為 kube-proxy 創建證書
更新證書
生成集群配置的yaml文件
1 kubeadm config view > /root/kubeadm.yaml
- kubeadm.yaml
1 apiServer: 2 extraArgs: 3 authorization-mode: Node,RBAC 4 timeoutForControlPlane: 4m0s 5 apiVersion: kubeadm.k8s.io/v1beta2 6 certificatesDir: /etc/kubernetes/pki 7 clusterName: kubernetes-test 8 controlPlaneEndpoint: 10.8.28.200:6443 9 controllerManager: {} 10 dns: 11 type: CoreDNS 12 etcd: 13 local: 14 dataDir: /data/etcd 15 imageRepository: k8s.gcr.io 16 kind: ClusterConfiguration 17 kubernetesVersion: v1.15.1 18 networking: 19 dnsDomain: cluster.local 20 podSubnet: 192.168.0.0/16 21 serviceSubnet: 10.96.0.0/12 22 scheduler: {}
證書更新使用幫助
[root@k8s-test-master-1 ~]# kubeadm alpha certs renew --help This command is not meant to be run on its own. See list of available subcommands. Usage: kubeadm alpha certs renew [flags] kubeadm alpha certs renew [command] Available Commands: admin.conf Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself all Renew all available certificates apiserver Renew the certificate for serving the Kubernetes API apiserver-etcd-client Renew the certificate the apiserver uses to access etcd apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet controller-manager.conf Renew the certificate embedded in the kubeconfig file for the controller manager to use etcd-healthcheck-client Renew the certificate for liveness probes to healtcheck etcd etcd-peer Renew the certificate for etcd nodes to communicate with each other etcd-server Renew the certificate for serving etcd front-proxy-client Renew the certificate for the front proxy client scheduler.conf Renew the certificate embedded in the kubeconfig file for the scheduler manager to use
更新證書操作
每個Master操作
kubeadm alpha certs renew all --config=/root/kubeadm.yaml # (也可以逐個更新) certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healtcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed # 再次查詢證書期限 [root@k8s-test-master-1 ~]# kubeadm alpha certs check-expiration CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED admin.conf Jul 29, 2020 06:47 UTC 364d no apiserver Jul 29, 2020 06:47 UTC 364d no apiserver-etcd-client Jul 29, 2020 06:47 UTC 364d no apiserver-kubelet-client Jul 29, 2020 06:47 UTC 364d no controller-manager.conf Jul 29, 2020 06:47 UTC 364d no etcd-healthcheck-client Jul 29, 2020 06:47 UTC 364d no etcd-peer Jul 29, 2020 06:47 UTC 364d no etcd-server Jul 29, 2020 06:47 UTC 364d no front-proxy-client Jul 29, 2020 06:47 UTC 364d no scheduler.conf Jul 29, 2020 06:47 UTC 364d no # 在三台Master上執行重啟kube-apiserver,kube-controller,kube-scheduler,etcd這4個容器，使證書生效 docker ps |grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' |xargs docker restart