1、k8s 集群架構描述
- kubeadm v1.10.2創建k8s集群。
- master節點高可用，三節點（10.18.60.3、10.18.60.4、10.18.60.5）。
- LVS實現master三節點代理。
2、K8S集群證書過期，日志報錯如下
Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
3、故障排定，查看證書
# openssl x509 -noout -text -in apiserver-kubelet-client.crt | grep Not
Not Before: May 22 01:58:06 2018 GMT
Not After : May 22 01:58:07 2019 GMT # 2019-5-22日過期
4、kubeadm 命令介紹（v1.10.2）
注：本次只需用到以下兩個參數命令，其它參數不做介紹
# kubeadm alpha phase certs -h # 創建證書 Usage: kubeadm alpha phase certs [command] Available Commands: all Generates all PKI assets necessary to establish the control plane apiserver Generates an API server serving certificate and key apiserver-etcd-client Generates a client certificate for the API server to connect to etcd securely apiserver-kubelet-client Generates a client certificate for the API server to connect to the kubelets securely ca Generates a self-signed kubernetes CA to provision identities for components of the cluster etcd-ca Generates a self-signed CA to provision identities for etcd etcd-healthcheck-client Generates a client certificate for liveness probes to healthcheck etcd etcd-peer Generates an etcd peer certificate and key etcd-server Generates an etcd serving certificate and key front-proxy-ca Generates a front proxy CA certificate and key for a Kubernetes cluster front-proxy-client Generates a front proxy CA client certificate and key for a Kubernetes cluster sa Generates a private key for signing service account tokens along with its public key
# kubeadm alpha phase kubeconfig -h # 生成配置文件（例如：admin.conf|controller-manager.conf|kubelet.conf|scheduler.conf） Usage: kubeadm alpha phase kubeconfig [command] Available Commands: admin Generates a kubeconfig file for the admin to use and for kubeadm itself all Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file controller-manager Generates a kubeconfig file for the controller manager to use kubelet Generates a kubeconfig file for the kubelet to use. Please note that this should be used *only* for bootstrapping purposes. scheduler Generates a kubeconfig file for the scheduler to use user Outputs a kubeconfig file for an additional user
# kubeadm alpha phase certs apiserver -h
--apiserver-advertise-address string 填寫本機apiserver ip。
--apiserver-cert-extra-sans strings 多master節點，在創建apiserver證書時，需要指定每個節點的IP，代理IP、域名。
--cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki")
--config string Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)
-h, --help help for apiserver
--service-cidr string Alternative range of IP address for service VIPs, from which derives the internal API server VIP that will be added to the API Server serving cert (default "10.96.0.0/12")
--service-dns-domain string Alternative domain for services, to use for the API server serving cert (default "cluster.local")
5、備份節點配置文件與證書
# cp -rfp /etc/kubernetes /etc/kubernetes.2019.5.23
6、創建證書
注：
1、因為之前三個master節點的配置文件中全部填寫的是LVS VIP（沒做域名解析），為了今后切換方便給VIP配置了一個域名，而apiserver證書中沒有配置該域名的認證，所以利用openssl對apiserver證書做了重簽替換，並設置apiserver證書有效期10年，所以這次證書過期不涉及apiserver，只需要對apiserver-kubelet-client與front-proxy-client證書重新創建，而下邊給出了對apiserver證書重簽的命令。
2、創建證書時需要配置VPN，kubeadm需要連接國外
# kubeadm alpha phase certs apiserver --apiserver-advertise-address 10.18.60.3 --apiserver-cert-extra-sans 10.18.60.4 --apiserver-cert-extra-sans 10.18.60.5 --apiserver-cert-extra-sans 10.16.60.6 --apiserver-cert-extra-sans k8s.m.api # 創建apiserver證書 # kubeadm alpha phase certs apiserver-kubelet-client # 創建apiserver-kubelet-client證書 # kubeadm alpha phase certs front-proxy-client # 創建front-proxy-client證書
7、創建配置文件（admin.conf|controller-manager.conf|kubelet.conf|scheduler.conf）
# 創建完會看到/etc/kubernetes下面出現了配置文件 # kubeadm alpha phase kubeconfig all --apiserver-advertise-address 10.18.60.3
8、准備替換
因為我三節點的代理IP配置了域名，所以需要做下替換。
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' admin.conf
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' controller-manager.conf
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' scheduler.conf
# sed -i 's/10\.5\.38\.39/k8s.m.api/g' kubelet.conf
# grep 'host:' /etc/kubernetes/manifests/kube-apiserver.yaml
host: k8s.m.api
# 直接覆蓋（注意文件權限） # cp -rfp /etc/kubernetes/admin.conf ~/.kube/config
# kubelet 客戶端簽發的不需要備份 # rm -rf /var/lib/kubelet/pki/*
9、重啟服務
# 重啟本機所有docker容器 # docker restart $(docker ps -q) # 重啟kubelet # systemctl restart kubelet.service
10、驗證
# 可以看到已經恢復 # kubectl get node
11、恢復其它master節點
# 注意拷貝kubelet.conf文件到其它服務器（其它服務器自己生成）
# scp admin.conf controller-manager.conf scheduler.conf 10.18.60.4:/etc/kubernetes # scp admin.conf controller-manager.conf scheduler.conf 10.18.60.5:/etc/kubernetes
# scp apiserver-kubelet-client.crt apiserver-kubelet-client.key front-proxy-client.crt front-proxy-client.key 10.18.60.4:/etc/kubernetes/pki
# scp apiserver-kubelet-client.crt apiserver-kubelet-client.key front-proxy-client.crt front-proxy-client.key 10.18.60.5:/etc/kubernetes/pki
# 其它兩個節點只生成各自的kubelet配置文件
# kubeadm alpha phase kubeconfig kubelet
# grep 'server:' kubelet.conf
server: https://k8s.m.api:6443
# 直接覆蓋（注意文件權限） # cp -rfp /etc/kubernetes/admin.conf ~/.kube/config
# kubelet 客戶端簽發的不需要備份 # rm -rf /var/lib/kubelet/pki/*
# 重啟本機所有docker容器 # docker restart $(docker ps -q) # 重啟kubelet # systemctl restart kubelet.service
# 可以看到已經恢復 # kubectl get node