# kubeadm安裝的k8s證書過期處理
## 一、背景說明
1. kubeadm默認證書為一年，一年過期后，會導致api service不可用，使用過程中會出現：`x509: certificate has expired or is not yet valid`
2. kubelet證書分為`server`和`client`兩種， k8s1.9開始默認啟用了client證書的自動輪換，但server證書自動輪換需要用戶配置開啟
## 二、開啟server證書自動輪換
> 此方案適用於證書還未過期
### 1. 增加kubelet參數
``` bash
#在/etc/sysconfig/kubelet增加，若多master，都需要配置：
KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true --rotate-server-certificates=true
```
### 2. 配置kube-controller-manager
``` yaml
cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --experimental-cluster-signing-duration=87600h0m0s #增加證書頒發時間參數
- --feature-gates=RotateKubeletServerCertificate=true #開啟server證書簽發
- --allocate-node-cidrs=true
```
### 3. 創建rbac對象，允許節點輪換kubelet server證書
``` bash
cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/selfnodeserver
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:node-autoapprove-certificate-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
EOF
kubectl apply -f ca-update.yaml
```
### 4. 重啟kubelet
``` bash
systemctl restart kubelet
#查看csr,狀態會由Pending to Approved
kubectl get csr
```
### 5. 多master其他節點一直處於Pending
``` bash
#出於安全原因,處於pending狀態的master節點需要手動審批
kubectl certificate approve <name>
```
## 三、替換server證書
> 此方案適用於證書已過期，處理完成后，再執行`開啟server證書自動輪換`
### 1. 報錯信息
``` bash
kubectl get po
Unable to connect to the server: x509: certificate has expired or is not yet valid
```
### 2. 證書備份
``` bash
cp -Ra /etc/kubernetes /opt/kubernetes-backup-time
```
### 3. 刪除過期證書
``` bash
#apiserver證書
rm -f /etc/kubernetes/pki/apiserver*
#front-proxy-client證書
rm -f /etc/kubernetes/pki/front-proxy-client.*
#etcd證書，若etcd是部署在集群外自簽證書，不執行以下命令
rm -rf /etc/kubernetes/pki/etcd/healthcheck-client.*
rm -rf /etc/kubernetes/pki/etcd/server.*
rm -rf /etc/kubernetes/pki/etcd/peer.*
```
### 4. 重新生成證書
``` bash
#下載對應版本的kubeadm
wget https://dl.k8s.io/release/v1.10.1/bin/linux/amd64/kubeadm
chmod a+x kubeadm
#生成證書,若使用HA需要配置成vip地址
./kubeadm alpha phase certs all --apiserver-advertise-address <IP address of your master server>
```
### 5. 重新生成配置文件
``` bash
#備份配置文件
mv /etc/kubernetes/*.conf /tmp
#生成配置文件
./kubeadm alpha phase kubeconfig all --apiserver-advertise-address <IP address of your master server>
```
### 6. 重啟kubelet
``` bash
systemctl restart kubelet
```
### 7. 驗證集群
``` bash
#查看證書過期時間
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
#集群節點狀態
kubectl get no
```
## 四、參考
* [kubelet-tls-bootstrapping](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping)
* [certificate-rotation](https://kubernetes.io/docs/tasks/tls/certificate-rotation/)