kubeadm 默認證書為一年，一年過期后，會導致api service不可用，使用過程中會出現：x509: certificate has expired or is not yet valid.
方案一 通過修改kubeadm 調整證書過期時間
修改代碼，調整過期時間
克隆代碼：git clone https://github.com/kubernetes/kubernetes.git, 切換到指定的tag或者版本修改vendor/k8s.io/client-go/util/cert/cert.go文件，git diff 對比如下：
diff --git a/staging/src/k8s.io/client-go/util/cert/cert.go b/staging/src/k8s.io/client-go/util/cert/cert.go index fb7f5fa..e800962 100644 --- a/staging/src/k8s.io/client-go/util/cert/cert.go +++ b/staging/src/k8s.io/client-go/util/cert/cert.go @@ -104,7 +104,7 @@ func NewSignedCert(cfg Config, key *rsa.PrivateKey, caCert *x509.Certificate, ca IPAddresses: cfg.AltNames.IPs, SerialNumber: serial, NotBefore: caCert.NotBefore, - NotAfter: time.Now().Add(duration365d).UTC(), + NotAfter: time.Now().Add(duration365d * 10).UTC(), KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature, ExtKeyUsage: cfg.Usages, } @@ -149,7 +149,7 @@ func GenerateSelfSignedCertKey(host string, alternateIPs []net.IP, alternateDNS CommonName: fmt.Sprintf("%s-ca@%d", host, time.Now().Unix()), }, NotBefore: time.Now(), - NotAfter: time.Now().Add(time.Hour * 24 * 365), + NotAfter: time.Now().Add(time.Hour * 24 * 3650), KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, BasicConstraintsValid: true,
編譯代碼
這里我們可以使用官方容器對代碼進行編譯。
k8s.gcr.io/kube-cross:v1.12.5-1，如果不能翻牆的用戶，可以拉取我個人下的鏡像skymyyang/kube-cross:v1.12.5-1
docker pull skymyyang/kube-cross:v1.12.5-1
編譯
docker run --rm -v 你修改后的代碼目錄:/go/src/k8s.io/kubernetes -it skymyyang/kube-cross:v1.12.5-1 bash cd /go/src/k8s.io/kubernetes # 編譯kubeadm, 這里主要編譯kubeadm 即可 make all WHAT=cmd/kubeadm GOFLAGS=-v # 編譯kubelet # make all WHAT=cmd/kubelet GOFLAGS=-v # 編譯kubectl # make all WHAT=cmd/kubectl GOFLAGS=-v #編譯完產物在 _output/bin/kubeadm 目錄下 #將kubeadm 文件拷貝出來，替換系統中的kubeadm
替換證書
#用新的kubeadm 替換官方的kubeadm chmod +x kubeadm && \cp -f kubeadm /usr/bin #備份原有的證書 mv /etc/kubernetes/pki /etc/kubernetes/pki.old #生成新的證書，kubeadm.yaml 指定你自己服務器上的 kubeadm alpha phase certs all --config ~/kubeadm.yaml #備份原有的conf文件 mv /etc/kubernetes/*conf /etc/kubernetes/*conf-old #根據新證書重新生成新的配置文件 kubeadm alpha phase kubeconfig all --config ~/kubeadm.yaml #替換老的config文件 \cp -f /etc/kubernetes/admin.conf ~/.kube/config
方案二 啟用自動輪換kubelet 證書
kubelet證書分為server和client兩種， k8s 1.9默認啟用了client證書的自動輪換，但server證書自動輪換需要用戶開啟
增加 kubelet 參數
# 在/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 增加如下參數 Environment="KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true"
增加 controller-manager 參數
# 在/etc/kubernetes/manifests/kube-controller-manager.yaml 添加如下參數 - command: - kube-controller-manager - --experimental-cluster-signing-duration=87600h0m0s - --feature-gates=RotateKubeletServerCertificate=true - ....
創建 rbac 對象
創建rbac對象，允許節點輪換kubelet server證書：
cat > ca-update.yaml << EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests/selfnodeserver verbs: - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubeadm:node-autoapprove-certificate-server roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:nodes EOF kubectl create –f ca-update.yaml
如果證書已經過期，如何進行重新簽發證書
針對kubeadm 1.13.x 及以上處理
准備kubeadm.conf 配置文件一份
apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration kubernetesVersion: v1.14.1 #-->這里改成你集群對應的版本 imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers #這里使用國內的鏡像倉庫，否則在重新簽發的時候會報錯：could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt"
重新簽發命令
kubeadm alpha certs renew all --config=/root/kubeadm.conf 運行如上命令會重新生成以下證書 #-- /etc/kubernetes/pki/apiserver.key #-- /etc/kubernetes/pki/apiserver.crt #-- /etc/kubernetes/pki/apiserver-etcd-client.key #-- /etc/kubernetes/pki/apiserver-etcd-client.crt #-- /etc/kubernetes/pki/apiserver-kubelet-client.key #-- /etc/kubernetes/pki/apiserver-kubelet-client.crt #-- /etc/kubernetes/pki/front-proxy-client.key #-- /etc/kubernetes/pki/front-proxy-client.crt #-- /etc/kubernetes/pki/etcd/healthcheck-client.key #-- /etc/kubernetes/pki/etcd/healthcheck-client.crt #-- /etc/kubernetes/pki/etcd/peer.key #-- /etc/kubernetes/pki/etcd/peer.crt #-- /etc/kubernetes/pki/etcd/server.key #-- /etc/kubernetes/pki/etcd/server.crt
完成后重啟kube-apiserver,kube-controller,kube-scheduler,etcd這4個容器
針對kubeadm 1.13.0（不包含1.13.0） 以下處理
移動證書和配置【注意！必須移動，不然會使用現有的證書，不會重新生成】
cd /etc/kubernetes mkdir ./pki_bak mkdir ./pki_bak/etcd mkdir ./conf_bak mv pki/apiserver* ./pki_bak/ mv pki/front-proxy-client.* ./pki_bak/ mv pki/etcd/healthcheck-client.* ./pki_bak/etcd/ mv pki/etcd/peer.* ./pki_bak/etcd/ mv pki/etcd/server.* ./pki_bak/etcd/ mv ./admin.conf ./conf_bak/ mv ./kubelet.conf ./conf_bak/ mv ./controller-manager.conf ./conf_bak/ mv ./scheduler.conf ./conf_bak/
創建證書
kubeadm alpha phase certs all --apiserver-advertise-address=${MASTER_API_SERVER_IP} --apiserver-cert-extra-sans=主機內網ip,主機公網ip
運行如上命令會重新生成以下證書
#-- /etc/kubernetes/pki/apiserver.key
#-- /etc/kubernetes/pki/apiserver.crt
#-- /etc/kubernetes/pki/apiserver-etcd-client.key
#-- /etc/kubernetes/pki/apiserver-etcd-client.crt
#-- /etc/kubernetes/pki/apiserver-kubelet-client.key
#-- /etc/kubernetes/pki/apiserver-kubelet-client.crt
#-- /etc/kubernetes/pki/front-proxy-client.key
#-- /etc/kubernetes/pki/front-proxy-client.crt
#-- /etc/kubernetes/pki/etcd/healthcheck-client.key
#-- /etc/kubernetes/pki/etcd/healthcheck-client.crt
#-- /etc/kubernetes/pki/etcd/peer.key
#-- /etc/kubernetes/pki/etcd/peer.crt
#-- /etc/kubernetes/pki/etcd/server.key
#-- /etc/kubernetes/pki/etcd/server.crt
不移動證書會有如下提示
#[certificates] Using the existing apiserver certificate and key.
#[certificates] Using the existing apiserver-kubelet-client certificate and key.
#[certificates] Using the existing front-proxy-client certificate and key.
#[certificates] Using the existing etcd/server certificate and key.
#[certificates] Using the existing etcd/peer certificate and key.
#[certificates] Using the existing etcd/healthcheck-client certificate and key.
#[certificates] Using the existing apiserver-etcd-client certificate and key.
#[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
#[certificates] Using the existing sa key.
生成新配置文件
kubeadm alpha phase kubeconfig all --apiserver-advertise-address=${MASTER_API_SERVER_IP}
將新生成的admin配置文件覆蓋掉原本的admin文件
mv $HOME/.kube/config $HOME/.kube/config.old cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config sudo chmod 777 $HOME/.kube/config
完成后重啟kube-apiserver,kube-controller,kube-scheduler,etcd這4個容器
如果有多台master，則將第一台生成的相關證書拷貝到其余master即可。
kubeadm 1.14 證書調整教程
kubeadm部署的kubernets證書一直都是個詬病，默認都只有一年有效期，kubeadm 1.14.x安裝后有部分證書還是一年有效期，但個別證書已修改為10年有效期，但對我們使用來說，一年有效期還是一個比較的坑，需要進行調整。
修改kubeadm 1.14.x源碼，調整證書過期時間
kubeadm1.14.x 安裝過后crt證書如下所示
/etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/front-proxy-ca.crt #10年有效期 /etc/kubernetes/pki/ca.crt #10年有效期 /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/front-proxy-client.crt #10年有效期 /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/ca.crt #10年有效期 /etc/kubernetes/pki/etcd/peer.crt #10年有效期 /etc/kubernetes/pki/etcd/healthcheck-client.crt /etc/kubernetes/pki/apiserver-kubelet-client.crt
如上所示，除了標注說明的證書為10年有效期，其余都是1年有效期，我們查看下原先調整證書有效期的源碼，克隆kubernetes 源碼，切換到1.14.1 tag 查看：
代碼目錄： staging/src/k8s.io/client-go/util/cert/cert.go
const duration365d = time.Hour * 24 * 365 func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) { now := time.Now() tmpl := x509.Certificate{ SerialNumber: new(big.Int).SetInt64(0), Subject: pkix.Name{ CommonName: cfg.CommonName, Organization: cfg.Organization, }, NotBefore: now.UTC(), //這里已經調整為10年有效期 NotAfter: now.Add(duration365d * 10).UTC(), KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, BasicConstraintsValid: true, IsCA: true, } certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key) if err != nil { return nil, err } return x509.ParseCertificate(certDERBytes) }
如上所示，通過NewSelfSignedCACert這個方法簽發的證書都默認為10年有效期了，但這個只影響部分證書，但這樣還沒滿足我們的需求，個別證書的有效期調整，在經過對源碼的分析后，找到了如下的邏輯：
發現部分證書是通過NewSignedCert這個方法簽發，而這個方法簽發的證書默認只有一年有效期，查看代碼邏輯：
代碼: cmd/kubeadm/app/util/pkiutil/pki_helpers.go
const duration365d = time.Hour * 24 * 365 func NewSignedCert(cfg *certutil.Config, key crypto.Signer, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, error) { serial, err := rand.Int(rand.Reader, new(big.Int).SetInt64(math.MaxInt64)) if err != nil { return nil, err } if len(cfg.CommonName) == 0 { return nil, errors.New("must specify a CommonName") } if len(cfg.Usages) == 0 { return nil, errors.New("must specify at least one ExtKeyUsage") } certTmpl := x509.Certificate{ Subject: pkix.Name{ CommonName: cfg.CommonName, Organization: cfg.Organization, }, DNSNames: cfg.AltNames.DNSNames, IPAddresses: cfg.AltNames.IPs, SerialNumber: serial, NotBefore: caCert.NotBefore, // 只有一年有效期 NotAfter: time.Now().Add(duration365d).UTC(), KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature, ExtKeyUsage: cfg.Usages, } certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &certTmpl, caCert, key.Public(), caKey) if err != nil { return nil, err } return x509.ParseCertificate(certDERBytes) }
至此，調整NewSignedCert這個方法，重新進行編譯，將證書有效期調整為你想要的任何時間。
原文鏈接：
http://icyxp.github.io/blog/2019/05/k8s-kubeadm14-ca-upgrade.html