Kubelet bootstrap認證配置步驟

kubelet 授權 kube-apiserver 的一些操作 exec run logs 等

RBAC 只需創建一次就可以

kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

 

創建 bootstrap kubeconfig 文件

注意: token 生效時間為 1day , 超過時間未創建自動失效，需要重新創建 token

kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:kubernetes-clientgroup --kubeconfig ~/.kube/config

查看生成的 token

kubeadm token list --kubeconfig ~/.kube/config

 TOKEN                    TTL  EXPIRES                    USAGES                  DESCRIPTION              EXTRA GROUPS
 2kcmsb.hyl5s4g0l1mkff9z  23h  2018-11-16T11:08:00+08:00  authentication,signing  kubelet-bootstrap-token  system:bootstrappers:kubernetes-clientgroup

配置集群參數，生成kubernetes-clientgroup-bootstrap.kubeconfig

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=https://192.168.1.7:6443 \  #master節點ip
  --kubeconfig=kubernetes-clientgroup-bootstrap.kubeconfig

配置客戶端認證

kubectl config set-credentials kubelet-bootstrap \
  --token= 2kcmsb.hyl5s4g0l1mkff9z \  #上面生成的token --kubeconfig=kubernetes-clientgroup-bootstrap.kubeconfig

配置關聯

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=kubernetes-clientgroup-bootstrap.kubeconfig

配置默認關聯

kubectl config use-context default --kubeconfig=kubernetes-clientgroup-bootstrap.kubeconfig

拷貝生成的 kubernetes-clientgroup-bootstrap.kubeconfig 文件到其它所有的node節點，並重命名

scp kubernetes-clientgroup-bootstrap.kubeconfig 192.168.1.8:/etc/kubernetes/bootstrap.kubeconfig

 

配置 bootstrap RBAC 權限

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers

否則報如下錯誤
failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:1jezb7" cannot create
certificatesigningrequests.certificates.k8s.io at the cluster scope

 

創建自動批准相關 CSR 請求的 ClusterRole

vi /etc/kubernetes/tls-instructs-csr.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]

導入 yaml 文件

kubectl apply -f /etc/kubernetes/tls-instructs-csr.yaml

clusterrole.rbac.authorization.k8s.io "system:certificates.k8s.io:certificatesigningrequests:selfnodeserver" created

查看創建的ClusterRole

kubectl describe ClusterRole/system:certificates.k8s.io:certificatesigningrequests:selfnodeserver

 

將 ClusterRole 綁定到適當的用戶組

# 自動批准 system:bootstrappers 組用戶 TLS bootstrapping 首次申請證書的 CSR 請求
kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient --group=system:bootstrappers 
# 自動批准 system:nodes 組用戶更新 kubelet 自身與 apiserver 通訊證書的 CSR 請求
kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes
# 自動批准 system:nodes 組用戶更新 kubelet 10250 api 端口證書的 CSR 請求
kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver --group=system:nodes

查看已有綁定 kubectl get clusterrolebindings

 

動態 kubelet 配置

創建kubelet服務文件

mkdir -p /var/lib/kubelet

vim /etc/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
 
ExecStart=/usr/local/bin/kubelet \
  --hostname-override=k8s-wjoyxt \   #本地node節點的hostname   --pod-infra-container-image=jicki/pause-amd64:3.1 \   #pod的基礎鏡像，即gcr的gcr.io/google_containers/pause-amd64:3.1鏡像 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet.config.json \
  --cert-dir=/etc/kubernetes/ssl \
  --logtostderr=true \
  --v=2
[Install]
WantedBy=multi-user.target

創建 kubelet config 配置文件

vim /etc/kubernetes/kubelet.config.json

{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "172.16.6.66", #本地node節點的IP "port": 10250,
  "readOnlyPort": 0,
  "cgroupDriver": "cgroupfs",
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "RotateCertificates": true,
  "featureGates": {
    "RotateKubeletClientCertificate": true,
    "RotateKubeletServerCertificate": true
  },
  "MaxPods": "512",
  "failSwapOn": false,
  "containerLogMaxSize": "10Mi",
  "containerLogMaxFiles": 5,
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.254.0.2"]
}

以上配置中:

cluster.local. 為 kubernetes 集群的 domain
10.254.0.2 預分配的 dns 地址
"clusterDNS": ["10.254.0.2"] 可配置多個 dns地址，逗號可開, 可配置宿主機dns

 

啟動Kubelet服務

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

驗證nodes

注意:這里的 ROLES 是節點標簽
關於 kubectl get node 中的 ROLES 的標簽
單 Master 打標簽 kubectl label node es-60 node-role.kubernetes.io/master=""，當標簽為 NoSchedule，表示不進行資源調度
更新標簽命令為 kubectl label nodes es-60 node-role.kubernetes.io/master=:NoSchedule --overwrite
單 Node 打標簽 kubectl label node es-61 node-role.kubernetes.io/node=""
關於刪除 label 可使用 - 號相連 如: kubectl label nodes es-61 node-role.kubernetes.io/node-

 

查看自動生成的證書配置文件

ls -lt /etc/kubernetes/ssl/kubelet-*