K8S從入門到放棄系列-(6)kubernetes集群之kube-controller-manager部署

摘要：

1、Kubernetes控制器管理器是一個守護進程它通過apiserver監視集群的共享狀態，並進行更改以嘗試將當前狀態移向所需狀態。

2、kube-controller-manager是有狀態的服務，會修改集群的狀態信息。如果多個master節點上的相關服務同時生效，則會有同步與一致性問題，所以多master節點中的kube-controller-manager服務只能是主備的關系，kukubernetes采用租賃鎖（lease-lock）實現leader的選舉，具體到kube-controller-manager，設置啟動參數"--leader-elect=true"。

1）創建kube-conftroller-manager證書簽名請求

1、kube-controller-mamager連接 apiserver 需要使用的證書，同時本身 10257 端口也會使用此證書

2、kube-controller-mamager與kubei-apiserver通信采用雙向TLS認證 

[root@k8s-master01 ~]# vim /opt/k8s/certs/kube-controller-manager-csr.json
{
    "CN": "system:kube-controller-manager",
    "hosts": [
      "127.0.0.1",
      "10.10.0.18",
      "10.10.0.19",
      "10.10.0.20",
      "localhost"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "ShangHai",
            "L": "ShangHai",
            "O": "system:kube-controller-manager",
            "OU": "System"
        }
    ]
}

1、hosts 列表包含所有 kube-controller-manager 節點 IP;

2、CN 為 system:kube-controller-manager;O 為 system:kube-controller-manager;kube-apiserver預定義的 RBAC使用的ClusterRoleBindings system:kube-controller-manager將用戶system:kube-controller-manager與ClusterRole system:kube-controller-manager綁定。

2）生成kube-controller-manager證書與私鑰

[root@k8s-master01 ~]# cd /opt/k8s/certs/
[root@k8s-master01 certs]# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
     -ca-key=/etc/kubernetes/ssl/ca-key.pem \
     -config=/opt/k8s/certs/ca-config.json \
     -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
2019/04/24 13:03:36 [INFO] generate received request
2019/04/24 13:03:36 [INFO] received CSR
2019/04/24 13:03:36 [INFO] generating key: rsa-2048
2019/04/24 13:03:36 [INFO] encoded CSR
2019/04/24 13:03:36 [INFO] signed certificate with serial number 461545639209226313174106252389263020486388400892
2019/04/24 13:03:36 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

3）查看證書

[root@k8s-master01 certs]# ll kube-controller-manager*
-rw-r--r-- 1 root root 1155 Apr 24 13:03 kube-controller-manager.csr
-rw-r--r-- 1 root root  432 Apr 24 13:00 kube-controller-manager-csr.json
-rw------- 1 root root 1679 Apr 24 13:03 kube-controller-manager-key.pem
-rw-r--r-- 1 root root 1529 Apr 24 13:03 kube-controller-manager.pem

4）分發證書

[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-controller-manager-key.pem dest=/etc/kubernetes/ssl/'
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-controller-manager.pem dest=/etc/kubernetes/ssl/'

5）生成配置文件kube-controller-manager.kubeconfig

 kube-controller-manager 組件開啟安全端口及RBAC認證所需配置

## 配置集群參數
### --kubeconfig：指定kubeconfig文件路徑與文件名；如果不設置，默認生成在~/.kube/config文件。
### 后面需要用到此文件，所以我們把配置信息單獨指向到指定文件中
[root@k8s-master01 ~]# kubectl config set-cluster kubernetes \
   --certificate-authority=/etc/kubernetes/ssl/ca.pem \
   --embed-certs=true \
   --server=https://127.0.0.1:6443 \
   --kubeconfig=kube-controller-manager.kubeconfig
Cluster "kubernetes" set.
## 配置客戶端認證參數
### --server：指定api-server，若不指定，后面腳本中，可以指定master
### 認證用戶為前文簽名中的"system:kube-controller-manager";
[root@k8s-master01 ~]# kubectl config set-credentials system:kube-controller-manager \
   --client-certificate=/etc/kubernetes/ssl/kube-controller-manager.pem \
   --embed-certs=true \
   --client-key=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
   --kubeconfig=kube-controller-manager.kubeconfig
User "system:kube-controller-manager" set
## 配置上下文參數
[root@k8s-master01 ~]# kubectl config set-context system:kube-controller-manager@kubernetes \
   --cluster=kubernetes \
   --user=system:kube-controller-manager \
   --kubeconfig=kube-controller-manager.kubeconfig
Context "system:kube-controller-manager@kubernetes" created.
## 配置默認上下文
[root@k8s-master01 ~]# kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=kube-controller-manager.kubeconfig
Switched to context "system:kube-controller-manager@kubernetes".

## 分發生成的配置文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/root/kube-controller-manager.kubeconfig dest=/etc/kubernetes/config/'

6）編輯kube-controller-manager核心文件

controller manager 將不安全端口 10252 綁定到 127.0.0.1 確保 kuebctl get cs 有正確返回;將安全端口 10257 綁定到 0.0.0.0 公開，提供服務調用;由於controller manager開始連接apiserver的6443認證端口,所以需要 --use-service-account-credentials 選項來讓 controller manager 創建單獨的 service account(默認 system:kube-controller-manager 用戶沒有那么高權限)

[root@k8s-master01 ~]# vim /opt/k8s/cfg/kube-controller-manager.conf
###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 \
                             --authentication-kubeconfig=/etc/kubernetes/config/kube-controller-manager.kubeconfig \
                             --authorization-kubeconfig=/etc/kubernetes/config/kube-controller-manager.kubeconfig \
                             --bind-address=0.0.0.0 \
                             --cluster-name=kubernetes \
                             --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
                             --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
                             --client-ca-file=/etc/kubernetes/ssl/ca.pem \
                             --controllers=*,bootstrapsigner,tokencleaner \
                             --deployment-controller-sync-period=10s \
                             --experimental-cluster-signing-duration=87600h0m0s \
                             --enable-garbage-collector=true \
                             --kubeconfig=/etc/kubernetes/config/kube-controller-manager.kubeconfig \
                             --leader-elect=true \
                             --node-monitor-grace-period=20s \
                             --node-monitor-period=5s \
                             --port=10252 \
                             --pod-eviction-timeout=2m0s \
                             --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem \
                             --terminated-pod-gc-threshold=50 \
                             --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
                             --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
                             --root-ca-file=/etc/kubernetes/ssl/ca.pem \
                             --secure-port=10257 \
                             --service-cluster-ip-range=10.254.0.0/16 \
                             --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
                             --use-service-account-credentials=true \
                             --v=2"
## 分發kube-controller-manager配置文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/kube-controller-manager.conf dest=/etc/kubernetes/config'

參數說明：

• address/bind-address：默認值：0.0.0.0，監聽--secure-port端口的IP地址。關聯的接口必須由集群的其他部分和CLI/web客戶端訪問。
• cluster-name：集群名稱
• cluster-signing-cert-file/cluster-signing-key-file：用於集群范圍認證
• controllers：啟動的contrller列表，默認為”*”，啟用所有的controller，但不包含” bootstrapsigner”與”tokencleaner”;
• kubeconfig：帶有授權和master位置信息的kubeconfig文件路徑
• leader-elect：在執行主邏輯之前，啟動leader選舉，並獲得leader權
• service-cluster-ip-range：集群service的IP地址范圍

8）啟動腳本

[root@k8s-master01 ~]# vim /opt/k8s/unit/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/config/kube-controller-manager.conf
User=kube
ExecStart=/usr/local/bin/kube-controller-manager \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
## 分發啟動腳本
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/unit/kube-controller-manager.service dest=/usr/lib/systemd/system/'

9）啟動服務

[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl daemon-reload'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl enable kube-controller-manager'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl start kube-controller-manager'

10）查看leader主機

[root@k8s-master01 ~]# kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master01_aef1b777-6658-11e9-beb0-000c295aa452","leaseDurationSeconds":15,"acquireTime":"2019-04-24T06:18:04Z","renewTime":"2019-04-24T06:20:43Z","leaderTransitions":2}'
  creationTimestamp: "2019-04-24T05:55:13Z"
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "4733"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
  uid: 870148c4-6655-11e9-bb69-000c29180723
## 可看到當前k8s-master01為leader節點