kubernetes部署kube-controller-manager服務

本文檔介紹部署高可用 kube-controller-manager 集群的步驟。

該集群包含 3 個節點，啟動后將通過競爭選舉機制產生一個 leader 節點，其它節點為阻塞狀態。當 leader 節點不可用后，剩余節點將再次進行選舉產生新的 leader 節點，從而保證服務的可用性。

以下是非認證的的配置文件（用127.0.0.1連接kube-apiserver）：

cat > /lib/systemd/system/kube-controller-manager.service <<"EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --address=127.0.0.1 \
  --master=http://127.0.0.1:8080 \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=10.254.0.0/16 \
  --cluster-cidr=172.30.0.0/16 \
  --cluster-name=kubernetes \
  --leader-elect=true \
  --cluster-signing-cert-file=/etc/kubernetes/ca/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ca/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/ca/ca-key.pem \
  --root-ca-file=/etc/kubernetes/ca/ca.pem \
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF

這個簡單很多，但是以后可能會取消，如果要使用認證授權的方式看下面：

------------------------------------------------------------------

為保證通信安全，本文檔先生成 x509 證書和私鑰，kube-controller-manager 在如下兩種情況下使用該證書：

與 kube-apiserver 的安全端口通信時;
在安全端口(https，10252) 輸出 prometheus 格式的 metrics；

創建 kube-controller-manager 證書和私鑰

cat > kube-controller-manager-csr.json <<EOF
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.111.10",
      "192.168.111.11",
      "192.168.111.12"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "ChongQing",
        "L": "ChongQing",
        "O": "system:kube-controller-manager",
        "OU": "yunwei"
      }
    ]
}
EOF


cfssl gencert -ca=/etc/kubernetes/ca/ca.pem \
  -ca-key=/etc/kubernetes/ca/ca-key.pem \
  -config=/etc/kubernetes/ca/ca-config.json \
  -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

創建和分發 kubeconfig 文件

kubeconfig 文件包含訪問 apiserver 的所有信息，如 apiserver 地址、CA 證書和自身使用的證書；

kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ca/ca.pem \
--embed-certs=true \
--server=https://192.168.111.9:8443 \
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
--client-certificate=/etc/kubernetes/ca/kube-controller-manager.pem \
--client-key=/etc/kubernetes/ca/kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

分發 kubeconfig 到所有 master 節點：

# scp kube-controller-manager.kubeconfig 192.168.111.11:/etc/kubernetes/
# scp kube-controller-manager.kubeconfig 192.168.111.12:/etc/kubernetes/

創建和分發 kube-controller-manager systemd unit 文件：

cat > /lib/systemd/system/kube-controller-manager.service <<"EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --port=0 \
  --secure-port=10252 \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.254.0.0/16 \
  --cluster-signing-cert-file=/etc/kubernetes/ca/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ca/ca-key.pem \
  --root-ca-file=/etc/kubernetes/ca/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/ca/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --horizontal-pod-autoscaler-use-rest-clients=true \
  --horizontal-pod-autoscaler-sync-period=10s \
  --tls-cert-file=/etc/kubernetes/ca/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/ca/kube-controller-manager-key.pem \
  --use-service-account-credentials=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on
Restart=on-failure
RestartSec=5
User=k8s

[Install]
WantedBy=multi-user.target
EOF

-------------------------------------------------

啟動kube-controller-manager

# systemctl daemon-reload&&for SERVICES in kube-controller-manager;do systemctl enable $SERVICES; systemctl start $SERVICES; systemctl status $SERVICES; done

查看輸出的 metric
注意：以下命令在 kube-controller-manager 節點上執行。

kube-controller-manager 監聽 10252 端口，接收 https 請求：

# ss -lnpt|grep kube-controll

# curl http://127.0.0.1:10252/metrics |head

測試 kube-controller-manager 集群的高可用
停掉一個或兩個節點的 kube-controller-manager 服務，觀察其它節點的日志，看是否獲取了 leader 權限。
查看當前的 leader
kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml