K8S從入門到放棄系列-(5)kubernetes集群之kube-apiserver部署

摘要：

1、kube-apiserver為是整個k8s集群中的數據總線和數據中心，提供了對集群的增刪改查及watch等HTTP Rest接口

2、kube-apiserver是無狀態的，雖然客戶端如kubelet可通過啟動參數"--api-servers"指定多個api-server，但只有第一個生效，並不能達到高可用的效果，關於kube-apiserver高可用方案，我們在后面介紹，本章，之介紹如何安裝。

創建k8s集群各組件運行用戶

安全性考慮，我們創建單獨的用戶運行k8s中各組件

[root@k8s-master01 ~]# ansible k8s-master -m group -a 'name=kube'
[root@k8s-master01 ~]# ansible k8s-master -m user -a 'name=kube group=kube comment="Kubernetes user" shell=/sbin/nologin createhome=no'

1）創建kube-apiserver證書請求文件

apiserver TLS 認證端口需要的證書

[root@k8s-master01 ~]# vim /opt/k8s/certs/kube-apiserver-csr.json
  {
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "10.10.0.18",
    "10.10.0.19",
    "10.10.0.20",
    "10.254.0.1",
    "localhost",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "ShangHai",
      "L": "ShangHai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

hosts字段列表中，指定了master節點ip，本地ip，10.254.0.1為集群service ip一般為設置的網絡段中第一個ip

2）生成 kubernetes 證書和私鑰

[root@k8s-master01 certs]# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
     -ca-key=/etc/kubernetes/ssl/ca-key.pem \
     -config=/opt/k8s/certs/ca-config.json \
     -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
2019/04/23 16:56:52 [INFO] generate received request
2019/04/23 16:56:52 [INFO] received CSR
2019/04/23 16:56:52 [INFO] generating key: rsa-2048
2019/04/23 16:56:52 [INFO] encoded CSR
2019/04/23 16:56:52 [INFO] signed certificate with serial number 22317568679091080825926949538404731378745389881
2019/04/23 16:56:52 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

3）查看證書生成

[root@k8s-master01 certs]# ll kube-apiserver*
-rw-r--r-- 1 root root 1277 Apr 23 16:56 kube-apiserver.csr
-rw-r--r-- 1 root root  489 Apr 23 16:56 kube-apiserver-csr.json
-rw------- 1 root root 1675 Apr 23 16:56 kube-apiserver-key.pem
-rw-r--r-- 1 root root 1651 Apr 23 16:56 kube-apiserver.pem

4）證書分發

[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-apiserver.pem dest=/etc/kubernetes/ssl'
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/certs/kube-apiserver-key.pem dest=/etc/kubernetes/ssl'

5）配置kube-apiserver客戶端使用的token文件

kubelet 啟動時向 kube-apiserver發送注冊信息，在雙向的TLS加密通信環境中需要認證，手工為kubelet生成證書/私鑰在node節點較少且數量固定時可行，采用TLS Bootstrapping 機制，可使大量的node節點自動完成向kube-apiserver的注冊請求。

原理：kubelet 首次啟動時向 kube-apiserver 發送 TLS Bootstrapping 請求，kube-apiserver 驗證 kubelet 請求中的 token 是否與它配置的 token.csv 一致，如果一致則自動為 kubelet生成證書和秘鑰。

[root@k8s-master01 ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
fb8f04963e38858eab0867e8d2296d6b
[root@k8s-master01 ~]# vim /opt/k8s/cfg/bootstrap-token.csv
fb8f04963e38858eab0867e8d2296d6b,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
##分發token文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/bootstrap-token.csv dest=/etc/kubernetes/config/'

6）生成 apiserver RBAC 審計配置文件

官方說明： https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

[root@k8s-master01 ~]# vim /opt/k8s/cfg/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
##分發審計文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/audit-policy.yaml dest=/etc/kubernetes/config/'

7）編輯kube-apiserver核心文件

apiserver 啟動參數配置文件,注意創建參數中涉及的日志目錄，並授權kube用戶訪問

[root@k8s-master01 ~]# vim /opt/k8s/cfg/kube-apiserver.conf

            ### 
           
# kubernetes system config 
           
# 
           
# The following values are used to configure the kube-apiserver 
           
# 
          

            # The address on the local server to listen to. 
           
KUBE_API_ADDRESS="--advertise-address=10.10.0.18 --bind-address=0.0.0.0" 
          

            # The port on the local server to listen on. 
           
KUBE_API_PORT="--secure-port=6443" 
          

            # Port minions listen on 
           
# KUBELET_PORT="--kubelet-port=10250" 
          

            # Comma separated list of nodes in the etcd cluster 
           
KUBE_ETCD_SERVERS="--etcd-servers=https://10.10.0.18:2379,https://10.10.0.19:2379,https://10.10.0.20:2379" 
          

            # Address range to use for services 
           
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" 
          

            # default admission control policies 
           
KUBE_ADMISSION_CONTROL="--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota" 
          

            # Add your own! 
           
KUBE_API_ARGS=" --allow-privileged=true \ 
           
                --anonymous-auth=false \ 
           
                --alsologtostderr \ 
           
                --apiserver-count=3 \ 
           
                --audit-log-maxage=30 \ 
           
                --audit-log-maxbackup=3 \ 
           
                --audit-log-maxsize=100 \ 
           
                --audit-log-path=/var/log/kube-audit/audit.log \ 
           
                --audit-policy-file=/etc/kubernetes/config/audit-policy.yaml \ 
           
                --authorization-mode=Node,RBAC \ 
           
                --client-ca-file=/etc/kubernetes/ssl/ca.pem \ 
          

                            --token-auth-file=/etc/kubernetes/config/bootstrap-token.csv \ 
           
                --enable-bootstrap-token-auth \ 
           
                --enable-garbage-collector \ 
           
                --enable-logs-handler \ 
           
                --endpoint-reconciler-type=lease \ 
           
                --etcd-cafile=/etc/kubernetes/ssl/ca.pem \ 
           
                --etcd-certfile=/etc/kubernetes/ssl/etcd.pem \ 
           
                --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \ 
           
                --etcd-compaction-interval=0s \ 
           
                --event-ttl=168h0m0s \ 
           
                --kubelet-https=true \ 
           
                --kubelet-certificate-authority=/etc/kubernetes/ssl/ca.pem \ 
           
                --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \ 
           
                --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \ 
           
                --kubelet-timeout=3s \ 
           
                --runtime-config=api/all=true \ 
           
                --service-node-port-range=30000-50000 \ 
           
                --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ 
           
                --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem \ 
           
                --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \ 
           
                --v=2" 
          

##分發參數配置文件，同時把參數中出現的IP修改為對應的本機IP
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/cfg/kube-apiserver.conf dest=/etc/kubernetes/config/'
##創建日志目錄並授權
[root@k8s-master01 ~]# ansible k8s-master -m file -a 'path=/var/log/kube-audit state=directory owner=kube group=kube'

個別參數解釋：

1. KUBE_API_ADDRESS：向集群成員通知apiserver消息的IP地址。這個地址必須能夠被集群中其他成員訪問。如果IP地址為空，將會使用--bind-address，如果未指定--bind-address，將會使用主機的默認接口地址
2. KUBE_API_PORT：用於監聽具有認證授權功能的HTTPS協議的端口。如果為0，則不會監聽HTTPS協議。 （默認值6443)
3. KUBE_ETCD_SERVERS：連接的etcd服務器列表
4. KUBE_ADMISSION_CONTROL：控制資源進入集群的准入控制插件的順序列表
5. apiserver-count：集群中apiserver數量
6. KUBE_SERVICE_ADDRESSES: CIDR IP范圍，用於分配service 集群IP。不能與分配給節點pod的任何IP范圍重疊

kube-apiserver啟動腳本配置文件kube-apiserver.service

[root@k8s-master01 ~]# vim /opt/k8s/unit/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/config/kube-apiserver.conf
User=kube
ExecStart=/usr/local/bin/kube-apiserver \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_ETCD_SERVERS \
        $KUBE_API_ADDRESS \
        $KUBE_API_PORT \
        $KUBELET_PORT \
        $KUBE_ALLOW_PRIV \
        $KUBE_SERVICE_ADDRESSES \
        $KUBE_ADMISSION_CONTROL \
        $KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
## 分發apiserver啟動腳本文件
[root@k8s-master01 ~]# ansible k8s-master -m copy -a 'src=/opt/k8s/unit/kube-apiserver.service dest=/usr/lib/systemd/system/'

8）啟動kube-apiserver 服務

[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl daemon-reload'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl enable kube-apiserver'
[root@k8s-master01 ~]# ansible k8s-master -m shell -a 'systemctl start kube-apiserver'

 9）授予 kubernetes 證書訪問 kubelet API 的權限

[root@k8s-master01 ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

后面部署好集群，在執行 kubectl exec、run、logs 等命令時，apiserver 會轉發到 kubelet。這里定義 RBAC 規則，授權 apiserver 調用 kubelet API，否則會報類似以下錯誤：

Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-8477bdff5d-2lf7k)