kubernetes部署 kube-apiserver 組件
本文檔講解使用 keepalived 和 haproxy 部署一個 3 節點高可用 master 集群的步驟。
kube-apiserver 集群各節點的名稱和 IP 如下：
kube-node0：192.168.111.10
kube-node1：192.168.111.11
kube-node2：192.168.111.12
創建 kubernetes 證書和私鑰
其中會用到上面的三個主機IP，一個vip（192.168.111.9），這些都是kube-apiserver的對外提供服務的IP，還有就是kubernetes本身會創建一個service，它的IP是我們在啟動kube-apiserver是定義的--service-cluster-ip-range 參數指定的IP地址段（10.254.0.0/24,）的第一個IP地址，后續可以通過kubectl get svc kubernetes命令獲取。
cat > kubernetes-csr.json <<EOF { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.111.9", "192.168.111.10", "192.168.111.11", "192.168.111.12", "10.254.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ChongQing", "L": "ChongQing", "O": "k8s", "OU": "yunwei" } ] } EOF
生成認證文件：
cfssl gencert -ca=/etc/kubernetes/ca/ca.pem \ -ca-key=/etc/kubernetes/ca/ca-key.pem \ -config=/etc/kubernetes/ca/ca-config.json \ -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
將生成的證書和私鑰文件拷貝到其他kube-apiserver節點
# scp /etc/kubernetes/ca/kubernetes* 192.168.111.11:/etc/kubernetes/ca/
# scp /etc/kubernetes/ca/kubernetes* 192.168.111.12:/etc/kubernetes/ca/
生成token認證文件
#生成隨機token
# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
8afdf3c4eb7c74018452423c29433609
#按照固定格式寫入token.csv，注意替換token內容
# echo "8afdf3c4eb7c74018452423c29433609,kubelet-bootstrap,10001,\"system:kubelet-bootstrap\"" > /etc/kubernetes/ca/token.csv
kube-apiserver的配置文件，三台一樣（配置文件中將127.0.0.1的非https的api開放，在kube-scheduler服務和kube-controller-manager服務就可以不用認證授權了。）：
cat > /lib/systemd/system/kube-apiserver.service <<EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-apiserver \ --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \ --insecure-bind-address=127.0.0.1 \ --kubelet-https=true \ --bind-address=192.168.111.12 \ --authorization-mode=Node,RBAC \ --runtime-config=api/all \ --enable-bootstrap-token-auth \ --token-auth-file=/etc/kubernetes/ca/token.csv \ --tls-cert-file=/etc/kubernetes/ca/kubernetes.pem \ --tls-private-key-file=/etc/kubernetes/ca/kubernetes-key.pem \ --client-ca-file=/etc/kubernetes/ca/ca.pem \ --service-account-key-file=/etc/kubernetes/ca/ca-key.pem \ --etcd-cafile=/etc/kubernetes/ca/ca.pem \ --etcd-certfile=/etc/kubernetes/ca/kubernetes.pem \ --etcd-keyfile=/etc/kubernetes/ca/kubernetes-key.pem \ --service-cluster-ip-range=10.254.0.0/16 \ --etcd-servers=https://192.168.111.10:2379,https://192.168.111.11:2379,https://192.168.111.12:2379 \ --enable-swagger-ui=true \ --allow-privileged=true \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/var/lib/audit.log \ --v=2 Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
systemctl daemon-reload && for SERVICES in kube-apiserver;do systemctl enable $SERVICES; systemctl restart $SERVICES; systemctl status $SERVICES; done
打印 kube-apiserver 寫入 etcd 的數據
ETCDCTL_API=3 etcdctl \ --endpoints=https://192.168.111.10:2379,https://192.168.111.11:2379,https://192.168.111.12:2379 \ --cacert=/etc/kubernetes/ca/ca.pem \ --cert=/etc/kubernetes/ca/etcd.pem \ --key=/etc/kubernetes/ca/etcd-key.pem \ get /registry/ --prefix --keys-only
部署 kubectl 命令行工具
cat > admin-csr.json <<EOF { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "ChongQing", "L": "ChongQing", "O": "system:masters", "OU": "yunwei" } ] } EOF
生成認證文件：
cfssl gencert -ca=/etc/kubernetes/ca/ca.pem \ -ca-key=/etc/kubernetes/ca/ca-key.pem \ -config=/etc/kubernetes/ca/ca-config.json \ -profile=kubernetes admin-csr.json | cfssljson -bare admin
生成kubectl的config文件（可以三台都執行一遍，也可以一台執行后復制過去）：
kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ca/ca.pem \ --embed-certs=true \ --server=https://192.168.111.9:8443 kubectl config set-credentials admin \ --client-certificate=/etc/kubernetes/ca/admin.pem \ --client-key=/etc/kubernetes/ca/admin-key.pem \ --embed-certs=true kubectl config set-context kubernetes \ --cluster=kubernetes \ --user=admin kubectl config use-context kubernetes
mkdir -p ~/.kube
scp ~/.kube/config 192.168.111.11:~/.kube/config
scp ~/.kube/config 192.168.111.12:~/.kube/config
檢查集群信息(任意一台)
# kubectl cluster-info Kubernetes master is running at https://192.168.111.9:8443 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. # kubectl get all --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 34d # kubectl get componentstatuses NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused etcd-2 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"}
檢查 kube-apiserver 監聽的端口
6443: 接收 https 請求的安全端口，對所有請求做認證和授權
# ss -netstat -lnpt|grep kube LISTEN 0 128 192.168.111.12:6443 *:* users:(("kube-apiserver",pid=878,fd=3)) timer:(keepalive,031ms,0) ino:23491 sk:ffff880078d34d80 <-> LISTEN 0 128 127.0.0.1:8080 *:* users:(("kube-apiserver",pid=4168,fd=68)) ino:35479 sk:ffff88002391ec80 <->