第六章 部署node運算節點服務

一、部署Kubelet

1.1 集群規划

主機名		 角色		IP
hdss7-21	kubelet		10.4.7.21
hdss7-22	kubelet		10.4.7.22

注意：部署以10.4.7.21為例，10.4.7.22節點類似

1.2 簽發kubelet證書

證書簽發需要在10.4.7.200上操作

[root@hdss7-200 ~]# cd /opt/certs/
注意：將所有可能的kubelet服務器的IP都加進去，后期如果需要再加入其他IP節點的話就需要重新簽發此證書，有計划的將證書替換成最新的，最好避免后期加入新的節點。
[root@hdss7-200 certs]# vim kubelet-csr.json
{
    "CN": "k8s-kubelet",
    "hosts": [
    "127.0.0.1",
    "10.4.7.10",
    "10.4.7.21",
    "10.4.7.22",
    "10.4.7.23",
    "10.4.7.24",
    "10.4.7.25",
    "10.4.7.26",
    "10.4.7.27",
    "10.4.7.28"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}
生成證書
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet
注意私鑰文件的屬性權限是600
certs]# ll kubelet*
-rw-r--r-- 1 root root 1115 6月  10 00:04 kubelet.csr
-rw-r--r-- 1 root root  452 6月  10 00:03 kubelet-csr.json
-rw------- 1 root root 1675 6月  10 00:04 kubelet-key.pem
-rw-r--r-- 1 root root 1468 6月  10 00:04 kubelet.pem
分發證書
[root@hdss7-200 certs]# scp kubelet.pem kubelet-key.pem hdss7-21:/opt/kubernetes/server/bin/certs/
[root@hdss7-200 certs]# scp kubelet.pem kubelet-key.pem hdss7-22:/opt/kubernetes/server/bin/certs/

1.3 創建kubelet的配置

在10.4.7.21,22服務器上操作
1.3.1 set-cluster：創建需要連接的集群信息，可以創建多個k8s信息（會將ca.pem證書編碼后嵌入到/opt/kubernetes/conf/kubelet.kubeconfig配置文件中）
注意：10.4.7.10是apiserver的VIP，之前我們在10.4.7.11/21上部署的nginx就是代理10.4.7.21/22的apiserver集群，部署的keepalived的VIP就是10.4.7.10

[root@hdss7-21 ~]# cd /opt/kubernetes/
[root@hdss7-21 conf]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Cluster "myk8s" set.
[root@hdss7-21 conf]# ll /opt/kubernetes/conf/
總用量 8
-rw-r--r-- 1 root root 2223 6月   8 22:00 audit.yaml
-rw------- 1 root root 1986 6月  10 00:14 kubelet.kubeconfig

1.3.2 set-credentials：創建用戶賬號，即用戶登錄的客戶端私有證書，可以創建多個證書（將client.pem證書和client-key.pem私鑰編碼后嵌入到kubelet.kubeconfig文件中）

[root@hdss7-21 conf]# kubectl config set-credentials k8s-node \
--client-certificate=/opt/kubernetes/server/bin/certs/client.pem \
--client-key=/opt/kubernetes/server/bin/certs/client-key.pem \
--embed-certs=true \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
User "k8s-node" set.

1.3.3 set-context：設置context，即確定賬號和集群對應關系

[root@hdss7-21 conf]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=k8s-node \
--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Context "myk8s-context" created.

1.3.4 use-context：設置當前使用哪個context

[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Switched to context "myk8s-context".
把此配置傳送給10.4.7.22，那么在22上就不需要重復操作以上4個步驟了
[root@hdss7-21 conf]# scp /opt/kubernetes/conf/kubelet.kubeconfig hdss7-22:/opt/kubernetes/conf/

1.4 授權k8s-node用戶

此步驟只需要在一台master節點上操作就行（10.4.7.21）
授權k8s-node用戶綁定集群角色system:node，讓k8s-node擁有具備運算節點的權限

[root@hdss7-21 conf]# vim /opt/kubernetes/conf/k8s-node.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: k8s-node
[root@hdss7-21 conf]# kubectl create -f /opt/kubernetes/conf/k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io/k8s-node created
創建資源（會存儲到etcd中）
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node
NAME       AGE
k8s-node   51s
注意：查看7443端口是否正常啟動，非常重要，7443端口無法連接會導致node節點無法加入到master節點
~]# telnet 10.4.7.10 7443
Trying 10.4.7.10...
Connected to 10.4.7.10.
Escape character is '^]'.
^]
telnet> q

刪除資源命令如下

[root@hdss7-21 conf]# kubectl delete -f k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io "k8s-node" deleted

查看資源配置

[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2021-06-10T13:51:06Z"
  name: k8s-node
  resourceVersion: "12725"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/k8s-node
  uid: e70f91af-c9f2-11eb-aaf3-000c29e396b1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: k8s-node

1.5 准備pause基礎鏡像

因為kubelet在啟動容器時需要有一個基礎鏡像初始化網絡空間等來幫助我們去啟動容器，從而讓我們能夠啟動pod；
將pause鏡像放入到harbor私有倉庫中，僅在10.4.7.200上操作，確保harbor和docker運行正常，提前檢查

下載鏡像
[root@hdss7-200 ~]# docker image pull kubernetes/pause
打標簽
[root@hdss7-200 ~]# docker image tag kubernetes/pause:latest harbor.od.com/public/pause:latest
登錄harbor
[root@hdss7-200 ~]# docker login -u admin harbor.od.com
推送pause鏡像到harbor私有倉庫
[root@hdss7-200 ~]# docker image push harbor.od.com/public/pause:latest

1.6 創建kubelet啟動腳本

在node節點創建啟動腳本，並啟動kubelet，在10.4.7.21/22上操作，以21為例
22上修改--hostname-override項

[root@hdss7-21 ~]# vim /opt/kubernetes/server/bin/kubelet-startup.sh
#!/bin/sh

WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/kubernetes/server/bin/kubelet \
    --anonymous-auth=false \
    --cgroup-driver systemd \
    --cluster-dns 192.168.0.2 \
    --cluster-domain cluster.local \
    --runtime-cgroups=/systemd/system.slice \
    --kubelet-cgroups=/systemd/system.slice \
    --fail-swap-on="false" \
    --client-ca-file ./certs/ca.pem \
    --tls-cert-file ./certs/kubelet.pem \
    --tls-private-key-file ./certs/kubelet-key.pem \
    --hostname-override hdss7-21.host.com \
    --image-gc-high-threshold 20 \
    --image-gc-low-threshold 10 \
    --kubeconfig ../../conf/kubelet.kubeconfig \
    --log-dir /data/logs/kubernetes/kube-kubelet \
    --pod-infra-container-image harbor.od.com/public/pause:latest \
    --root-dir /data/kubelet

1.7 添加權限，創建目錄

[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kubelet-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet

1.8 配置supervisor配置

[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-kubelet.ini
[program:kube-kubelet-7-21]
command=/opt/kubernetes/server/bin/kubelet-startup.sh
numprocs=1
directory=/opt/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false

1.9 啟動服務並檢查

[root@hdss7-21 ~]# supervisorctl update
kube-kubelet-7-21: added process group
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 1172, uptime 1:06:46
kube-apiserver-7-21              RUNNING   pid 1183, uptime 1:06:46
kube-controller-manager-7-21     RUNNING   pid 1167, uptime 1:06:46
kube-kubelet-7-21                RUNNING   pid 2280, uptime 0:01:44
kube-scheduler-7-21              RUNNING   pid 1169, uptime 1:06:46
[root@hdss7-22 ~]# tail -100f /data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
I0713 21:44:08.453953    2265 kubelet_node_status.go:75] Successfully registered node hdss7-21.host.com
I0713 21:44:08.509328    2265 cpu_manager.go:155] [cpumanager] starting with none policy
I0713 21:44:08.509382    2265 cpu_manager.go:156] [cpumanager] reconciling every 10s
I0713 21:44:08.509441    2265 policy_none.go:42] [cpumanager] none policy: Start
W0713 21:44:08.644794    2265 manager.go:540] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0713 21:44:08.878478    2265 reconciler.go:154] Reconciler: start to sync state
出現如上表示正常啟動
查看node節點是否加入到集群中
[root@hdss7-21 ~]#  kubectl get node
NAME                STATUS     ROLES    AGE   VERSION
hdss7-21.host.com   Ready      <none>   13s   v1.14.10
hdss7-22.host.com   NotReady   <none>   0s    v1.14.10
別急，需要加載一會
[root@hdss7-21 ~]#  kubectl get node
NAME                STATUS   ROLES    AGE   VERSION
hdss7-21.host.com   Ready    <none>   55m   v1.14.10
hdss7-22.host.com   Ready    <none>   54m   v1.14.10

1.10 修改節點角色

[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/node=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/master=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/master=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/node=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]#  kubectl get node
NAME                STATUS   ROLES         AGE   VERSION
hdss7-21.host.com   Ready    master,node   57m   v1.14.10
hdss7-22.host.com   Ready    master,node   57m   v1.14.10

1.11 安裝部署其他節點

在10.4.7.22上同樣操作

1.12 報錯排查

在10.4.7.21（10.4.7.10）master節點上查看node，發現無任何資源可訪問，如下

[root@hdss7-21 ~]#  kubectl get node
No resources found.

查看kubectl日志

[root@hdss7-21 ~]# tail -100f /data/logs/kubernetes/kube-kubelet/kubelet.stdout.log

第一種如下：

failed to ensure node lease exists connect: no route to host
原因：這是根本就沒有10.4.7.10這個ip，或者無法連接到此ip，ping一下，檢查是否可以連接，添加此虛IP重新，重新執行本章1.2步驟即可

第二種如下：

E0611 20:41:55.908234    1414 kubelet.go:2246] node "hdss7-22.host.com" not found
E0611 20:41:56.008667    1414 kubelet.go:2246] node "hdss7-22.host.com" not found
這個報錯可以忽略

第三種如下：

E0611 20:41:55.838167    1414 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://10.4.7.10:7443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 10.4.7.10:7443: connect: connection refused
這就說明雖然找到了這個服務器，但是拒絕連接，測試一下端口的連通性，發現被拒絕，無此端口
[root@hdss7-21 ~]# telnet 10.4.7.10 7443
Trying 10.4.7.10...
telnet: connect to address 10.4.7.10: Connection refused
檢查虛IP是否正確配置，檢查1.2步驟是否正確執行，在本地telnet一下，是否正確啟動7443端口，正常啟動后重啟一下kube-kubelet-7-21，kube-kubelet-7-22服務即可
正確的日志如下：
I0611 21:06:18.917499    9153 kubelet_node_status.go:72] Attempting to register node hdss7-22.host.com
I0611 21:06:18.947122    9153 kubelet_node_status.go:75] Successfully registered node hdss7-22.host.com
I0611 21:06:18.989477    9153 kubelet.go:1825] skipping pod synchronization - container runtime status check may not have completed yet.
I0611 21:06:19.015529    9153 cpu_manager.go:155] [cpumanager] starting with none policy
I0611 21:06:19.015565    9153 cpu_manager.go:156] [cpumanager] reconciling every 10s

二、部署kube-proxy

Kube-proxy實際上是維護了pod網絡、節點（node）網絡與cluster（service）網絡三者之間的關系

2.1 集群規划

注意：部署以10.4.7.21為例，22節點部署類似

主機名			角色			IP
hdss7-21		kube-proxy		10.4.7.21
hdss7-22		kube-proxy		10.4.7.22

2.2 簽發kube-proxy證書

在10.4.7.200證書簽發服務器上操作

創建簽發證書的請求文件
[root@hdss7-200 ~]# cd /opt/certs/
[root@hdss7-200 certs]# vim kube-proxy-csr.json
{
    "CN": "system:kube-proxy",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "od",
            "OU": "ops"
        }
    ]
}
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client kube-proxy-csr.json |cfssl-json -bare kube-proxy-client
certs]# ll  kube-proxy*
-rw-r--r-- 1 root root 1005 6月  12 20:49 kube-proxy-client.csr
-rw------- 1 root root 1679 6月  12 20:49 kube-proxy-client-key.pem
-rw-r--r-- 1 root root 1375 6月  12 20:49 kube-proxy-client.pem
-rw-r--r-- 1 root root  267 6月  12 20:49 kube-proxy-csr.json
分發證書
[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-21:/opt/kubernetes/server/bin/certs/
[root@hdss7-200 certs]# scp kube-proxy-client-key.pem kube-proxy-client.pem hdss7-22:/opt/kubernetes/server/bin/certs/

2.3 創建kube-proxy配置

在所有node節點部署，涉及服務器10.4.7.21，22，步驟與創建kubelet相同

[root@hdss7-21 ~]# kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
Cluster "myk8s" set.
[root@hdss7-21 ~]# kubectl config set-credentials kube-proxy \
--client-certificate=/opt/kubernetes/server/bin/certs/kube-proxy-client.pem \
--client-key=/opt/kubernetes/server/bin/certs/kube-proxy-client-key.pem \
--embed-certs=true \
--kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
User "kube-proxy" set.
[root@hdss7-21 ~]# kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=kube-proxy \
--kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
Context "myk8s-context" created.
[root@hdss7-21 ~]# kubectl config use-context myk8s-context --kubeconfig=/opt/kubernetes/conf/kube-proxy.kubeconfig
Switched to context "myk8s-context".
傳送配置，22上就不需要操作了
[root@hdss7-21 ~]# scp /opt/kubernetes/conf/kube-proxy.kubeconfig hdss7-22:/opt/kubernetes/conf/

2.4 加載IPvs模塊

kube-proxy共有三種流量調度模式，分別是userspace，iptables和ipvs，目前ipvs是最匹配的
在21和22上操作

查看現有的ipvs模塊
[root@hdss7-21 ~]# lsmod | grep ip_vs
加載ipvs模塊
[root@hdss7-21 ~]# for i in $(ls /usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*");do echo $i; /sbin/modinfo -F filename $i >/dev/null 2>&1 && /sbin/modprobe $i;done
ip_vs_dh
ip_vs_ftp
ip_vs
ip_vs_lblc
ip_vs_lblcr
ip_vs_lc
ip_vs_nq
ip_vs_pe_sip
ip_vs_rr
ip_vs_sed
ip_vs_sh
ip_vs_wlc
ip_vs_wrr
查看
[root@hdss7-21 ~]# lsmod | grep ip_vs
ip_vs_wrr              12697  0
ip_vs_wlc              12519  0
ip_vs_sh               12688  0
ip_vs_sed              12519  0
ip_vs_rr               12600  0
ip_vs_pe_sip           12740  0
nf_conntrack_sip       33780  1 ip_vs_pe_sip
ip_vs_nq               12516  0
ip_vs_lc               12516  0
ip_vs_lblcr            12922  0
ip_vs_lblc             12819  0
ip_vs_ftp              13079  0
ip_vs_dh               12688  0
nf_nat                 26583  3 ip_vs_ftp,nf_nat_ipv4,nf_nat_masquerade_ipv4
ip_vs                 145497  24 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_pe_sip,ip_vs_lblcr,ip_vs_lblc
nf_conntrack          139224  8 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_sip,nf_conntrack_ipv4
libcrc32c              12644  3 ip_vs,nf_nat,nf_conntrack

2.5 創建kube-proxy啟動腳本

此操作需要在21和22上進行操作
--hostname-override需要修改為本機的主機名

[root@hdss7-21 ~]# vim /opt/kubernetes/server/bin/kube-proxy-startup.sh
#!/bin/sh

WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit

/opt/kubernetes/server/bin/kube-proxy \
  --cluster-cidr 172.7.0.0/16 \
  --hostname-override hdss7-21.host.com \
  --proxy-mode=ipvs \
  --ipvs-scheduler=nq \
  --kubeconfig ../../conf/kube-proxy.kubeconfig

2.6 設置權限，創建目錄

[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kube-proxy-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-proxy

2.7 創建supervisor配置

[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-proxy.ini
[program:kube-proxy-7-21]
command=/opt/kubernetes/server/bin/kube-proxy-startup.sh
numprocs=1
directory=/opt/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false

2.8 啟動服務並檢查

[root@hdss7-21 ~]# supervisorctl update
kube-proxy-7-21: added process group
查看代理狀態
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 1319, uptime 0:48:02
kube-apiserver-7-21              RUNNING   pid 1328, uptime 0:48:02
kube-controller-manager-7-21     RUNNING   pid 1305, uptime 0:48:02
kube-kubelet-7-21                RUNNING   pid 1308, uptime 0:48:02
kube-proxy-7-21                  RUNNING   pid 11663, uptime 0:02:10
kube-scheduler-7-21              RUNNING   pid 1316, uptime 0:48:02
[root@hdss7-21 ~]# yum -y install ipvsadm
查看ipvs代理信息，可以看到192.168.0.1:443端口代理了10.4.7.21/22:6443端口
[root@hdss7-21 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.1:443 nq
  -> 10.4.7.21:6443               Masq    1      0          0
  -> 10.4.7.22:6443               Masq    1      0          0
查看service的信息
[root@hdss7-21 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   192.168.0.1   <none>        443/TCP   3d23h

2.9 安裝部署集群其他節點

部署集群其他節點