前言
基於前面的所有文章完成后，這里介紹一下如何添加一個全新的節點；
對新節點做解析
方法一 hosts 文件解析
如果使用的是 hosts 文件解析，則需要在所有節點的 hsots 文件添加下面內容：
10.0.20.15 node05 node05.k8s.com
方法二 bind 解析
如果使用的時候內部bind 做DNS解析，則只需要添加一條A記錄即可
node05 IN A 10.0.20.15
重載bind配置文件
rndc reload
測試
在 node01 機器上測試
[root@node01 work]# ping -c 1 node05
PING node05.k8s.com (10.0.20.15) 56(84) bytes of data.
64 bytes from 10.0.20.15 (10.0.20.15): icmp_seq=1 ttl=64 time=0.122 ms
--- node05.k8s.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.122/0.122/0.122/0.000 ms
[root@node01 work]# ping -c 1 node05.k8s.com
PING node05.k8s.com (10.0.20.15) 56(84) bytes of data.
64 bytes from 10.0.20.15 (10.0.20.15): icmp_seq=1 ttl=64 time=0.121 ms
--- node05.k8s.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.121/0.121/0.121/0.000 ms
分發密鑰對
為了方便推送文件，這一步可做可不做
ssh-copy-id -i ~/.ssh/id_rsa.pub node05
推送 CA 證書
ssh node05 "mkdir -p /etc/kubernetes/cert"
scp ca*.pem ca-config.json node05:/etc/kubernetes/cert
flanneld 部署
下面所有步驟都在 node01 上執行
推送flanneld二進制命令
scp flannel/{flanneld,mk-docker-opts.sh} node05:/opt/k8s/bin/
推送flanneld秘鑰
ssh node05 "mkdir -p /etc/flanneld/cert"
scp flanneld*.pem node05:/etc/flanneld/cert
推送flanneld啟動文件
scp flanneld.service node05:/etc/systemd/system/
啟動flanneld
ssh node05 "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"
ssh node05 "systemctl status flanneld|grep Active"
查看flanneld網絡
ssh node05 "/usr/sbin/ip addr show flannel.1|grep -w inet"
查看etcd中網絡數據
查看一下etcd中CIDR分配的數量是否添加
source /opt/k8s/bin/environment.sh
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.80.0-21
查看輸出結果：
[root@node01 work]# etcdctl \
> --endpoints=${ETCD_ENDPOINTS} \
> --ca-file=/etc/kubernetes/cert/ca.pem \
> --cert-file=/etc/flanneld/cert/flanneld.pem \
> --key-file=/etc/flanneld/cert/flanneld-key.pem \
> ls ${FLANNEL_ETCD_PREFIX}/subnets
/kubernetes/network/subnets/172.30.80.0-21
/kubernetes/network/subnets/172.30.48.0-21
/kubernetes/network/subnets/172.30.216.0-21
/kubernetes/network/subnets/172.30.224.0-21
/kubernetes/network/subnets/172.30.160.0-21
從上面輸出看出，新節點的flanneld已經正常
docker 安裝配置
這里直接在 node05 節點上操作
安裝
yum install docker-ce-18.09.6 -y
創建配置文件
mkdir -p /etc/docker/
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://hjvrgh7a.mirror.aliyuncs.com"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
修改Docker啟動參數
vim /usr/lib/systemd/system/docker.service
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock
啟動查看
[root@localhost ~]# vim /usr/lib/systemd/system/docker.service
[root@localhost ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@localhost ~]# systemctl start docker
查看docker0網橋是否是 flanneld 網絡的網關
[root@localhost ~]# ip addr show flannel.1 && /usr/sbin/ip addr show docker0
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether fe:64:95:d1:b5:7c brd ff:ff:ff:ff:ff:ff
inet 172.30.216.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:79:3b:10:f3 brd ff:ff:ff:ff:ff:ff
inet 172.30.216.1/21 brd 172.30.223.255 scope global docker0
valid_lft forever preferred_lft forever
kubelet 安裝
這里在 node01 上操作
推送 kubelet 二進制文件
scp kubernetes/server/bin/kubelet node05:/opt/k8s/bin/
創建 kubelet bootstrap kubeconfig 文件
export BOOTSTRAP_TOKEN=$(kubeadm token create \
--description kubelet-bootstrap-token \
--groups system:bootstrappers:node05 \
--kubeconfig ~/.kube/config)
# 設置集群參數
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/cert/ca.pem \
--embed-certs=true \
--server=https://vip.k8s.com:8443 \
--kubeconfig=kubelet-bootstrap-node05.kubeconfig
# 設置客戶端認證參數
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=kubelet-bootstrap-node05.kubeconfig
# 設置上下文參數
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=kubelet-bootstrap-node05.kubeconfig
# 設置默認上下文
kubectl config use-context default --kubeconfig=kubelet-bootstrap-node05.kubeconfig
分發 bootstrap kubeconfig 文件到所有 worker 節點
scp kubelet-bootstrap-node05.kubeconfig root@node05:/etc/kubernetes/kubelet-bootstrap.kubeconfig
查看kubeadm為新節點創建的token
[root@node01 work]# kubeadm token list --kubeconfig ~/.kube/config
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
cu4q2e.ogvim78s3p252ysg 7h 2019-12-06T17:44:24+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:node03
nrypmb.35fyygbwr9failr5 7h 2019-12-06T17:44:23+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:node01
r5luwb.6x6c5lnit5utyotz 7h 2019-12-06T17:44:23+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:node02
# 下面多了一條node05
ss66d3.yse8ia5bt1s06jmg 23h 2019-12-07T10:01:58+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:node05
sx8n4m.vlltkkv8m23ogxg9 7h 2019-12-06T17:44:24+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:node04
查看 token 關聯的 Secret
[root@node01 work]# kubectl get secrets -n kube-system|grep bootstrap-token
bootstrap-token-cu4q2e bootstrap.kubernetes.io/token 7 16h
bootstrap-token-nrypmb bootstrap.kubernetes.io/token 7 16h
bootstrap-token-r5luwb bootstrap.kubernetes.io/token 7 16h
# 根據上面查看的token，下面這條是新添加的
bootstrap-token-ss66d3 bootstrap.kubernetes.io/token 7 99s
bootstrap-token-sx8n4m bootstrap.kubernetes.io/token 7 16h
創建和分發kubelet參數配置
cd /opt/k8s/work
sed -e "s/##NODE_IP##/10.0.20.15/" kubelet-config.yaml.template > kubelet-config-10.0.20.15.yaml.template
scp kubelet-config-10.0.20.15.yaml.template root@node05:/etc/kubernetes/kubelet-config.yaml
創建和分發kubelet啟動文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
sed -e "s/##NODE_NAME##/node05/" kubelet.service.template > kubelet-node05.service
scp kubelet-node05.service root@node05:/etc/systemd/system/kubelet.service
啟動 kubelet 服務
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
ssh root@node05 "mkdir -p ${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/"
ssh root@node05 "/usr/sbin/swapoff -a"
ssh root@node05 "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"
手動approve server cert csr
稍等片刻后，需要手動通過證書請求
基於安全考慮，CSR approving controllers不會自動approve kubelet server證書簽名請求，需要手動approve
kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
查看新節點
此時再次使用 查看節點的命令時，發現已經添加成功
[root@node01 work]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node01 Ready <none> 16h v1.15.6
node02 Ready <none> 16h v1.15.6
node03 Ready <none> 16h v1.15.6
node04 Ready <none> 16h v1.15.6
node05 Ready <none> 74s v1.15.6 # 這里顯示正常
kube-proxy 安裝
此處均在 node01 上執行
推送 kube-proxy 二進制啟動文件
cd /opt/k8s/work/
scp kubernetes/server/bin/kube-proxy node05:/opt/k8s/bin/
分發 kubeconfig 文件
cd /opt/k8s/work/
scp kube-proxy.kubeconfig root@node05:/etc/kubernetes/
分發和創建kube-proxy配置文件
cd /opt/k8s/work/
sed -e "s/##NODE_NAME##/node05/" -e "s/##NODE_IP##/10.0.20.15/" kube-proxy-config.yaml.template > kube-proxy-config-node05.yaml.template
scp kube-proxy-config-node05.yaml.template root@node05:/etc/kubernetes/kube-proxy-config.yaml
分發 kube-proxy systemd unit 文件
scp kube-proxy.service root@node05:/etc/systemd/system/
啟動 kube-proxy 服務
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
ssh root@node05 "mkdir -p ${K8S_DIR}/kube-proxy"
ssh root@node05 "modprobe ip_vs_rr"
ssh root@node05 "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy"
檢查啟動結果
ssh root@node05 "systemctl status kube-proxy|grep Active"
ssh root@node05 "netstat -lnpt|grep kube-prox"
查看ipvs路由規則
ssh root@node05 "/usr/sbin/ipvsadm -ln"
輸出：
[root@node01 work]# ssh root@node05 "/usr/sbin/ipvsadm -ln"
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.254.0.1:443 rr
-> 10.0.20.11:6443 Masq 1 0 0
-> 10.0.20.12:6443 Masq 1 0 0
-> 10.0.20.13:6443 Masq 1 0 0
TCP 10.254.0.2:53 rr
-> 172.30.48.2:53 Masq 1 0 0
-> 172.30.160.2:53 Masq 1 0 0
TCP 10.254.0.2:9153 rr
-> 172.30.48.2:9153 Masq 1 0 0
-> 172.30.160.2:9153 Masq 1 0 0
UDP 10.254.0.2:53 rr
-> 172.30.48.2:53 Masq 1 0 0
-> 172.30.160.2:53 Masq 1 0 0
至此，全新的node節點加入集群操作完成。
查看當前集群 nodes
[root@node01 work]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node01 Ready <none> 16h v1.15.6 10.0.20.11 <none> CentOS Linux 7 (Core) 5.4.1-1.el7.elrepo.x86_64 docker://18.9.6
node02 Ready <none> 16h v1.15.6 10.0.20.12 <none> CentOS Linux 7 (Core) 5.4.1-1.el7.elrepo.x86_64 docker://18.9.6
node03 Ready <none> 16h v1.15.6 10.0.20.13 <none> CentOS Linux 7 (Core) 5.4.1-1.el7.elrepo.x86_64 docker://18.9.6
node04 Ready <none> 16h v1.15.6 10.0.20.14 <none> CentOS Linux 7 (Core) 5.4.1-1.el7.elrepo.x86_64 docker://18.9.6
node05 Ready <none> 12m v1.15.6 10.0.20.15 <none> CentOS Linux 7 (Core) 5.4.1-1.el7.elrepo.x86_64 docker://18.9.6