K8S部署筆記
一、集群環境說明
| 主機名 | IP地址 | 說明 |
|---|---|---|
| k8s-master01 | 192.168.1.107 | master節點 |
| k8s-master02 | 192.168.1.108 | master節點 |
| k8s-master03 | 192.168.1.109 | master節點 |
| k8s-master-lb（在master節點） | 192.168.1.236 | keepalived虛擬IP |
| k8s-node01 | 192.168.1.110 | worker節點 |
| k8s-node02 | 192.168.1.111 | worker節點 |
| 配置信息 | 備注 |
|---|---|
| 系統版本 | CentOS 7.9 |
| Docker版本 | 19.03.x |
| Pod網段 | 172.168.0.0/12 |
| Service網段 | 10.96.0.0/12 |
注意：
VIP（虛擬IP）不要和公司內網IP重復，首先去ping一下，不通才可用。VIP需要和主機在同一個局域網內！
二、基礎環境配置（以下操作所有節點都得執行）
2.1、配置hosts解析
[root@k8s-master01 ~]# cat /etc/hosts
192.168.1.107 k8s-master01
192.168.1.108 k8s-master02
192.168.1.109 k8s-master03
192.168.1.236 k8s-master-lb # 如果不是高可用集群，該IP為Master01的IP
192.168.1.110 k8s-node01
192.168.1.111 k8s-node02
2.2、更換yum源碼
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
2.3、安裝常用工具
yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y
2.4、關閉防火牆、selinux、dnsmasq、swap
systemctl disable --now firewalld
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
關閉swap分區
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
2.5、時間同步配置
# 安裝ntpdate
rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y
# 更改時區
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
# 設置定時任務同步時間
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com
# 加入到crontab
*/5 * * * * ntpdate time2.aliyun.com
2.6、優化Linux
ulimit -SHn 65535
vim /etc/security/limits.conf
# 末尾添加如下內容
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
# 然后重啟Linux
reboot
2.7、所有節點升級系統並重啟，此處升級沒有升級內核，下節會單獨升級內核：
# CentOS7需要升級，CentOS8可以按需升級系統
yum update -y --exclude=kernel* && reboot
三、內核升級
3.1、配置免密登錄（Master01上）
Master01節點免密鑰登錄其他節點，安裝過程中生成配置文件和證書均在Master01上操作，集群管理也在Master01上操作，阿里雲或者AWS上需要單獨一台kubectl服務器。密鑰配置如下：
# 一直回車就行
ssh-keygen -t rsa
for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
3.2、下載安裝所有的源碼文件：（Master01上）
cd /root/ ; git clone https://github.com/dotbalo/k8s-ha-install.git
3.3、下載升級所需安裝包（Master01上）
CentOS7 需要升級內核至4.18+，本地升級的版本為4.19
# 在master01節點下載內核
cd /root
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
# 從master01節點傳到其他節點：
for i in k8s-master02 k8s-master03 k8s-node01 k8s-node02;do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/ ; done
3.4、內核升級（所有節點）
cd /root && yum localinstall -y kernel-ml*
grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
# 檢查默認內核是不是4.19
grubby --default-kernel /boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64
# 所有節點重啟，然后檢查內核是不是4.19
reboot
[root@k8s-node02 ~]# uname -a
Linux k8s-node02 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
3.5、安裝ipvsadm（所有節點）
yum install ipvsadm ipset sysstat conntrack libseccomp -y
所有節點配置ipvs模塊，在內核4.19+版本nf_conntrack_ipv4已經改為nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可：
vim /etc/modules-load.d/ipvs.conf
# 加入以下內容
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack # 4.18 改成這個nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
# 然后執行
systemctl enable --now systemd-modules-load.service
3.6、開啟一些k8s集群中必須的內核參數，配置k8s內核（所有節點）
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
# 所有節點配置完內核后，重啟服務器，保證重啟后內核依舊加載
reboot
[root@k8s-master01 ~]# lsmod | grep --color=auto -e ip_vs -e nf_conntrack
ip_vs_ftp 16384 0
nf_nat 32768 1 ip_vs_ftp
ip_vs_sed 16384 0
ip_vs_nq 16384 0
ip_vs_fo 16384 0
ip_vs_sh 16384 0
ip_vs_dh 16384 0
ip_vs_lblcr 16384 0
ip_vs_lblc 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs_wlc 16384 0
ip_vs_lc 16384 0
ip_vs 151552 24 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_conntrack 143360 2 nf_nat,ip_vs
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 4 nf_conntrack,nf_nat,xfs,ip_vs
四、基本組件安裝
本節主要安裝的是集群中用到的各種組件，比如Docker-ce、Kubernetes各組件等。
4.1、安裝Docker-ce 19.03（所有節點）
yum install docker-ce-19.03.* -y
4.1.1溫馨提示：
由於新版kubelet建議使用systemd，所以可以把docker的CgroupDriver改成systemd
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
4.1.2、所有節點設置開機自啟動Docker
systemctl daemon-reload && systemctl enable --now docker
4.2、安裝k8s組件（所有節點）
所有節點安裝最新版本kubeadm：
yum list kubeadm.x86_64 --showduplicates | sort -r
yum install kubeadm -y
默認配置的pause鏡像使用gcr.io倉庫，國內可能無法訪問，所以這里配置Kubelet使用阿里雲的pause鏡像：
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
EOF
設置Kubelet開機自啟動：
systemctl daemon-reload
systemctl enable --now kubelet
五、高可用組件安裝（所有Master節點）
（注意：如果不是高可用集群，haproxy和keepalived無需安裝）
5.1、通過yum安裝HAProxy和KeepAlived
yum install keepalived haproxy -y
5.2、配置HAProxy
所有Master節點配置HAProxy（詳細配置參考HAProxy文檔，所有Master節點的HAProxy配置相同）：
mkdir /etc/haproxy
[root@k8s-master01 ~]# cat /etc/haproxy/haproxy.cfg
global
maxconn 2000
ulimit-n 16384
log 127.0.0.1 local0 err
stats timeout 30s
defaults
log global
mode http
option httplog
timeout connect 5000
timeout client 50000
timeout server 50000
timeout http-request 15s
timeout http-keep-alive 15s
frontend monitor-in
bind *:33305
mode http
option httplog
monitor-uri /monitor
frontend k8s-master
bind 0.0.0.0:16443
bind 127.0.0.1:16443
mode tcp
option tcplog
tcp-request inspect-delay 5s
default_backend k8s-master
backend k8s-master
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server k8s-master01 192.168.1.107:6443 check
server k8s-master02 192.168.1.108:6443 check
server k8s-master03 192.168.1.109:6443 check
5.3、配置KeepAlived
注意每個節點的IP和網卡（interface參數）
Master01節點的配置：
mkdir /etc/keepalived
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
script_user root
enable_script_security
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state MASTER
interface ens33
mcast_src_ip 192.168.1.107
virtual_router_id 51
priority 101
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.1.236
}
track_script {
chk_apiserver
}
}
Master02節點的配置：
mkdir /etc/keepalived
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
script_user root
enable_script_security
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
mcast_src_ip 192.168.1.108
virtual_router_id 51
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.1.236
}
track_script {
chk_apiserver
}
}
Master03節點的配置：
mkdir /etc/keepalived
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
script_user root
enable_script_security
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
mcast_src_ip 192.168.1.109
virtual_router_id 51
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.1.236
}
track_script {
chk_apiserver
}
}
5.4、所有master節點配置KeepAlived健康檢查文件：
cat > /etc/keepalived/check_apiserver.sh << EFO
#!/bin/bash
err=0
for k in $(seq 1 3)
do
check_code=$(pgrep haproxy)
if [[ $check_code == "" ]]; then
err=$(expr $err + 1)
sleep 1
continue
else
err=0
break
fi
done
if [[ $err != "0" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi
EFO
5.4.1、腳本授權
chmod +x /etc/keepalived/check_apiserver.sh
5.4.2、啟動haproxy和keepalived
systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived
5.5、如果安裝了keepalived和haproxy，需要測試keepalived是否是正常的
# 測試VIP（任意節點上）
[root@k8s-node02 ~]# ping 192.168.1.236 -c 4
PING 192.168.1.236 (192.168.1.236) 56(84) bytes of data.
64 bytes from 192.168.1.236: icmp_seq=1 ttl=64 time=0.923 ms
64 bytes from 192.168.1.236: icmp_seq=2 ttl=64 time=0.433 ms
64 bytes from 192.168.1.236: icmp_seq=3 ttl=64 time=0.554 ms
64 bytes from 192.168.1.236: icmp_seq=4 ttl=64 time=0.457 ms
--- 192.168.1.236 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3061ms
rtt min/avg/max/mdev = 0.433/0.591/0.923/0.198 ms
# 測試haproxy（任意節點上）
[root@k8s-node01 ~]# telnet 192.168.1.236 16443
Trying 192.168.1.236...
Connected to 192.168.1.236.
Escape character is '^]'.
Connection closed by foreign host.
# 排查思路
如果ping不通且telnet沒有出現 ] ，則認為VIP不可以，不可在繼續往下執行，需要排查keepalived的問題，比如防火牆和selinux，haproxy和keepalived的狀態，監聽端口等
所有節點查看防火牆狀態必須為disable和inactive：systemctl status firewalld
所有節點查看selinux狀態，必須為disable：getenforce
master節點查看haproxy和keepalived狀態：systemctl status keepalived haproxy
master節點查看監聽端口：netstat -lntp
六、 集群初始化
官方初始化文檔：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
Master01：（# 注意，如果不是高可用集群，192.168.1.236:16443改為master01的地址，16443改為apiserver的端口，默認是6443，注意更改v1.18.5自己服務器kubeadm的版本：kubernetesVersion）
6.1、Master01節點創建kubeadm-config.yaml配置（Master01上）
[root@k8s-master01 ~]# cat /root/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: 7t2weq.bjbawausm0jaxury
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.1.107
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: k8s-master01
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
certSANs:
- 192.168.1.236
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.1.236:16443
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.20.0 # 改成對應的版本
networking:
dnsDomain: cluster.local
podSubnet: 172.168.0.0/12 # 如果跟公司ip沖突得改
serviceSubnet: 10.96.0.0/12 # 如果跟公司ip沖突得改
scheduler: {}
更新kubeadm文件的命令（Master01上）
# 更新之后的文件是new.yaml
[root@k8s-master01 ~]# kubeadm config migrate --old-config kubeadm-config.yaml --new-config new.yaml
6.2、將new.yaml文件復制到其他節點（Master01上）
scp new.yaml k8s-master02:/root/
scp new.yaml k8s-master03:/root/
scp new.yaml k8s-node01:/root/
scp new.yaml k8s-node02:/root/
6.3、下載鏡像（Master01、02、03上）
[root@k8s-master01 ~]# kubeadm config images pull --config /root/new.yaml
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.20.0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.20.0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.20.0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0
6.4、設置開機自啟動kubelet（所有節點）
systemctl enable --now kubelet
6.5、Master01節點初始化，初始化以后會在/etc/kubernetes目錄下生成對應的證書和配置文件，之后其他Master節點加入Master01即可：（Master01上）
kubeadm init --config /root/new.yaml --upload-certs
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy
# 初始化成功以后，會產生Token值，用於其他節點加入時使用，因此要記錄下初始化成功生成的token值（令牌值）：
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
# 這個就是要在另外2個master節點執行的加入集群的命令
kubeadm join 192.168.1.236:16443 --token 7t2weq.bjbawausm0jaxury \
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4 \
--control-plane --certificate-key b66aa49bb32a780ac5b58841c50a7767a9638a9d313e23c9413218473e527ec2
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
# 這個就是要在另外2個node節點執行的加入集群的命令
kubeadm join 192.168.1.236:16443 --token 7t2weq.bjbawausm0jaxury \
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4
6.6、如果初始化失敗，重置后再次初始化，命令如下：
kubeadm reset -f ; ipvsadm --clear ; rm -rf ~/.kube
6.7、Master01節點配置環境變量，用於訪問Kubernetes集群（Master01上）
cat <<EOF >> /root/.bashrc
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
source /root/.bashrc
# 查看節點狀態：
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady control-plane,master 3m24s v1.20.0
6.8、采用初始化安裝方式，所有的系統組件均以容器的方式運行並且在kube-system命名空間內，此時可以查看Pod狀態：
[root@k8s-master01 ~]# kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-54d67798b7-8w5hd 0/1 Pending 0 4m18s <none> <none> <none> <none>
coredns-54d67798b7-vb2ll 0/1 Pending 0 4m18s <none> <none> <none> <none>
etcd-k8s-master01 1/1 Running 0 4m19s 192.168.1.107 k8s-master01 <none> <none>
kube-apiserver-k8s-master01 1/1 Running 0 4m19s 192.168.1.107 k8s-master01 <none> <none>
kube-controller-manager-k8s-master01 1/1 Running 0 4m19s 192.168.1.107 k8s-master01 <none> <none>
kube-proxy-5bws8 1/1 Running 0 4m18s 192.168.1.107 k8s-master01 <none> <none>
kube-scheduler-k8s-master01 1/1 Running 0 4m19s 192.168.1.107 k8s-master01 <none> <none>
七、高可用Master
7.1、matser02節點執行
[root@k8s-master02 ~]# kubeadm join 192.168.1.236:16443 --token 7t2weq.bjbawausm0jaxury \
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4 \
--control-plane --certificate-key b66aa49bb32a780ac5b58841c50a7767a9638a9d313e23c9413218473e527ec2
# 成功顯示，如果想在另外的master節點使用kubectl命令，就執行以下操作。
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
# matser01節點查看，已經成功加入集群
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady control-plane,master 16m v1.20.0
k8s-master02 NotReady control-plane,master 2m21s v1.20.0
7.2、如果證書失效、采用以下方法重新生成證書（master01上）
# Token過期后生成新的token：
[root@k8s-master01 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.1.236:16443 --token 8k8qzk.d43ed9gfgw1st3xi --discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4
# Master需要生成--certificate-key
[root@k8s-master01 ~]# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
43c5695789c0dc4433f480a05683d55887e836b71b452b407138d8dd54cad937
7.2.1、把master03節點加入集群（master03上）
# 執行（kubeadm token create --print-join-command）后生成的命令+執行（kubeadm init phase upload-certs --upload-certs）生成的key替換到參數（--control-plane --certificate-key）然后在master03執行
kubeadm join 192.168.1.236:16443 --token 8k8qzk.d43ed9gfgw1st3xi \
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4 \
--control-plane --certificate-key 43c5695789c0dc4433f480a05683d55887e836b71b452b407138d8dd54cad937
7.3、把node節點加入集群（node01、02上）
Node節點上主要部署公司的一些業務應用，生產環境中不建議Master節點部署系統組件之外的其他Pod，測試環境可以允許Master節點部署Pod以節省系統資源。
# 只要執行（kubeadm token create --print-join-command）后生成的命令
kubeadm join 192.168.1.236:16443 --token 8k8qzk.d43ed9gfgw1st3xi \
--discovery-token-ca-cert-hash sha256:3aa4cf3c52c1956cb86d2911fe0f6b8898bfa43c06966b2f1095e5000a00d1a4
# 看到以下的就是成功了
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
# 查看master01上查看節點信息
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady control-plane,master 33m v1.20.0
k8s-master02 NotReady control-plane,master 19m v1.20.0
k8s-master03 NotReady control-plane,master 10m v1.20.0
k8s-node01 NotReady <none> 116s v1.20.0
k8s-node02 NotReady <none> 113s v1.20.0
報錯：
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR DirAvailable--etc-kubernetes-manifests]: /etc/kubernetes/manifests is not empty
[ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
[ERROR Port-10250]: Port 10250 is in use
# /etc/kubernetes/manifests 刪除這個目錄
# 10250停了這端口
八、 Calico組件的安裝（master01執行）
8.1、切換分支
[root@k8s-master01 ~]# cd /root/k8s-ha-install && git checkout manual-installation-v1.20.x && cd calico/
Already on 'manual-installation-v1.20.x'
8.2、修改calico-etcd.yaml的以下位置
sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.1.107:2379,https://192.168.1.108:2379,https://192.168.1.109:2379"#g' calico-etcd.yaml
ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/kubernetes/pki/etcd/server.key | base64 | tr -d '\n'`
sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml
sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml
POD_SUBNET=`cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= '{print $NF}'`
sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@# value: "192.168.1.0/16"@ value: '"${POD_SUBNET}"'@g' calico-etcd.yaml
[root@k8s-master01 calico]# kubectl apply -f calico-etcd.yaml
secret/calico-etcd-secrets created
configmap/calico-config created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
查看容器狀態（需等待幾分鍾）
[root@k8s-master01 dashboard]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-5f6d4b864b-khq4h 1/1 Running 0 13m
calico-node-5tvxh 1/1 Running 0 13m
calico-node-kffn7 1/1 Running 0 13m
calico-node-lltfs 1/1 Running 0 13m
calico-node-nhgn8 1/1 Running 0 13m
coredns-54d67798b7-8w5hd 1/1 Running 0 117m
coredns-54d67798b7-vb2ll 1/1 Running 0 117m
etcd-k8s-master01 1/1 Running 0 117m
etcd-k8s-master02 1/1 Running 0 104m
kube-apiserver-k8s-master01 1/1 Running 0 117m
kube-apiserver-k8s-master02 1/1 Running 0 104m
kube-controller-manager-k8s-master01 1/1 Running 1 117m
kube-controller-manager-k8s-master02 1/1 Running 0 104m
kube-proxy-5bws8 1/1 Running 0 117m
kube-proxy-pbqjc 1/1 Running 0 104m
kube-proxy-tpwbt 1/1 Running 0 86m
kube-proxy-vbpc5 1/1 Running 0 86m
kube-scheduler-k8s-master01 1/1 Running 1 117m
kube-scheduler-k8s-master02 1/1 Running 0 104m
metrics-server-545b8b99c6-hkgnz 1/1 Running 0 2m38s
九、Metrics部署
在新版的Kubernetes中系統資源的采集均使用Metrics-server，可以通過Metrics采集節點和Pod的內存、磁盤、CPU和網絡的使用率。
9.1、將Master01節點的front-proxy-ca.crt復制到所有Node節點（Master01上）
# 其他節點自行拷貝
scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node01:/etc/kubernetes/pki/front-proxy-ca.crt
scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node02:/etc/kubernetes/pki/front-proxy-ca.crt
9.2、安裝metrics server
[root@k8s-master01 calico]# cd /root/k8s-ha-install/metrics-server-0.4.x-kubeadm/
[root@k8s-master01 metrics-server-0.4.x-kubeadm]# kubectl create -f comp.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
9.3、查看狀態
[root@k8s-master01 metrics-server-0.4.x-kubeadm]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master01 374m 18% 1158Mi 62%
k8s-master02 354m 17% 1027Mi 54%
k8s-master03 104m 2% 1082Mi 28%
k8s-node01 188m 9% 773Mi 41%
k8s-node02 446m 22% 803Mi 43
10、Dashboard部署
Dashboard用於展示集群中的各類資源，同時也可以通過Dashboard實時查看Pod的日志和在容器中執行一些命令等。
10.1、安裝指定版本dashboard
[root@k8s-master01 metrics-server-0.4.x-kubeadm]# cd /root/k8s-ha-install/dashboard/
[root@k8s-master01 dashboard]# kubectl create -f .
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
# 查看所有容器狀態
[root@k8s-master01 dashboard]# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5f6d4b864b-khq4h 1/1 Running 0 16m
kube-system calico-node-5tvxh 1/1 Running 0 16m
kube-system calico-node-kffn7 1/1 Running 0 16m
kube-system calico-node-lltfs 1/1 Running 0 16m
kube-system calico-node-nhgn8 1/1 Running 0 16m
kube-system coredns-54d67798b7-8w5hd 1/1 Running 0 121m
kube-system coredns-54d67798b7-vb2ll 1/1 Running 0 121m
kube-system etcd-k8s-master01 1/1 Running 0 121m
kube-system etcd-k8s-master02 1/1 Running 0 107m
kube-system kube-apiserver-k8s-master01 1/1 Running 0 121m
kube-system kube-apiserver-k8s-master02 1/1 Running 0 107m
kube-system kube-controller-manager-k8s-master01 1/1 Running 1 121m
kube-system kube-controller-manager-k8s-master02 1/1 Running 0 107m
kube-system kube-proxy-5bws8 1/1 Running 0 121m
kube-system kube-proxy-pbqjc 1/1 Running 0 107m
kube-system kube-proxy-tpwbt 1/1 Running 0 90m
kube-system kube-proxy-vbpc5 1/1 Running 0 90m
kube-system kube-scheduler-k8s-master01 1/1 Running 1 121m
kube-system kube-scheduler-k8s-master02 1/1 Running 0 107m
kube-system metrics-server-545b8b99c6-hkgnz 1/1 Running 0 6m25s
kubernetes-dashboard dashboard-metrics-scraper-7645f69d8c-hftvd 1/1 Running 0 2m31s
kubernetes-dashboard kubernetes-dashboard-78cb679857-zksm9 1/1 Running 0 2m32s
10.2、更改dashboard的svc為NodePort
# 將type的ClusterIP更改為NodePort（如果已經為NodePort忽略此步驟）：
[root@k8s-master01 dashboard]# kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
type: NodePort
10.3、登錄dashboard
# 查看dashboard信息，端口映射
[root@k8s-master01 dashboard]# kubectl get svc kubernetes-dashboard -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.111.114.182 <none> 443:32209/TCP 10m
# 頁面訪問，VIP:映射的端口
https://192.168.1.236:32209/
# 獲取token
[root@k8s-master01 dashboard]# kubectl -n kube-system describe secret
十一、一些必須的配置更改
11.1、將Kube-proxy改為ipvs模式，因為在初始化集群的時候注釋了ipvs配置，所以需要自行修改一下：（master01上）
# 查看原來的mode模式
[root@k8s-master01 ~]# curl 127.0.0.1:10249/proxyMode
iptables
# 修改
[root@k8s-master01 ~]# kubectl edit cm kube-proxy -n kube-system
mode: "ipvs" # 44行
# 更新Kube-Proxy的Pod
[root@k8s-master01 ~]# kubectl patch daemonset kube-proxy -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}" -n kube-system
daemonset.apps/kube-proxy patched
# 再次查看mode模式
[root@k8s-master01 ~]# curl 127.0.0.1:10249/proxyMode
ipvs
十二、注意事項
注意：kubeadm安裝的集群，證書有效期默認是一年。master節點的kube-apiserver、kube-scheduler、kube-controller-manager、etcd都是以容器運行的。可以通過kubectl get po -n kube-system查看。
啟動和二進制不同的是，
kubelet的配置文件在/etc/sysconfig/kubelet和/var/lib/kubelet/config.yaml
其他組件的配置文件在/etc/Kubernetes/manifests目錄下，比如kube-apiserver.yaml，該yaml文件更改后，kubelet會自動刷新配置，也就是會重啟pod。不能再次創建該文件
Kubeadm安裝后，master節點默認不允許部署pod，可以通過以下方式打開：
# 查看Taints：
[root@k8s-master01 ~]# kubectl describe node -l node-role.kubernetes.io/master= | grep Taints
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
# 刪除Taint：
[root@k8s-master01 ~]# kubectl taint node -l node-role.kubernetes.io/master node-role.kubernetes.io/master:NoSchedule-
node/k8s-master01 untainted
node/k8s-master02 untainted
node/k8s-master03 untainted
[root@k8s-master01 ~]# kubectl describe node -l node-role.kubernetes.io/master= | grep Taints
Taints: <none>
Taints: <none>
Taints: <none>