高可用簡介
kubernetes高可用部署參考：
https://kubernetes.io/docs/setup/independent/high-availability/
https://github.com/kubernetes-sigs/kubespray
https://github.com/wise2c-devops/breeze
https://github.com/cookeem/kubeadm-ha
一、拓撲選擇
配置高可用（HA）Kubernetes集群，有以下兩種可選的etcd拓撲：
- 集群master節點與etcd節點共存，etcd也運行在控制平面節點上
- 使用外部etcd節點，etcd節點與master在不同節點上運行
1、堆疊的etcd拓撲
堆疊HA集群是這樣的拓撲，其中etcd提供的分布式數據存儲集群與由kubeamd管理的運行master組件的集群節點堆疊部署。
每個master節點運行kube-apiserver，kube-scheduler和kube-controller-manager的一個實例。kube-apiserver使用負載平衡器暴露給工作節點。
每個master節點創建一個本地etcd成員，該etcd成員僅與本節點kube-apiserver通信。這同樣適用於本地kube-controller-manager 和kube-scheduler實例。
該拓撲將master和etcd成員耦合在相同節點上。比設置具有外部etcd節點的集群更簡單，並且更易於管理復制。
但是，堆疊集群存在耦合失敗的風險。如果一個節點發生故障，則etcd成員和master實例都將丟失，並且冗余會受到影響。您可以通過添加更多master節點來降低此風險。
因此，您應該為HA群集運行至少三個堆疊的master節點。
這是kubeadm中的默認拓撲。使用kubeadm init和kubeadm join --experimental-control-plane命令時，在master節點上自動創建本地etcd成員。
2、外部etcd拓撲
具有外部etcd的HA集群是這樣的拓撲，其中由etcd提供的分布式數據存儲集群部署在運行master組件的節點形成的集群外部。
像堆疊ETCD拓撲結構，在外部ETCD拓撲中的每個master節點運行一個kube-apiserver，kube-scheduler和kube-controller-manager實例。並且kube-apiserver使用負載平衡器暴露給工作節點。但是，etcd成員在不同的主機上運行，每個etcd主機與kube-apiserver每個master節點進行通信。
此拓撲將master節點和etcd成員分離。因此，它提供了HA設置，其中丟失master實例或etcd成員具有較小的影響並且不像堆疊的HA拓撲那樣影響集群冗余。
但是，此拓撲需要兩倍於堆疊HA拓撲的主機數。具有此拓撲的HA群集至少需要三個用於master節點的主機和三個用於etcd節點的主機。
二、初始化環境
使用kubeadm部署高可用性Kubernetes集群的兩種不同方法：
- 使用堆疊master節點。這種方法需要較少的基礎設施，etcd成員和master節點位於同一位置。
- 使用外部etcd集群。這種方法需要更多的基礎設施， master節點和etcd成員是分開的。
在繼續之前，您應該仔細考慮哪種方法最能滿足您的應用程序和環境的需求。
部署要求
- 至少3個master節點
- 至少3個worker節點
- 所有節點網絡全部互通（公共或私有網絡）
- 所有機器都有sudo權限
- 從一個設備到系統中所有節點的SSH訪問
- 所有節點安裝kubeadm和kubelet，kubectl是可選的。
- 針對外部etcd集群，你需要為etcd成員額外提供3個節點
負載均衡
部署集群前首選需要為kube-apiserver創建負載均衡器。
注意：負載平衡器有許多中配置方式。可以根據你的集群要求選擇不同的配置方案。在雲環境中，您應將master節點作為負載平衡器TCP轉發的后端。此負載平衡器將流量分配到其目標列表中的所有健康master節點。apiserver的運行狀況檢查是對kube-apiserver偵聽的端口的TCP檢查（默認值:6443）。
負載均衡器必須能夠與apiserver端口上的所有master節點通信。它還必須允許其偵聽端口上的傳入流量。另外確保負載均衡器的地址始終與kubeadm的ControlPlaneEndpoint地址匹配。
haproxy/nignx+keepalived是其中可選的負載均衡方案，針對公有雲環境可以直接使用運營商提供的負載均衡產品。
部署時首先將第一個master節點添加到負載均衡器並使用以下命令測試連接：
# nc -v LOAD_BALANCER_IP PORT
由於apiserver尚未運行，因此預計會出現連接拒絕錯誤。但是，超時意味着負載均衡器無法與master節點通信。如果發生超時，請重新配置負載平衡器以與master節點通信。將剩余的master節點添加到負載平衡器目標組。
本次使用kubeadm部署kubernetes v1.16.2 高可用集群，包含3個master節點和1個node節點，部署步驟以官方文檔為基礎，負載均衡部分采用haproxy+keepalived容器方式實現。所有組件版本以kubernetes v1.16.2為准，其他組件以當前最新版本為准。
節點信息：
主機名 IP地址 角色 OS CPU/MEM 磁盤 網卡 k8s-master01 192.168.130.130 master CentOS7.7 2C2G 60G x1 k8s-master02 192.168.130.131 master CentOS7.7 2C2G 60G x1 k8s-master03 192.168.130.132 master CentOS7.7 2C2G 60G x1 K8S VIP 192.168.130.134 – – – - -
以下操作在所有節點執行。
1、配置主機名
hostnamectl set-hostname k8s-master1 hostnamectl set-hostname k8s-master2 hostnamectl set-hostname k8s-master3
2、設置源
cat > /etc/yum.repos.d/docker-ce.repo <<EOF
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/\$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum clean all
yum makecache fast
3、執行初始化安裝腳本 chushihua.sh
#!/bin/bash yum install -y vim wget net-tools netstat bash-completion #### 配置host ####
cat >> /etc/hosts << EOF 192.168.130.130 k8s-master1 192.168.130.131 k8s-master2 192.168.130.132 k8s-master3 EOF EOF #### 關閉防火牆和iptables #### sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0 systemctl stop firewalld && systemctl disable firewalld #### 內核調優 #### cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.ipv4.ip_nonlocal_bind = 1 EOF sysctl -p echo "1" >/proc/sys/net/ipv4/ip_forward #### 關閉swap #### swapoff -a sed -i 's/.*swap.*/#&/' /etc/fstab #### 安裝ipvs模塊 #### yum install ipset ipvsadm -y cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4 #### 安裝docker #### yum install -y docker-ce-19.03.13-3.el7.x86_64 mkdir -p /data/docker /etc/docker sed -i "s#dockerd#& --graph=/data/docker#" /usr/lib/systemd/system/docker.service cat >/etc/docker/daemon.json <<EOF { "registry-mirrors": ["http://10.2.57.16:5000"], "insecure-registries": ["10.2.57.16:5000","10.2.55.8:5000"] } EOF systemctl daemon-reload && systemctl restart docker.service systemctl enable docker.service #### 安裝k8s #### yum install -y kubeadm-1.16.2-0.x86_64 kubelet-1.16.2-0.x86_64 kubectl-1.16.2-0.x86_64
三、安裝 haproxy和keepalived
kubernetes master 節點運行如下組件：
- kube-apiserver
- kube-scheduler
- kube-controller-manager
kube-scheduler 和 kube-controller-manager 可以以集群模式運行，通過 leader 選舉產生一個工作進程，其它進程處於阻塞模式。
kube-apiserver可以運行多個實例，但對其它組件需要提供統一的訪問地址，該地址需要高可用。本次部署使用 keepalived+haproxy 實現 kube-apiserver VIP 高可用和負載均衡。
haproxy+keepalived配置vip，實現了api唯一的訪問地址和負載均衡。keepalived 提供 kube-apiserver 對外服務的 VIP。haproxy 監聽 VIP，后端連接所有 kube-apiserver 實例，提供健康檢查和負載均衡功能。
運行 keepalived 和 haproxy 的節點稱為 LB 節點。由於 keepalived 是一主多備運行模式，故至少兩個 LB 節點。
本次部署復用 master 節點的三台機器，在所有3個master節點部署haproxy和keepalived組件，以達到更高的可用性，haproxy 監聽的端口(6444) 需要與 kube-apiserver的端口 6443 不同，避免沖突。
keepalived 在運行過程中周期檢查本機的 haproxy 進程狀態，如果檢測到 haproxy 進程異常，則觸發重新選主的過程，VIP 將飄移到新選出來的主節點，從而實現 VIP 的高可用。
所有組件（如 kubeclt、apiserver、controller-manager、scheduler 等）都通過 VIP +haproxy 監聽的6444端口訪問 kube-apiserver 服務。
負載均衡架構圖如下：
1、部署 haproxy
haproxy提供高可用性，負載均衡，基於TCP和HTTP的代理，支持數以萬記的並發連接。https://github.com/haproxy/haproxy
haproxy可安裝在主機上，也可使用docker容器實現。文本采用第一種。
安裝haproxy
yum install -y haproxy
創建配置文件/etc/haproxy/haproxy.cfg，重要配置以中文注釋標出：
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# https://www.haproxy.org/download/2.1/doc/configuration.txt
# https://cbonte.github.io/haproxy-dconv/2.1/configuration.html
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
mode tcp
bind *:9443 ## 監聽9443端口
option tcplog
default_backend kubernetes-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
mode tcp
balance roundrobin
server master-192.168.130.130 192.168.130.130:6443 check
server master-192.168.130.131 192.168.130.131:6443 check
server master-192.168.130.132 192.168.130.132:6443 check
分別在三個master節點啟動haproxy
systemctl restart haproxy && systemctl status haproxy
systemctl enable haproxy
2、部署keepalived
keepalived是以VRRP(虛擬路由冗余協議)協議為基礎, 包括一個master和多個backup。 master劫持vip對外提供服務。master發送組播，backup節點收不到vrrp包時認為master宕機，此時選出剩余優先級最高的節點作為新的master, 劫持vip。keepalived是保證高可用的重要組件。
keepalived可安裝在主機上，也可使用docker容器實現。文本采用第一種。(https://github.com/osixia/docker-keepalived)
安裝 keepalived
yum install -y keepalived
配置keepalived.conf, 重要部分以中文注釋標出：
! Configuration File for keepalived
global_defs {
script_user root
enable_script_security
}
vrrp_script chk_haproxy {
script "/bin/bash -c 'if [[ $(netstat -nlp | grep 9443) ]]; then exit 0; else systemctl stop keepalived;fi'"
interval 2
weight -2
}
vrrp_instance VI_1 {
interface ens33
state BACKUP
virtual_router_id 51
priority 100
nopreempt
unicast_peer {
}
virtual_ipaddress {
192.168.130.133
}
authentication {
auth_type PASS
auth_pass password
}
track_script {
chk_haproxy
}
}
- vrrp_script用於檢測haproxy是否正常。如果本機的haproxy掛掉，即使keepalived劫持vip，也無法將流量負載到apiserver。
- 我所查閱的網絡教程全部為檢測進程, 類似
killall -0 haproxy。這種方式用在主機部署上可以，但容器部署時，在keepalived容器中無法知道另一個容器haproxy的活躍情況，因此我在此處通過檢測端口號來判斷haproxy的健康狀況。 - weight可正可負。為正時檢測成功+weight，相當與節點檢測失敗時本身priority不變，但其他檢測成功節點priority增加。為負時檢測失敗本身priority減少。
- 另外很多文章中沒有強調nopreempt參數，意為不可搶占，此時master節點失敗后，backup節點也不能接管vip，因此我將此配置刪去。
分別在三台節點啟動keepalived:
systemctl start keepalived && systemctl status keepalived
systemctl enable keepalived
驗證HA狀態
也可以在本地執行該nc命令查看結果
[root@k8s-master02 ~]# yum install -y nc [root@k8s-master02 ~]# nc -v -w 2 -z 127.0.0.1 6444 2>&1 | grep 'Connected to' | grep 6444 Ncat: Connected to 127.0.0.1:6444.
四、部署k8s master節點
1、初始化master節點
初始化參考：
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1
2、創建初始化配置文件
可以使用如下命令生成初始化配置文件
kubeadm config print init-defaults > kubeadm-config.yaml
根據實際部署環境修改信息：
apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration nodeRegistration: criSocket: /var/run/dockershim.sock taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: 192.168.130.133:9443 ### 虛擬VIP + haproxy暴露的端口 controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /data/k8s/etcd ### 修改etcd數據目錄，也可以默認 imageRepository: 10.2.55.8:5000/kubernetes ###鏡像倉庫地址 kind: ClusterConfiguration kubernetesVersion: v1.16.2 ### kubernetes版本 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 scheduler: {} --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration featureGates: SupportIPVSProxyMode: true mode: ipvs
配置說明：
- controlPlaneEndpoint：為vip地址和haproxy監聽端口9443
- imageRepository:由於國內無法訪問google鏡像倉庫k8s.gcr.io，這里指定為私有倉庫地址 10.2.55.8:5000/kubernetes
- podSubnet:指定的IP地址段與后續部署的網絡插件相匹配，這里需要部署flannel插件，所以配置為10.244.0.0/16
- mode: ipvs:最后追加的配置為開啟ipvs模式。
在集群搭建完成后可以使用如下命令查看生效的配置文件：
kubectl -n kube-system get cm kubeadm-config -oyaml
3、執行創建集群
這里追加tee命令將初始化日志輸出到kubeadm-init.log中以備用（可選）。
kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.log
該命令指定了初始化時需要使用的配置文件，其中添加–experimental-upload-certs參數可以在后續執行加入節點時自動分發證書文件。
初始化示例
kubeadm init主要執行了以下操作：
- [init]：指定版本進行初始化操作
- [preflight] ：初始化前的檢查和下載所需要的Docker鏡像文件
- [kubelet-start]：生成kubelet的配置文件”/var/lib/kubelet/config.yaml”，沒有這個文件kubelet無法啟動，所以初始化之前的kubelet實際上啟動失敗。
- [certificates]：生成Kubernetes使用的證書，存放在/etc/kubernetes/pki目錄中。
- [kubeconfig] ：生成 KubeConfig 文件，存放在/etc/kubernetes目錄中，組件之間通信需要使用對應文件。
- [control-plane]：使用/etc/kubernetes/manifest目錄下的YAML文件，安裝 Master 組件。
- [etcd]：使用/etc/kubernetes/manifest/etcd.yaml安裝Etcd服務。
- [wait-control-plane]：等待control-plan部署的Master組件啟動。
- [apiclient]：檢查Master組件服務狀態。
- [uploadconfig]：更新配置
- [kubelet]：使用configMap配置kubelet。
- [patchnode]：更新CNI信息到Node上，通過注釋的方式記錄。
- [mark-control-plane]：為當前節點打標簽，打了角色Master，和不可調度標簽，這樣默認就不會使用Master節點來運行Pod。
- [bootstrap-token]：生成token記錄下來，后邊使用kubeadm join往集群中添加節點時會用到
- [addons]：安裝附加組件CoreDNS和kube-proxy
說明：無論是初始化失敗或者集群已經完全搭建成功，你都可以直接執行kubeadm reset命令清理集群或節點，然后重新執行kubeadm init或kubeadm join相關操作即可。
4、其他master加入集群
從初始化輸出或kubeadm-init.log中獲取命令
kubeadm join 192.168.130.133:9443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:9c9303b7f9e2efee5e79de84617642dcb0e62426aa4e82ae75cfd4156a5bf63c --control-plane --certificate-key db64578aa6f76ded25eb0ddcade8cae0326a81e7537539deaedbc8e0b9d4c2d9
依次將k8s-master2和k8s-master3加入到集群中，示例
[root@k8s-master2 ~]# kubeadm join 192.168.130.133:9443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:9c9303b7f9e2efee5e79de84617642dcb0e62426aa4e82ae75cfd4156a5bf63c \
> --control-plane --certificate-key db64578aa6f76ded25eb0ddcade8cae0326a81e7537539deaedbc8e0b9d4c2d9
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.13. Latest validated version: 18.09
[WARNING Hostname]: hostname "k8s-master2" could not be reached
[WARNING Hostname]: hostname "k8s-master2": lookup k8s-master2 on 192.168.130.2:53: server misbehaving
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master2 localhost] and IPs [192.168.130.131 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master2 localhost] and IPs [192.168.130.131 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.130.131 192.168.130.133]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
{"level":"warn","ts":"2020-12-25T18:55:00.821+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"passthrough:///https://192.168.130.131:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node k8s-master2 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
This node has joined the cluster and a new control plane instance was created:
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
5、配置kubectl命令
無論在master節點或node節點，要能夠執行kubectl命令必須進行以下配置：
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
等集群配置完成后，可以在所有master節點和node節點進行以上配置，以支持kubectl命令。針對node節點復制任意master節點/etc/kubernetes/admin.conf到本地。
查看當前狀態
由於未安裝網絡插件，coredns處於pending狀態，node處於notready狀態。
6、部署k8s node節點
在三台node上分別執行如下命令：
kubeadm join 192.168.130.133:9443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:9c9303b7f9e2efee5e79de84617642dcb0e62426aa4e82ae75cfd4156a5bf63c
六、部署網絡插件和dashboard
1、部署flannel網絡
kubernetes支持多種網絡方案，這里簡單介紹常用的flannel和calico安裝方法，選擇其中一種方案進行部署即可。
以下操作在master01節點執行即可。
安裝flannel網絡插件：
由於kube-flannel.yml文件指定的鏡像從coreos鏡像倉庫拉取，可能拉取失敗，可以從dockerhub搜索相關鏡像進行替換，另外可以看到yml文件中定義的網段地址段為10.244.0.0/16。
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-amd64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: 10.2.57.16:5000/kubernetes/flannel:v0.12.0-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: 10.2.57.16:5000/kubernetes/flannel:v0.12.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
執行創建
kubectl apply -f kube-flannel.yml
再次查看node和 Pod狀態，全部為Running
[root@k8s-master1 package]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready master 166m v1.16.2
k8s-master2 Ready master 166m v1.16.2
k8s-master3 Ready master 165m v1.16.2
安裝calico網絡插件（可選）：
安裝參考：https://docs.projectcalico.org/v3.6/getting-started/kubernetes/
kubectl apply -f \ https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
注意該yaml文件中默認CIDR為10.244.0.0/16，需要與初始化時kube-config.yaml中的配置一致，如果不同請下載該yaml修改后運行。
2、部署dashboard
創建yaml文件
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kube-system
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
namespace: kube-system
type: Opaque
data:
csrf: ""
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-key-holder
namespace: kube-system
type: Opaque
---
kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: kubernetesui/dashboard:v2.0.4
#imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
- --namespace=kube-system
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
nodeSelector:
"kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kube-system
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: dashboard-metrics-scraper
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
spec:
containers:
- name: dashboard-metrics-scraper
image: kubernetesui/metrics-scraper:v1.0.4
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
serviceAccountName: kubernetes-dashboard
nodeSelector:
"kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}
執行創建
[root@k8s-master1 ~]# kubectl apply -f kubernetes-dashboard.yaml
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
dashboard登錄方式
1、使用token登錄，獲取token方式如下：
kubectl describe secret -n kube-system $(kubectl get secret -n kube-system | grep kubernetes-dashboard-token | awk '{print $1}') | grep -w token | awk 'END {print $2}'
使用上面生成的token即可登錄
2、生成config配置文件
TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secret -n kube-system | grep kubernetes-dashboard-token | awk '{print $1}') | grep -w token | awk 'NR==3 {print $2}')
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true --server=10.2.57.17:9443 --kubeconfig=product.kubeconfig
kubectl config set-credentials dashboard_user --token=${TOKEN} --kubeconfig=product.kubeconfig
kubectl config set-context default --cluster=kubernetes --user=dashboard_user --kubeconfig=product.kubeconfig
kubectl config use-context default --kubeconfig=product.kubeconfig
使用生成的 product.kubeconfig 文件即可登錄
七、驗證集群狀態
查看nodes運行情況
[root@k8s-master1 package]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master1 Ready master 169m v1.16.2 192.168.130.130 <none> CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://19.3.13
k8s-master2 Ready master 169m v1.16.2 192.168.130.131 <none> CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://19.3.13
k8s-master3 Ready master 168m v1.16.2 192.168.130.132 <none> CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://19.3.13
驗證IPVS
查看kube-proxy日志，第一行輸出Using ipvs Proxier.
[root@k8s-master1 package]# kubectl -n kube-system logs -f pod/kube-proxy-55l7n
W1225 10:54:52.193126 1 feature_gate.go:208] Setting GA feature gate SupportIPVSProxyMode=true. It will be removed in a future release.
E1225 10:54:59.898661 1 node.go:124] Failed to retrieve node info: rpc error: code = Unavailable desc = etcdserver: leader changed
I1225 10:55:06.897612 1 node.go:135] Successfully retrieved node IP: 192.168.130.131
I1225 10:55:06.897651 1 server_others.go:176] Using ipvs Proxier.
W1225 10:55:06.897892 1 proxier.go:420] IPVS scheduler not specified, use rr by default
I1225 10:55:06.898219 1 server.go:529] Version: v1.16.2
I1225 10:55:06.898758 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1225 10:55:06.899497 1 config.go:313] Starting service config controller
I1225 10:55:06.899561 1 shared_informer.go:197] Waiting for caches to sync for service config
I1225 10:55:06.900251 1 config.go:131] Starting endpoints config controller
I1225 10:55:06.900268 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I1225 10:55:06.999709 1 shared_informer.go:204] Caches are synced for service config
I1225 10:55:07.000848 1 shared_informer.go:204] Caches are synced for endpoints config
查看代理規則
[root@k8s-master1 package]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.130.130:30001 rr
-> 10.244.1.3:8443 Masq 1 1 0
TCP 192.168.130.133:30001 rr
-> 10.244.1.3:8443 Masq 1 0 0
TCP 10.96.0.1:443 rr
-> 192.168.130.130:6443 Masq 1 0 0
-> 192.168.130.131:6443 Masq 1 1 0
-> 192.168.130.132:6443 Masq 1 2 0
TCP 10.96.0.10:53 rr
-> 10.244.0.2:53 Masq 1 0 0
-> 10.244.0.3:53 Masq 1 0 0
TCP 10.96.0.10:9153 rr
-> 10.244.0.2:9153 Masq 1 0 0
-> 10.244.0.3:9153 Masq 1 0 0
TCP 10.106.45.88:443 rr
-> 10.244.1.3:8443 Masq 1 0 0
TCP 10.109.201.42:8000 rr
-> 10.244.2.3:8000 Masq 1 0 0
TCP 10.244.0.0:30001 rr
-> 10.244.1.3:8443 Masq 1 0 0
TCP 10.244.0.1:30001 rr
-> 10.244.1.3:8443 Masq 1 0 0
TCP 127.0.0.1:30001 rr
-> 10.244.1.3:8443 Masq 1 0 0
TCP 172.17.0.1:30001 rr
-> 10.244.1.3:8443 Masq 1 0 0
UDP 10.96.0.10:53 rr
-> 10.244.0.2:53 Masq 1 0 0
-> 10.244.0.3:53 Masq 1 0 0
執行以下命令查看etcd集群狀態
kubectl -n kube-system exec etcd-k8s-master1 -- etcdctl --endpoints=https://192.168.130.130:2379 --ca-file=/etc/kubernetes/pki/etcd/ca.crt --cert-file=/etc/kubernetes/pki/etcd/server.crt --key-file=/etc/kubernetes/pki/etcd/server.key cluster-health
示例
[root@k8s-master1 package]# kubectl -n kube-system exec etcd-k8s-master1 -- etcdctl --endpoints=https://192.168.130.130:2379 --ca-file=/etc/kubernetes/pki/etcd/ca.crt --cert-file=/etc/kubernetes/pki/etcd/server.crt --key-file=/etc/kubernetes/pki/etcd/server.key cluster-health
member 10d7e869fb290ae4 is healthy: got healthy result from https://192.168.130.132:2379
member 5f903da6ce6e7909 is healthy: got healthy result from https://192.168.130.131:2379
member fb412e80741ff7af is healthy: got healthy result from https://192.168.130.130:2379
cluster is healthy
[root@k8s-master1 package]# kubectl -n kube-system exec etcd-k8s-master1 -- etcdctl --endpoints=https://192.168.130.130:2379 --ca-file=/etc/kubernetes/pki/etcd/ca.crt --cert-file=/etc/kubernetes/pki/etcd/server.crt --key-file=/etc/kubernetes/pki/etcd/server.key member list 10d7e869fb290ae4: name=k8s-master3 peerURLs=https://192.168.130.132:2380 clientURLs=https://192.168.130.132:2379 isLeader=false 5f903da6ce6e7909: name=k8s-master2 peerURLs=https://192.168.130.131:2380 clientURLs=https://192.168.130.131:2379 isLeader=false fb412e80741ff7af: name=k8s-master1 peerURLs=https://192.168.130.130:2380 clientURLs=https://192.168.130.130:2379 isLeader=true
查看 kube-controller-manager 狀態
[root@k8s-master1 package]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master1_456327ee-e1d5-4857-8131-e969b411e287","leaseDurationSeconds":15,"acquireTime":"2020-12-25T10:55:28Z","renewTime":"2020-12-25T14:12:51Z","leaderTransitions":1}' creationTimestamp: "2020-12-25T10:54:36Z" name: kube-controller-manager namespace: kube-system resourceVersion: "20672" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: fe5d782a-f5ad-4e3c-b146-ee25892b0ad6
查看kube-scheduler狀態
[root@k8s-master1 package]# kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml apiVersion: v1 kind: Endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-master1_cfbf1bda-431a-428c-960a-da07cdd87130","leaseDurationSeconds":15,"acquireTime":"2020-12-25T10:55:28Z","renewTime":"2020-12-25T14:14:37Z","leaderTransitions":1}' creationTimestamp: "2020-12-25T10:54:34Z" name: kube-scheduler namespace: kube-system resourceVersion: "20849" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler uid: fe659cb7-0c3e-43fc-9a73-6cc2bd9857c2
原文鏈接：https://blog.csdn.net/networken/java/article/details/89599004