02-kubeadm安裝

一. 環境准備

K8S版本v1.19.03

IP地址	角色	主機名	系統	CPU	內存	硬盤
192.168.40.11	master	node-01.in.cn	centos7	8	24G	2TB
192.168.40.12	node	node-02.in.cn	centos7	4	32G	1TB
192.168.40.13	node	node-03.in.cn	centos7	4	16G	2TB

• 操作系統: CentOS7.x
• 硬件要求: CPU 2核以上，內存4GB以上, 硬盤30G以上
• docker所在宿主機分區的格式必須是 ext2, ext3, ext4
• 關閉防火牆
• 關閉SELinux
• 關閉swap
• 所有節點部署docker, kubeadm, kubelet
• 內網互通, hosts解析主機名
• 私有倉庫或者可以訪問互聯網，拉取鏡像

注意: 在所有節點執行以下操作

1. 設置hosts

cat << EOF >> /etc/hosts
192.168.40.11 node-01.in.cn
192.168.40.12 node-02.in.cn
192.168.40.13 node-03.in.cn
EOF

2. 內核參數

net.bridge.bridge-nf-call-ip6tables = 1 
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
net.ipv4.ip_forward = 1

3. 安裝Docker

官網: https://kubernetes.io/zh/docs/setup/production-environment/container-runtimes/

#!/bin/bash

### Uninstall old versions
yum remove -y docker docker-client docker-client-latest docker-common docker-latest \
	docker-latest-logrotate docker-logrotate docker-engine

### Install required packages.
yum install yum-utils device-mapper-persistent-data lvm2 -y

### Add Docker repository.
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum-config-manager --enable docker-ce-edge

## Install Docker CE.
yum install docker-ce -y

## Create /etc/docker directory.
mkdir -p /etc/docker

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": ["https://bxba8hkt.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker

4. 添加阿里雲源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

5. 安裝 kubeadm, kubelet

# 版本 v1.19.3
yum install -y kubelet kubeadm kubectl
rpm -ql kubelet
rpm -ql kubeadm
rpm -ql kubectl
rpm -ql kubernetes-cni
# init或者join時, 自動啟動kubelet
systemctl enable kubelet 

二. 安裝Master

1. 手動下載鏡像(網絡不好時使用)

kubeadm config images list|awk -F "/" '{print $2}'

2. 下載腳本

#!/bin/bash
images=(
    kube-apiserver:v1.19.3
    kube-controller-manager:v1.19.3
    kube-scheduler:v1.19.3
    kube-proxy:v1.19.3
    pause:3.2
    etcd:3.4.13-0
    coredns:1.7.0
)

for imageName in ${images[@]};do
	docker pull mirrorgooglecontainers/$imageName  
	docker tag  mirrorgooglecontainers/$imageName registry.aliyuncs.com/google_containers/$imageName  
	docker rmi  mirrorgooglecontainers/$imageName
done

3. 初始化Master

kubeadm init \
    --apiserver-advertise-address 192.168.40.11 \
    --image-repository registry.aliyuncs.com/google_containers \
    --kubernetes-version v1.19.3 \
    --service-cidr=10.96.0.0/12 \
    --pod-network-cidr=10.244.0.0/16 

• apiserver-advertise-address: 指定用Master的哪個地址與Cluster的其他節點通信
• image-repository: 指定鏡像地址, 默認值是k8s.gcr.io
• kubernetes-version: 指定kubenets版本號, 默認值會導致從網絡上獲取最新版本號
• service-cidr: 指定service的網絡范圍
• pod-network-cidr: 指定Pod的網絡范圍

4. 使用kubectl

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 啟用kubectl命令的自動補全功能
echo "source <(kubectl completion bash)" >> ~/.bashrc 

5. 安裝網絡插件

# 添加hosts解析
199.232.68.133 raw.githubusercontent.com
# 下載yaml
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# 應用
kubectl apply -f kube-flannel.yml

6. 查看pod(大概等待2分鍾)

kubectl get pods -n kube-system
--- output
NAME                                    READY   STATUS    RESTARTS   AGE
coredns-6d56c8448f-6f7f2                1/1     Running   0          2m31s
coredns-6d56c8448f-w7vkd                1/1     Running   0          2m31s
etcd-node-01.in.cn                      1/1     Running   0          2m39s
kube-apiserver-node-01.in.cn            1/1     Running   0          2m39s
kube-controller-manager-node-01.in.cn   1/1     Running   0          2m39s
kube-flannel-ds-p9sdl                   1/1     Running   0          26s
kube-proxy-69qnj                        1/1     Running   0          2m31s
kube-scheduler-node-01.in.cn            1/1     Running   0          2m39

6. 卸載Master

# 卸載網絡插件
kubectl delete -f kube-flannel.yml
ifconfig cni0 down && ip link delete cni0
ifconfig flannel.1 down && ip link delete flannel.1
rm -rf /var/lib/cni
rm -f /etc/cni/net.d/*
### 重置iptables
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

kubeadm reset
rm -fr $HOME/.kube
systemctl stop kubepods.slice
yum remove -y kubelet kubeadm kubectl
systemctl daemon-reload
# .bashrc 刪除source <(kubectl completion bash)

三. 添加Node

1. 添加node

kubeadm join 192.168.40.11:6443 --token v5lnvk.nmcnl24les4cumci \
    --discovery-token-ca-cert-hash sha256:ddfe030df98fe66ff880ae1ce4675bb4aad29eeb11ca5e1d701f854a6449bc71
# 查看日志
journalctl -f

2. 清理node

https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#tear-down

# master執行
### 排除node上的pod
kubectl drain node-02.in.cn --delete-local-data --force --ignore-daemonsets
# node02執行
kubeadm reset
### 刪除網絡插件
ifconfig flannel.1 down && ip link delete flannel.1
rm -f /etc/cni/net.d/*
### 重置iptables
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
rm -fr /etc/kubernetes/pki

systemctl stop kubepods.slice
yum remove -y kubelet kubeadm kubectl
systemctl daemon-reload
# master 執行
kubectl delete nodes node-02.in.cn

3. 再次加入node

# master 節點獲取token
kubeadm token list
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
# node節點
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet 
systemctl start kubelet 
kubeadm join 192.168.40.11:6443 --token v5lnvk.nmcnl24les4cumci \
    --discovery-token-ca-cert-hash sha256:ddfe030df98fe66ff880ae1ce4675bb4aad29eeb11ca5e1d701f854a6449bc71

四. 故障解決

參考:  https://cloud.tencent.com/developer/article/1697300

1. kubelet 啟動不了

Failed to start ContainerManager failed to initialize top level QOS containers: failed to update top level Burstable QOS cgroup : failed to set supported cgroup subsystems for cgroup [kubepods burstable]: failed to find subsystem mount for required subsystem: pids

解決方法

方法一：編輯 kubelet 配置文件

vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--feature-gates=SupportPodPidsLimit=false,SupportNodePidsLimit=false"

五. 測試集群

kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod, svc