一、前期准備
獲取最新更新以及文章用到的軟件包，請移步點擊：查看更新
1、生產環境可部署Kubernetes集群的兩種方式
1）kubeadm：Kubeadm是一個K8s部署工具，提供kubeadm init和kubeadm join，用於快速部署Kubernetes集群。
官方地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
2）二進制包：從github下載發行版的二進制包，手動部署每個組件，組成Kubernetes集群。
Kubeadm降低部署門檻，但屏蔽了很多細節，遇到問題很難排查。如果想更容易可控，推薦使用二進制包部署Kubernetes集群，雖然手動部署麻煩點，期間可以學習很多工作原理，也利於后期維護。
2、安裝要求
1) 一台或多台機器，操作系統 CentOS7.x-86_x64
2) 硬件配置：2GB或更多RAM，2個CPU或更多CPU，硬盤30GB或更多
3) 集群中所有機器之間網絡互通
4) 可以訪問外網，需要拉取鏡像，如果服務器不能上網，需要提前下載鏡像並導入節點
5) 禁止swap分區
3、環境版本
| 操作系統 | CentOS7.5_x64 |
| Docker | docker-20.10.5 |
| Kubernetes | 1.20.4 |
4、單Master服務器規划
| 角色 | ip | 組件 |
| k8s-master | 192.168.112.110 | kube-apiserver，kube-controller-manager，kube-scheduler，etcd |
| k8s-node1 | 192.168.112.112 | kubelet，kube-proxy，docker etcd |
| k8s-node2 | 192.168.112.113 | kubelet，kube-proxy，docker etcd |
5、多Master服務器規划
| 角色 | ip | 組件 |
| k8s-master1 | 192.168.112.110 ，vip：192.168.112.120 |
kube-apiserver，kube-controller-manager，kube-scheduler，etcd, nginx，flannel |
| k8s-master2 | 192.168.112.111 | kube-apiserver，kube-controller-manager，kube-scheduler，etcd, nginx，flannel |
| k8s-node1 | 192.168.112.112 | kubelet，kube-proxy，docker etcd、flannel |
| k8s-node2 | 192.168.112.113 | kubelet，kube-proxy，docker etcd、flannel |
| k8s-node3 | 192.168.112.114 | kubelet，kube-proxy，docker etcd、flannel |
6、操作系統初始化配置
# 關閉防火牆 systemctl stop firewalld systemctl disable firewalld # 關閉selinux sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久 setenforce 0 # 臨時 # 關閉swap swapoff -a # 臨時 sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久 # 根據規划設置主機名 hostnamectl set-hostname <hostname> # 在master添加hosts cat >> /etc/hosts << EOF 192.168.1.71 k8s-master 192.168.1.72 k8s-node1 192.168.1.73 k8s-node2 EOF # 將橋接的IPv4流量傳遞到iptables的鏈 cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system # 生效 # 時間同步 yum install ntpdate -y ntpdate time.windows.com
二、部署etcd數據庫
Etcd 是一個分布式鍵值存儲系統，Kubernetes使用Etcd進行數據存儲，所以先准備一個Etcd數據庫，為解決Etcd單點故障，應采用集群方式部署，這里使用3台組建集群，可容忍1台機器故障，當然，你也可以使用5台組建集群，可容忍2台機器故障。
| 節點名稱 | IP地址 |
etcd-1 |
192.168.112.110 |
etcd-2 |
192.168.112.111 |
etcd-3 |
192.168.112.112 |
注：為了節省機器，這里與k8s節點機器復用。也可以獨立於k8s集群之外部署，只要apiserver能連接到就行。
1、准備cfssl證書生成工具
cfssl是一個開源的證書管理工具，使用json文件生成證書，相比openssl更方便使用。
找任意一台服務器操作，這里用Master節點。
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
2、生成Etcd證書
mkdir -p ~/TLS/{etcd,k8s} 創建工作目錄 cd ~/TLS/etcd 進入工作目錄
cat > ca-config.json << EOF { "signing": { "default": { "expiry": "175200h" }, "profiles": { "www": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json << EOF { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca - 生成證書 ls *pem ca-key.pem ca.pem
3、創建證書申請文件，使用自簽CA簽發Etcd HTTPS證書
cat > server-csr.json << EOF { "CN": "etcd", "hosts": [ "192.168.112.110", "192.168.112.111", "192.168.112.112" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server 生成證書 ls server*pem server-key.pem server.pem
注：上述文件hosts字段中IP為所有etcd節點的集群內部通信IP，一個都不能少！為了方便后期擴容可以多寫幾個預留的IP。
4、下載etcd二進制文件
下載地址：https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
5、部署etcd
創建工作目錄並解壓二進制包
mkdir /opt/etcd/{bin,cfg,ssl} -p
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
6、創建etcd配置文件
cat > /opt/etcd/cfg/etcd.conf << EOF #[Member] ETCD_NAME="etcd-1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.112.110:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.112.110:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.110:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.110:2379" ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.112.110:2380,etcd-2=https://192.168.112.111:2380,etcd-3=https://192.168.112.112:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_HEARTBEAT_INTERVAL=100 ETCD_ELECTION_TIMEOUT=500 ETCD_SNAPSHOT_COUNT=5000 EOF ETCD_NAME：節點名稱，集群中唯一 ETCD_DATA_DIR：數據目錄 ETCD_LISTEN_PEER_URLS：集群通信監聽地址 ETCD_LISTEN_CLIENT_URLS：客戶端訪問監聽地址 ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址 ETCD_ADVERTISE_CLIENT_URLS：客戶端通告地址 ETCD_INITIAL_CLUSTER：集群節點地址 ETCD_INITIAL_CLUSTER_TOKEN：集群Token ETCD_INITIAL_CLUSTER_STATE：加入集群的當前狀態，new是新集群，existing表示加入已有集群
ETCD_HEARTBEAT_INTERVAL：參數叫心跳間隔
ETCD_ELECTION_TIMEOUT：選舉超時
ETCD_SNAPSHOT_COUNT：快照時間
7、使用systemd管理etcd
cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd.conf ExecStart=/opt/etcd/bin/etcd \ --cert-file=/opt/etcd/ssl/server.pem \ --key-file=/opt/etcd/ssl/server-key.pem \ --peer-cert-file=/opt/etcd/ssl/server.pem \ --peer-key-file=/opt/etcd/ssl/server-key.pem \ --trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \ --logger=zap Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
8、拷貝剛才生成的證書，把剛才生成的證書拷貝到配置文件中的路徑
cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/
9、拷貝所有etcd文件至etcd-2，etcd-3節點
scp -r /opt/etcd/ root@192.168.112.111:/opt/ scp /usr/lib/systemd/system/etcd.service root@192.168.112.111:/usr/lib/systemd/system/ scp -r /opt/etcd/ root@192.168.112.112:/opt/ scp /usr/lib/systemd/system/etcd.service root@192.168.112.112:/usr/lib/systemd/system/
注：在節點2和節點3分別修改etcd.conf配置文件中的節點名稱和當前服務器IP：
vi /opt/etcd/cfg/etcd.conf #[Member] ETCD_NAME="etcd-1" #修改此處，節點2改為etcd-2，節點3改為etcd-3 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.112.111:2380" #修改此處為當前服務器IP ETCD_LISTEN_CLIENT_URLS="https://192.168.112.111:2379" #修改此處為當前服務器IP #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.111:2380" #修改此處為當前服務器IP ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.111:2379" #修改此處為當前服務器IP ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.112.110:2380,etcd-2=https://192.168.112.111:2380,etcd-3=https://192.168.112.112:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
10、所有服務器啟動etcd
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
11、查看集群狀態
ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.112.110:2379,https://192.168.112.111:2379,https://192.168.112.112:2379" endpoint health https://192.168.112.110:2379 is healthy: successfully committed proposal: took = 8.154404ms https://192.168.112.111:2379 is healthy: successfully committed proposal: took = 9.044117ms https://192.168.112.112:2379 is healthy: successfully committed proposal: took = 10.000825ms
如果輸出上面信息，就說明集群部署成功。如果有問題第一步先看日志：/var/log/message 或 journalctl -u etcd
三、安裝部署docker
1、下載docker
下載地址：https://download.docker.com/linux/static/stable/x86_64/docker-20.10.5.tgz
2、解壓二進制包，拷貝至其他節點
tar zxvf docker-20.10.5.tgz scp docker/* root@192.168.112.111:/usr/bin scp docker/* root@192.168.112.112:/usr/bin scp docker/* root@192.168.112.113:/usr/bin scp docker/* root@192.168.112.114:/usr/bin mv docker/* /usr/bin
3、systemd管理docker（docker節點）
cat > /usr/lib/systemd/system/docker.service << EOF [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify ExecStart=/usr/bin/dockerd ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TimeoutStartSec=0 Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target EOF
4、創建配置文件（docker節點）
mkdir /etc/docker cat > /etc/docker/daemon.json << EOF { "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"] } EOF
5、啟動並設置開機啟動（docker節點）
systemctl daemon-reload
systemctl start docker
systemctl enable docker