一、環境介紹
剛開始學k8s，使用二進制搭建k8s集群，網上教程大多都是v1.20版本的，要搞就搞難的，直接部署一個目前最新版v1.22.2的，想着報了錯就地解決以后工作中遇到了也好整。
好家伙，真沒想到搞了將近15天，裝的我人都傻了。
本套搭建是參考這篇博文，非常感謝該博主解惑。
1.1 規划
先部署單Master節點環境，之后再擴容成為多Master節點，以及多Work節點。
| 節點 | IP | 復用 |
|---|---|---|
| k8s-master01 | 10.154.0.111 | etcd01 |
| k8s-node01 | 10.154.0.112 | etcd02 |
| k8s-node02 | 10.154.0.113 | etcd03 |
這里節點復用，把etcd集群裝在這三個節點上
如果你的實驗環境IP跟我的不一樣，不要手動改，直接ctrl+h替換為你的IP，一定要注意，這樣避免改錯
1.2 環境配置
以下如未特別說明，則所有機器都要做，使用xshell-->工具-->發送鍵到所有會話會很方便操作
如果你的linux內核小於5.x，需要先更新內核
#修改時區，同步時間
yum install ntpdate -y
ntpdate time2.aliyun.com
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' > /etc/timezone
crontab -e
0 12 * * * /usr/sbin/ntpdate time2.aliyun.com
#關閉防火牆，selinux，swap
systemctl stop firewalld
systemctl disable firewalld
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
#系統優化
cat > /etc/sysctl.d/k8s_better.conf << EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF
sysctl -p /etc/sysctl.d/k8s_better.conf
#修改hosts文件
cat >> /etc/hosts << "EOF"
10.154.0.111 k8s-master01
10.154.0.112 k8s-node01
10.154.0.113 k8s-node02
EOF
#確保每台機器的uuid不一致，如果是克隆機器，修改網卡配置文件刪除uuid那一行
cat /sys/class/dmi/id/product_uuid
#更改主機名，根據實際情況修改，分別在每台操作
hostnamectl set-hostname k8s-master01
hostnamectl set-hostname k8s-node01
hostnamectl set-hostname k8s-node02
#配置免密登錄，在master01上操作
ssh-keygen -t rsa
ssh-copy-id root@10.154.0.112
ssh-copy-id root@10.154.0.113
#重啟
reboot
二、安裝docker
這里介紹二進制安裝，yum源安裝也可以
下載docker：官方下載
我使用的是目前最新版本為v20.10.9
1.卸載舊版本docker
所有機器都做
yum remove -y docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine \
docker-ce*
rm -rf /var/lib/docker
2.解壓二進制包
在master01上操作
mkdir ~/tools
cd ~/tools
tar zxvf docker-20.10.9.tgz
cp docker/* /usr/bin
scp -r docker/* root@10.154.0.112:/usr/bin
scp -r docker/* root@10.154.0.113:/usr/bin
3.使用systemd管理docker
在master01上操作
cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
scp -r /usr/lib/systemd/system/docker.service root@10.154.0.112:/usr/lib/systemd/system/docker.service
scp -r /usr/lib/systemd/system/docker.service root@10.154.0.113:/usr/lib/systemd/system/docker.service
4.配置鏡像加速
在master01上操作
mkdir /etc/docker
cat > /etc/docker/daemon.json << "EOF"
{
"exec-opts": [
"native.cgroupdriver=systemd"
],
"log-driver": "json-file",
"log-level": "warn",
"log-opts": {
"max-size": "1000m",
"max-file": "3"
},
"registry-mirrors": [
"https://xxxxxx.mirror.aliyuncs.com"
],
"insecure-registries": [],
"selinux-enabled": false
}
EOF
scp -r /etc/docker/daemon.json root@10.154.0.112:/etc/docker/daemon.json
scp -r /etc/docker/daemon.json root@10.154.0.113:/etc/docker/daemon.json
最好在這里把cgroupdriver改為systemd，不然API啟動會報錯
這里使用的阿里雲鏡像加速器，需要自己申請
5.啟動並設置開機啟動
所有機器上操作
systemctl daemon-reload
systemctl start docker
systemctl enable docker
systemctl status docker
三、部署etcd
下載etcd：官方下載
我使用的是目前最新版本為3.5.1
3.1 頒發證書
以下沒有特殊說明均在master01上操作
1.創建目錄
mkdir -p /opt/cluster/ssl/{rootca,etcd,kubernetes}
mkdir -p /opt/cluster/kubelet/ssl
mkdir -p /opt/cluster/log/{kube-apiserver,kube-controller-manager,kube-scheduler,kube-proxy,kubelet}
mkdir -p /opt/cluster/plugins/{calico,coredns}
mkdir -p /opt/cluster/etcd/{data,wal}
2.上傳工具
cd ~/tools
mv cfssl_1.6.1_linux_amd64 cfssl
mv cfssl-certinfo_1.6.1_linux_amd64 cfssl-certinfo
mv cfssljson_1.6.1_linux_amd64 cfssljson
chmod +x cfssl*
cp cfssl* /usr/local/bin
3.生成證書
cd /opt/cluster/ssl
cat > cfssl-conf.json << "EOF"
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"common": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
cd /opt/cluster/ssl
cat > rootca/rootca-csr.json << "EOF"
{
"CN": "rootca",
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "ROOTCA",
"OU": "tz"
}]
}
EOF
cd /opt/cluster/ssl
cat > etcd/etcd-csr.json << "EOF"
{
"CN": "etcd-cluster",
"hosts": [
"127.0.0.1",
"10.154.0.111",
"10.154.0.112",
"10.154.0.113"
],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "KUBERNETES-ETCD",
"OU": "tz"
}]
}
EOF
cd /opt/cluster/ssl
cfssl gencert -initca rootca/rootca-csr.json | cfssljson -bare rootca/rootca
cfssl gencert \
-ca=rootca/rootca.pem \
-ca-key=rootca/rootca-key.pem \
--config=cfssl-conf.json \
-profile=common etcd/etcd-csr.json | cfssljson -bare etcd/etcd
4.證書傳給其他機器
scp -r /opt/cluster/ssl 10.154.0.112:/opt/cluster/
scp -r /opt/cluster/ssl 10.154.0.113:/opt/cluster/
3.2 部署etcd
1.上傳工具
cd ~/tools/
tar zxvf etcd-v3.5.1-linux-amd64.tar.gz
cp etcd-v3.5.1-linux-amd64/{etcd,etcdctl} /usr/local/bin
chmod +x /usr/local/bin/
scp -r etcd-v3.5.1-linux-amd64/{etcd,etcdctl} root@10.154.0.112:/usr/local/bin
scp -r etcd-v3.5.1-linux-amd64/{etcd,etcdctl} root@10.154.0.113:/usr/local/bin
2.編寫systemd配置文件
k8s-master01配置文件如下
cat > /usr/lib/systemd/system/etcd.service << "EOF"
[Unit]
Description=Kubernetes:Etcd
After=network.target network-online.target
Wants=network-online.target
[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/etcd \
--name=etcd01 \
--data-dir=/opt/cluster/etcd/data \
--wal-dir=/opt/cluster/etcd/wal \
--listen-peer-urls=https://10.154.0.111:2380 \
--listen-client-urls=https://10.154.0.111:2379,http://127.0.0.1:2379 \
--initial-advertise-peer-urls=https://10.154.0.111:2380 \
--initial-cluster=etcd01=https://10.154.0.111:2380,etcd02=https://10.154.0.112:2380,etcd03=https://10.154.0.113:2380 \
--initial-cluster-state=new \
--initial-cluster-token=373b3543a301630c \
--advertise-client-urls=https://10.154.0.111:2379 \
--cert-file=/opt/cluster/ssl/etcd/etcd.pem \
--key-file=/opt/cluster/ssl/etcd/etcd-key.pem \
--peer-cert-file=/opt/cluster/ssl/etcd/etcd.pem \
--peer-key-file=/opt/cluster/ssl/etcd/etcd-key.pem \
--trusted-ca-file=/opt/cluster/ssl/rootca/rootca.pem \
--peer-trusted-ca-file=/opt/cluster/ssl/rootca/rootca.pem \
--client-cert-auth=true \
--peer-client-cert-auth=true \
--logger=zap \
--log-outputs=default \
--log-level=info \
--listen-metrics-urls=https://10.154.0.111:2381 \
--enable-pprof=false
[Install]
WantedBy=multi-user.target
EOF
k8s-node01配置文件如下
cat > /usr/lib/systemd/system/etcd.service << "EOF"
[Unit]
Description=Kubernetes:Etcd
After=network.target network-online.target
Wants=network-online.target
[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/etcd \
--name=etcd02 \
--data-dir=/opt/cluster/etcd/data \
--wal-dir=/opt/cluster/etcd/wal \
--listen-peer-urls=https://10.154.0.112:2380 \
--listen-client-urls=https://10.154.0.112:2379,http://127.0.0.1:2379 \
--initial-advertise-peer-urls=https://10.154.0.112:2380 \
--initial-cluster=etcd01=https://10.154.0.111:2380,etcd02=https://10.154.0.112:2380,etcd03=https://10.154.0.113:2380 \
--initial-cluster-state=new \
--initial-cluster-token=373b3543a301630c \
--advertise-client-urls=https://10.154.0.112:2379 \
--cert-file=/opt/cluster/ssl/etcd/etcd.pem \
--key-file=/opt/cluster/ssl/etcd/etcd-key.pem \
--peer-cert-file=/opt/cluster/ssl/etcd/etcd.pem \
--peer-key-file=/opt/cluster/ssl/etcd/etcd-key.pem \
--trusted-ca-file=/opt/cluster/ssl/rootca/rootca.pem \
--peer-trusted-ca-file=/opt/cluster/ssl/rootca/rootca.pem \
--client-cert-auth=true \
--peer-client-cert-auth=true \
--logger=zap \
--log-outputs=default \
--log-level=info \
--listen-metrics-urls=https://10.154.0.112:2381 \
--enable-pprof=false
[Install]
WantedBy=multi-user.target
EOF
k8s-node02配置文件如下
cat > /usr/lib/systemd/system/etcd.service << "EOF"
[Unit]
Description=Kubernetes:Etcd
After=network.target network-online.target
Wants=network-online.target
[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/etcd \
--name=etcd03 \
--data-dir=/opt/cluster/etcd/data \
--wal-dir=/opt/cluster/etcd/wal \
--listen-peer-urls=https://10.154.0.113:2380 \
--listen-client-urls=https://10.154.0.113:2379,http://127.0.0.1:2379 \
--initial-advertise-peer-urls=https://10.154.0.113:2380 \
--initial-cluster=etcd01=https://10.154.0.111:2380,etcd02=https://10.154.0.112:2380,etcd03=https://10.154.0.113:2380 \
--initial-cluster-state=new \
--initial-cluster-token=373b3543a301630c \
--advertise-client-urls=https://10.154.0.113:2379 \
--cert-file=/opt/cluster/ssl/etcd/etcd.pem \
--key-file=/opt/cluster/ssl/etcd/etcd-key.pem \
--peer-cert-file=/opt/cluster/ssl/etcd/etcd.pem \
--peer-key-file=/opt/cluster/ssl/etcd/etcd-key.pem \
--trusted-ca-file=/opt/cluster/ssl/rootca/rootca.pem \
--peer-trusted-ca-file=/opt/cluster/ssl/rootca/rootca.pem \
--client-cert-auth=true \
--peer-client-cert-auth=true \
--logger=zap \
--log-outputs=default \
--log-level=info \
--listen-metrics-urls=https://10.154.0.113:2381 \
--enable-pprof=false
[Install]
WantedBy=multi-user.target
EOF
3.啟動etcd
所有機器都操作
systemctl daemon-reload && \
systemctl enable etcd.service && \
systemctl start etcd.service && \
systemctl status etcd.service
報錯看這里
journalctl -u etcd >error.log
vim error.log
如果要重新部署一定要清理殘留數據
rm -rf /opt/cluster/etcd/wal/
rm -rf /opt/cluster/etcd/data/
rm -rf /opt/cluster/ssl/etcd/
4.驗證
任意一台都可執行
ETCDCTL_API=3 /usr/local/bin/etcdctl \
--cacert=/opt/cluster/ssl/rootca/rootca.pem \
--cert=/opt/cluster/ssl/etcd/etcd.pem \
--key=/opt/cluster/ssl/etcd/etcd-key.pem \
--endpoints="https://10.154.0.111:2379,https://10.154.0.112:2379,https://10.154.0.113:2379" \
endpoint health --write-out=table
四、部署API
本小節搭建是參考這篇博文，非常感謝該博主解惑。
4.1 整體規划
前面我們簡單說了一下單點master部署環境，這里說一下整體架構環境。
| 節點 | IP | 復用 |
|---|---|---|
| k8s-master01 | 10.154.0.111 | etcd01 |
| k8s-master02 | 10.154.0.115 | |
| k8s-node01 | 10.154.0.112 | etcd02 |
| k8s-node02 | 10.154.0.113 | etcd03 |
| k8s-node03 | 10.154.0.114 | |
| k8s-lb01 | 10.154.0.116 | |
| k8s-lb02 | 10.154.0.117 | |
| vip地址 | 10.154.0.118 |
這里使用兩台Master節點，三台Work節點，兩台負載均衡服務器，一共七台機器。
負載均衡器使用KEEPALIVE+LVS來實現，會用到一個虛擬IP地址。
etcd集群復用的一台Master兩台Work節點。
在這里說這些主要是kube-api生成證書需要用到這些ip地址
4.2 下載API
我使用的是目前最新版本為v1.22.2
下載api：官方下載
注：打開鏈接你會發現里面有很多包，下載一個Server Binaries包就夠了，里面包含了Master和Worker節點的二進制文件。
4.3 頒發證書
1.上傳工具
cd ~/tools/
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /usr/local/bin
cp kubectl /usr/local/bin
2.生成證書
cd /opt/cluster/ssl
cat > kubernetes/kube-apiserver-csr.json << "EOF"
{
"CN": "kube-apiserver",
"hosts": [
"127.0.0.1",
"10.154.0.111",
"10.154.0.115",
"10.154.0.116",
"10.154.0.117",
"10.154.0.118",
"10.96.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "tz"
}]
}
EOF
cd /opt/cluster/ssl
cfssl gencert \
-ca=rootca/rootca.pem \
-ca-key=rootca/rootca-key.pem \
--config=cfssl-conf.json \
-profile=common kubernetes/kube-apiserver-csr.json | cfssljson -bare kubernetes/kube-apiserver
Work節點的證書使用API授權，不自己簽發，所以這里的IP地址除了Work節點不用寫，其他都要寫。
10.96.0.1是service-cluster-ip的首個IP
4.4 部署API
1.生成token.csv
cd /opt/cluster/ssl
echo $(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap" > kubernetes/kube-apiserver.token.csv
Work節點請求證書需要用到，這里是注冊了一個低權限的用戶kubelet-bootstrap，工作節點使用該用戶向API請求證書
2.編寫systemd配置文件
cat > /usr/lib/systemd/system/kube-apiserver.service << "EOF"
[Unit]
Description=Kubernetes:Apiserver
After=network.target network-online.target
Wants=network-online.target
[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/kube-apiserver \
--runtime-config=api/all=true \
--anonymous-auth=false \
--bind-address=0.0.0.0 \
--advertise-address=10.154.0.111 \
--secure-port=6443 \
--tls-cert-file=/opt/cluster/ssl/kubernetes/kube-apiserver.pem \
--tls-private-key-file=/opt/cluster/ssl/kubernetes/kube-apiserver-key.pem \
--client-ca-file=/opt/cluster/ssl/rootca/rootca.pem \
--etcd-cafile=/opt/cluster/ssl/rootca/rootca.pem \
--etcd-certfile=/opt/cluster/ssl/etcd/etcd.pem \
--etcd-keyfile=/opt/cluster/ssl/etcd/etcd-key.pem \
--etcd-servers=https://10.154.0.111:2379,https://10.154.0.112:2379,https://10.154.0.113:2379 \
--kubelet-client-certificate=/opt/cluster/ssl/kubernetes/kube-apiserver.pem \
--kubelet-client-key=/opt/cluster/ssl/kubernetes/kube-apiserver-key.pem \
--service-account-key-file=/opt/cluster/ssl/rootca/rootca-key.pem \
--service-account-signing-key-file=/opt/cluster/ssl/rootca/rootca-key.pem \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/cluster/ssl/kubernetes/kube-apiserver.token.csv \
--allow-privileged=true \
--service-cluster-ip-range=10.96.0.0/16 \
--service-node-port-range=30000-50000 \
--authorization-mode=RBAC,Node \
--enable-aggregator-routing=true \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/cluster/log/kube-apiserver/audit.log \
--logtostderr=false \
--v=2 \
--log-dir=/opt/cluster/log/kube-apiserver
[Install]
WantedBy=multi-user.target
EOF
3.啟動
systemctl daemon-reload && \
systemctl enable --now kube-apiserver.service && \
systemctl status kube-apiserver.service
報錯看日志
journalctl -u kube-apiserver > error.log
vim error.log
五、部署kubectl
kubectl是一個管理集群的工具
1.生成證書
cd /opt/cluster/ssl
cat > kubernetes/kubectl-csr.json << "EOF"
{
"CN": "clusteradmin",
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "tz"
}]
}
EOF
cd /opt/cluster/ssl
cfssl gencert -ca=rootca/rootca.pem \
-ca-key=rootca/rootca-key.pem \
--config=cfssl-conf.json \
-profile=common kubernetes/kubectl-csr.json | cfssljson -bare kubernetes/kubectl
2.生成kubeconfig配置文件
我是先部署的單點Master集群，還沒用負載均衡器，所以這里的--server填寫的是k8s-master01的地址，如果部署了負載均衡器，則填寫VIP地址。
cd /opt/cluster/ssl
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/cluster/ssl/rootca/rootca.pem \
--embed-certs=true \
--server=https://10.154.0.111:6443 \
--kubeconfig=kubernetes/kubectl.kubeconfig
kubectl config set-credentials clusteradmin \
--client-certificate=/opt/cluster/ssl/kubernetes/kubectl.pem \
--client-key=/opt/cluster/ssl/kubernetes/kubectl-key.pem \
--embed-certs=true \
--kubeconfig=kubernetes/kubectl.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=clusteradmin \
--kubeconfig=kubernetes/kubectl.kubeconfig
kubectl config use-context default \
--kubeconfig=kubernetes/kubectl.kubeconfig
mkdir /root/.kube
cp /opt/cluster/ssl/kubernetes/kubectl.kubeconfig /root/.kube/config
報錯看日志
journalctl -u kubectl > error.log
vim error.log
如果重新部署需要刪除相關證書
rm -rf /opt/cluster/ssl/kubernetes/kubectl*
rm -rf /opt/cluster/ssl/kubernetes/kube-api*
3.獲取集群信息
kubectl cluster-info
kubectl get cs
kubectl get all --all-namespaces
#命令補全[需要退出SHELL環境重新進入]
kubectl completion bash > /usr/share/bash-completion/completions/kubectl
六、部署controller-manager
本小節搭建是參考這篇博文，非常感謝該博主解惑。
6.1 簽發證書
這里給Master節點簽發證書
cd /opt/cluster/ssl
cat > kubernetes/kube-controller-manager-csr.json << "EOF"
{
"CN": "system:kube-controller-manager",
"hosts": [
"127.0.0.1",
"10.154.0.111",
"10.154.0.115"
],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "KUBERNETES",
"OU": "tz"
}]
}
EOF
cd /opt/cluster/ssl
cfssl gencert -ca=rootca/rootca.pem \
-ca-key=rootca/rootca-key.pem \
--config=cfssl-conf.json \
-profile=common kubernetes/kube-controller-manager-csr.json | cfssljson -bare kubernetes/kube-controller-manager
6.2 部署controller-manager
1.編寫kubeconfig配置文件
cd /opt/cluster/ssl
kubectl config set-cluster kubernetes --certificate-authority=/opt/cluster/ssl/rootca/rootca.pem \
--embed-certs=true --server=https://10.154.0.111:6443 \
--kubeconfig=kubernetes/kube-controller-manager.kubeconfig
kubectl config set-credentials kube-controller-manager --client-certificate=kubernetes/kube-controller-manager.pem \
--client-key=kubernetes/kube-controller-manager-key.pem --embed-certs=true \
--kubeconfig=kubernetes/kube-controller-manager.kubeconfig
kubectl config set-context default --cluster=kubernetes --user=kube-controller-manager \
--kubeconfig=kubernetes/kube-controller-manager.kubeconfig
kubectl config use-context default --kubeconfig=kubernetes/kube-controller-manager.kubeconfig
2.編寫systemd配置文件
cat > /usr/lib/systemd/system/kube-controller-manager.service << "EOF"
[Unit]
Description=Kubernetes:Kube-Controller-Manager
After=network.target network-online.target
Wants=network-online.target
[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/kube-controller-manager \
--cluster-name=kubernetes \
--secure-port=10257 \
--bind-address=127.0.0.1 \
--service-cluster-ip-range=10.96.0.0/16 \
--allocate-node-cidrs=true \
--cluster-cidr=10.97.0.0/16 \
--leader-elect=true \
--controllers=*,bootstrapsigner,tokencleaner \
--kubeconfig=/opt/cluster/ssl/kubernetes/kube-controller-manager.kubeconfig \
--tls-cert-file=/opt/cluster/ssl/kubernetes/kube-controller-manager.pem \
--tls-private-key-file=/opt/cluster/ssl/kubernetes/kube-controller-manager-key.pem \
--cluster-signing-cert-file=/opt/cluster/ssl/rootca/rootca.pem \
--cluster-signing-key-file=/opt/cluster/ssl/rootca/rootca-key.pem \
--cluster-signing-duration=87600h0m0s \
--use-service-account-credentials=true \
--root-ca-file=/opt/cluster/ssl/rootca/rootca.pem \
--service-account-private-key-file=/opt/cluster/ssl/rootca/rootca-key.pem \
--logtostderr=false \
--v=2 \
--log-dir=/opt/cluster/log/kube-controller-manager
[Install]
WantedBy=multi-user.target
EOF
3.啟動
systemctl daemon-reload && \
systemctl enable --now kube-controller-manager.service && \
systemctl status kube-controller-manager.service
驗證
kubectl get componentstatuses
報錯查看日志
journalctl -u kube-controller-manager > error.log
vim error.log
七、部署scheduler
7.1 簽發證書
這里給Master節點簽發證書
cd /opt/cluster/ssl
cat > kubernetes/kube-scheduler-csr.json << "EOF"
{
"CN": "system:kube-scheduler",
"hosts": [
"127.0.0.1",
"10.154.0.111",
"10.154.0.115"
],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "KUBERNETES",
"OU": "tz"
}]
}
EOF
cd /opt/cluster/ssl
cfssl gencert \
-ca=rootca/rootca.pem \
-ca-key=rootca/rootca-key.pem \
--config=cfssl-conf.json \
-profile=common kubernetes/kube-scheduler-csr.json | cfssljson -bare kubernetes/kube-scheduler
7.2 部署kube-scheduler
1.編寫kubeconfig配置文件
cd /opt/cluster/ssl
kubectl config set-cluster kubernetes --certificate-authority=/opt/cluster/ssl/rootca/rootca.pem \
--embed-certs=true --server=https://10.154.0.111:6443 \
--kubeconfig=kubernetes/kube-scheduler.kubeconfig
kubectl config set-credentials kube-scheduler --client-certificate=kubernetes/kube-scheduler.pem \
--client-key=kubernetes/kube-scheduler-key.pem --embed-certs=true \
--kubeconfig=kubernetes/kube-scheduler.kubeconfig
kubectl config set-context default --cluster=kubernetes --user=kube-scheduler \
--kubeconfig=kubernetes/kube-scheduler.kubeconfig
kubectl config use-context default --kubeconfig=kubernetes/kube-scheduler.kubeconfig
2.編寫systemd配置文件
cat > /usr/lib/systemd/system/kube-scheduler.service << "EOF"
[Unit]
Description=Kubernetes:Kube-Scheduler
After=network.target network-online.target
Wants=network-online.target
[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/kube-scheduler \
--kubeconfig=/opt/cluster/ssl/kubernetes/kube-scheduler.kubeconfig \
--address=127.0.0.1 \
--leader-elect=true \
--logtostderr=false \
--v=2 \
--log-dir=/opt/cluster/log/kube-scheduler
[Install]
WantedBy=multi-user.target
EOF
3.啟動
systemctl daemon-reload && \
systemctl enable --now kube-scheduler.service && \
systemctl status kube-scheduler.service
驗證
kubectl get cs
報錯查看日志
journalctl -u kube-controller-manager > error.log
vim error.log
八、部署kubelet
本小節搭建是參考這篇博文，非常感謝該博主解惑。
1.上傳工具
cd /root/tools/kubernetes/server/bin
cp kubelet kube-proxy /usr/local/bin
scp -r kubelet kube-proxy root@10.154.0.112:/usr/local/bin
scp -r kubelet kube-proxy root@10.154.0.113:/usr/local/bin
2.編寫kubeconfig配置文件
cd /opt/cluster/ssl
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
kubectl config set-cluster kubernetes --certificate-authority=/opt/cluster/ssl/rootca/rootca.pem \
--embed-certs=true --server=https://10.154.0.111:6443 \
--kubeconfig=kubernetes/kubelet-bootstrap.kubeconfig
kubectl config set-credentials kubelet-bootstrap --token=$(awk -F "," '{print $1}' /opt/cluster/ssl/kubernetes/kube-apiserver.token.csv) \
--kubeconfig=kubernetes/kubelet-bootstrap.kubeconfig
kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap \
--kubeconfig=kubernetes/kubelet-bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=kubernetes/kubelet-bootstrap.kubeconfig
3.編寫kubelet.conf配置文件
cd /opt/cluster/ssl
cat > kubernetes/kubelet.conf << "EOF"
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 0
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /opt/cluster/ssl/rootca/rootca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
healthzBindAddress: 127.0.0.1
healthzPort: 10248
rotateCertificates: true
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF
4.編寫systemd配置文件
cat > /usr/lib/systemd/system/kubelet.service << "EOF"
[Unit]
Description=Kubernetes:Kubelet
After=network.target network-online.target docker.service
Requires=docker.service
[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/kubelet \
--bootstrap-kubeconfig=/opt/cluster/ssl/kubernetes/kubelet-bootstrap.kubeconfig \
--config=/opt/cluster/ssl/kubernetes/kubelet.conf \
--kubeconfig=/opt/cluster/kubelet/kubelet.kubeconfig \
--cert-dir=/opt/cluster/kubelet/ssl \
--network-plugin=cni \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 \
--logtostderr=false \
--v=2 \
--log-dir=/opt/cluster/log/kubelet
[Install]
WantedBy=multi-user.target
EOF
5.復制證書配置文件到其他節點
scp -r /opt/cluster/ssl root@10.154.0.112:/opt/cluster/
scp -r /opt/cluster/ssl root@10.154.0.113:/opt/cluster/
scp -r /usr/lib/systemd/system/kubelet.service root@10.154.0.112:/usr/lib/systemd/system/kubelet.service
scp -r /usr/lib/systemd/system/kubelet.service root@10.154.0.113:/usr/lib/systemd/system/kubelet.service
6.啟動
所有節點都執行
systemctl daemon-reload && \
systemctl enable --now kubelet.service && \
systemctl status kubelet.service
報錯查看日志
journalctl -u kubelet> error.log
vim error.log
7.授權證書
#查看需要授權的證書
kubectl get csr
#授權證書
kubectl certificate approve <CSR_NAME>
kubectl get node
九、部署kube-proxy
1.生成證書
cd /opt/cluster/ssl
cat > kubernetes/kube-proxy-csr.json << "EOF"
{
"CN": "system:kube-proxy",
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "KUBERNETES",
"OU": "tz"
}]
}
EOF
cd /opt/cluster/ssl
cfssl gencert \
-ca=rootca/rootca.pem \
-ca-key=rootca/rootca-key.pem \
--config=cfssl-conf.json \
-profile=common kubernetes/kube-proxy-csr.json | cfssljson -bare kubernetes/kube-proxy
2.編寫kubeconfig文件
cd /opt/cluster/ssl
kubectl config set-cluster kubernetes --certificate-authority=/opt/cluster/ssl/rootca/rootca.pem \
--embed-certs=true --server=https://10.154.0.111:6443 \
--kubeconfig=kubernetes/kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy --client-certificate=/opt/cluster/ssl/kubernetes/kube-proxy.pem \
--client-key=/opt/cluster/ssl/kubernetes/kube-proxy-key.pem --embed-certs=true \
--kubeconfig=kubernetes/kube-proxy.kubeconfig
kubectl config set-context default --cluster=kubernetes --user=kube-proxy \
--kubeconfig=kubernetes/kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kubernetes/kube-proxy.kubeconfig
3.編寫kube-proxy配置文件
cat > kubernetes/kube-proxy.conf << "EOF"
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
kubeconfig: /opt/cluster/ssl/kubernetes/kube-proxy.kubeconfig
bindAddress: 0.0.0.0
clusterCIDR: "10.97.0.0/16"
healthzBindAddress: "0.0.0.0:10256"
metricsBindAddress: "0.0.0.0:10249"
mode: ipvs
ipvs:
scheduler: "rr"
EOF
4.編寫systemd配置文件
這里需要注意，我把Master01也部署為Work節點，一方面可以監控，還能跑Pod，如果不想在Master上跑Pod，可以添加污點。
這里是在Master01上，所以--hostname-override值為k8s-master01
cat > /usr/lib/systemd/system/kube-proxy.service << "EOF"
[Unit]
Description=Kubernetes:Kube-Proxy
After=network.target network-online.target
Wants=network-online.target
[Service]
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/kube-proxy \
--config=/opt/cluster/ssl/kubernetes/kube-proxy.conf \
--logtostderr=false \
--v=2 \
--log-dir=/opt/cluster/log/kube-proxy \
--hostname-override=k8s-master01
[Install]
WantedBy=multi-user.target
EOF
5.復制證書跟配置文件到其他節點
scp -r /opt/cluster/ssl 10.154.0.112:/opt/cluster/
scp -r /opt/cluster/ssl 10.154.0.113:/opt/cluster/
scp -r /usr/lib/systemd/system/kube-proxy.service root@10.154.0.112:/usr/lib/systemd/system/kube-proxy.service
scp -r /usr/lib/systemd/system/kube-proxy.service root@10.154.0.113:/usr/lib/systemd/system/kube-proxy.service
在k8s-node01跟k8s-node02上修改
#在node01上修改
vim /usr/lib/systemd/system/kube-proxy.service
...
--hostname-override=k8s-node01
...
#在node02上修改
vim /usr/lib/systemd/system/kube-proxy.service
...
--hostname-override=k8s-node02
...
6.啟動
systemctl daemon-reload && \
systemctl enable --now kube-proxy.service && \
systemctl status kube-proxy.service
報錯查看日志
journalctl -u kubelet> error.log
vim error.log
十、部署網絡組件
10.1 部署calico插件
本小節環境搭建是參考這篇博文，非常感謝該博主解惑。
目前最新版本為v3.20
下載地址：官網下載
1.修改calico.yaml文件
cd /opt/cluster/plugins/calico
#在3878行
vim calico.yaml
- name: CALICO_IPV4POOL_CIDR
value: "10.97.0.0/16"
2.應用yaml文件
kubectl apply -f calico.yaml
calico網絡插件是以容器化啟動的，需要下載以下四個容器
當無法啟動calico插件時，需要先使用
docker pull拉取它們以排查是否是網絡原因造成的無法啟動
3.驗證
kubectl get pods -n kube-system
#-w可以實時查看
kubectl get pods -n kube-system -w
kubectl get node
正常情況下，calico插件狀態為Running，各節點狀態為Ready
報錯查看
#查看容器事件描述，用來排查故障
kubectl describe pod -n kube-system calico-node-b7z7v
#查看calico日志
tail -f /var/log/calico/cni/cni.log
如果重新部署需要刪除calico網絡環境
#清理網絡環境
kubectl delete -f calico.yaml
rm -rf /run/calico \
/sys/fs/bpf/calico \
/var/lib/calico \
/var/log/calico \
/opt/cluster/plugins/calico \
/opt/cni/bin/calico
#查看是否還有殘留的calico的pod
kubectl get pods -n kube-system
#強制刪除Pod
kubectl delete pod <pod名字> -n kube-system --force --grace-period=0
10.2 CoreDNS插件
目前最新版本為v1.8.6
下載地址：官網下載
1.修改coredns.yaml文件
cd /opt/cluster/plugins/coredns
vim coredns.yaml
---
...
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes CLUSTER_DOMAIN REVERSE_CIDRS { # 修改此處的"CLUSTER_DOMAIN"為"cluster.local"，表示集群域名
fallthrough in-addr.arpa ip6.arpa # 修改此處的"REVERSE_CIDRS"為"in-addr.arpa ip6.arpa"；本處的配置涉及的是DNS的反向解釋功能
}
prometheus :9153
forward . UPSTREAMNAMESERVER { # 修改此處的"UPSTREAMNAMESERVER"為"/etc/resolv.conf"；本處的配置涉及的是DNS的正向解釋功能
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}STUBDOMAINS # 刪除此處的"STUBDOMAINS"；
# 新版本的YAML文件中有這個字段[若不存在則不需要任何操作]
---
...
spec:
selector:
k8s-app: kube-dns
clusterIP: CLUSTER_DNS_IP # 修改此處的"CLUSTER_DNS_IP"為"10.96.0.10";本處為定義K8S集群內的DNS服務器的地址；
# 這個值應該與"kubelet.conf"中定義的"clusterDNS"配置項的值相同；
也可參考：部署coredns
2.應用yaml文件
cd /opt/cluster/plugins/coredns
kubectl apply -f coredns.yaml
3.驗證
#-w可以實時查看
kubectl get pods -n kube-system -w
kubectl get node
報錯查看
#查看事件日志
kubectl describe pod -n kube-system coredns-675db8b7cc-bnrn7
如果重新部署需要刪除coredns網絡環境
kubectl delete -f coredns.yaml
至此，單Master節點部署完成，后續會寫部署多Master節點跟新增Work節點