K8S-二進制安裝部署高可用集群環境

本文轉載自查看原文 2021-05-28 17:33 2009 雲原生(K8S)

簡述

在CentOS|RHEL平台，用二進制方式安裝高可用k8s集群1.20.x

環境說明

軟硬件環境

kubernetes軟件版本選擇

Kubernetes v1.21.2-alpha.1：內測版本（alpha）
Kubernetes v1.21.0-beta.1：公測版本（beta）
Kubernetes v1.20.2：穩定版本 (Stable)

通過 CHANGELOG 查看關聯軟件版本選型

在changelog中查找etcd默認版本('Update default etcd server version to')，如下：

網絡配置規划

網絡	主機名稱	角色	組件
192.168.10.221/24	k8s-master01	master	kube-apiserver、kube-controller-manager、kube-scheduler、etcd
192.168.10.222/24	k8s-master02	master	kube-apiserver、kube-controller-manager、kube-scheduler、etcd
192.168.10.223/24	k8s-master03	master	kube-apiserver、kube-controller-manager、kube-scheduler、etcd
192.168.10.231/24	k8s-node01	node	kubelet、kube-proxy, docker
192.168.10.232/24	k8s-node02	node	kubelet、kube-proxy, docker
192.168.10.225/32		VIP
172.16.0.0/16		Pod網段
10.96.0.0/12		Service網段

建議主機網絡，Pod網絡，Service網絡使用不同地址段

架構規划圖

基礎環境配置（所有節點）

沒有特意指出時，默認所有節點配置

升級系統

CentOS7|RHEL7 因docker|kubernetes Bug, 需要升級內核4.18+

背景原因

CentOS|RHEL 7.x 系統自帶的 3.10.x 內核存在一些 Bugs，導致運行的 Docker、Kubernetes 不穩定

解決方案

升級內核到 4.4.X (kernel-lt) 或4.18.X (kernel-ml) 以上
或手動編譯內核，disable CONFIG_MEMCG_KMEM 特性

或安裝 Docker 18.09.1 及以上的版本。但由於 kubelet 也會設置 kmem（它 vendor 了 runc），所以需要重新編譯 kubelet 並指定 GOFLAGS="-tags=nokmem"；

git clone --branch v1.14.1 --single-branch --depth 1 https://github.com/kubernetes/kubernetes
cd kubernetes
KUBE_GIT_VERSION=v1.14.1 ./build/run.sh make kubelet GOFLAGS="-tags=nokmem"

升級系統包

yum update -y --exclude=kernel*
reboot

升級內核(RHEL7|CentOS7)

# 下載 rpm 包
## 官方鏡像 http://elrepo.reloumirrors.net/kernel/el7/x86_64/RPMS/
wget -LO http://hkg.mirror.rackspace.com/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
wget -LO http://hkg.mirror.rackspace.com/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm


# 安裝
yum localinstall -y kernel-ml*

修改默認啟動內核版本

grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg

# 在 Centos/RedHat Linux 7 中啟用 user namespace
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

# 檢查確認啟動的 namespace, 如果是 y，則啟用了對應的namespace，否則未啟用
grep "CONFIG_[USER,IPC,PID,UTS,NET]*_NS" $(ls /boot/config*|tail -1)

# 在 Centos/RedHat Linux 7 中關閉 user namespace
grubby --remove-args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

升級內核（CentOS8|RHEL8)

# 可以使用dnf升級，也可以使用上面7版本的步驟升級
rpm --import https://www.elrepo.ora/RPM-GPG-KEY-elrepo.ora
yum install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm

dnf --disablerepo=\* --enablerepo=elrepo-kernel -y install kernel-ml kernel-ml-devel

grubby --default-kernel && reboot

dnf 方式自動修改成使用新版本內核啟動

檢查確認

grubby --default-kernel
reboot
uname -r

配置主機名，host文件

hostnamectl --static set-hostname  k8s-master01
hostnamectl --static set-hostname  k8s-master02
hostnamectl --static set-hostname  k8s-master03
hostnamectl --static set-hostname  k8s-node01
hostnamectl --static set-hostname  k8s-node02


cat >> /etc/hosts <<-'EOF'

# k8s hosts
192.168.10.221 k8s-master01
192.168.10.222 k8s-master02
192.168.10.223 k8s-master03

192.168.10.231 k8s-node01
192.168.10.232 k8s-node02

192.168.10.225 k8s-vip
EOF

關閉不用的服務

/sbin/chkconfig rhnsd off
systemctl stop rhnsd
systemctl disable --now NetworkManager
systemctl disable --now firewalld
systemctl disable --now postfix
systemctl disable --now rhsmcertd
systemctl disable --now irqbalance.service

關閉防火牆

systemctl disable --now firewalld.service
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager  # rhel7|CentOS7版本
# 開啟 dnsmasq 會導致 docker 容器無法解析域名，需要關閉

開啟 dnsmasq 會導致 docker 容器無法解析域名，需要關閉
rhel7|CentOS7 關閉NetworkManager

禁用SELinux

setenforce 0
# 修改/etc/selinux/config 文件，將SELINUX=enforcing改為SELINUX=disabled
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
# 該項設置需要重啟后才能生效。

關閉swap分區

cp /etc/fstab /etc/fstab_bak
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
# 選一方式
sed -i 's/.*swap.*/#&/' /etc/fstab
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

配置免秘鑰登陸

mkdir -p ~/.ssh
chmod 700 ~/.ssh
cd ~/.ssh
rm -f ~/.ssh/*

ssh-keygen -b 2048 -q -t rsa -P '' -f ~/.ssh/id_rsa
# or
# ssh-keygen -q -t rsa -N "" -f ~/.ssh/id_rsa

ssh-keygen -q -t dsa -P '' -f ~/.ssh/id_dsa

cat ~/.ssh/*.pub >> ~/.ssh/authorized_keys
ssh-keyscan -t ecdsa -H "$host_ip" >> ~/.ssh/known_hosts

chmod 600 id_dsa id_rsa
chmod 644 id_dsa.pub id_rsa.pub 
chmod 644 authorized_keys

檢查測試

alias mssh='ssh -o ConnectTimeout=3 -o ConnectionAttempts=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no'

for host in $(grep 'k8s' /etc/hosts|grep -Ev '^#|vip'); do
  echo "------ ${host} ------"
  ping $host -c 1 >/dev/null && mssh $host date
done

安裝依賴軟件包

RHEL|CentOS平台（yum）

repo文件內容

# 阿里雲 CentOS7 源
cat > /etc/yum.repos.d/ali-docker-ce.repo <<-'EOF'
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF

# 清華鏡像源
cat > /etc/yum.repos.d/th-docker-ce.repo <<-'EOF'
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/7/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/gpg
EOF

基礎軟件安裝

# 阿里雲CentOS7源
curl -o /etc/yum.repos.d/aliyun.repo https://mirrors.aliyun.com/repo/Centos-7.repo
sed -ri -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' -e 's/\$releasever/7/g' /etc/yum.repos.d/aliyun.repo

# 安裝 epel 源，用於安裝 container-selinux
yum -y install epel-release

## 安裝基礎包
yum -y install bash-completion net-tools tree wget curl make cmake gcc gcc-c++ createrepo yum-utils device-mapper-persistent-data lvm2 jq psmisc vim lrzsz git vim-enhanced ntpdate ipvsadm ipset sysstat conntrack-tools libseccomp

## 檢查
rpm -q --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" yum-utils container-selinux device-mapper-persistent-data lvm2 git wget jq psmisc vim net-tools conntrack-tools ipvsadm ipset jq sysstat curl libseccomp ntpdate 
ipvsadm -l -n

# master節點部署高可用軟件
yum -y install keepalived haproxy

kubernetes源（忽略）

# k8s 源
cat > /etc/yum.repos.d/kubernetes.repo <<-'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

[] 中括號中的是repository id，唯一，用來標識不同倉庫
name 倉庫名稱，自定義
baseurl 倉庫地址
enable 是否啟用該倉庫，默認為1表示啟用
gpgcheck 是否驗證從該倉庫獲得程序包的合法性，1為驗證
repo_gpgcheck 是否驗證元數據的合法性元數據就是程序包列表，1為驗證
gpgkey=URL 數字簽名的公鑰文件所在位置，如果gpgcheck值為1，此處就需要指定gpgkey文件的位置，如果gpgcheck值為0就不需要此項了

時間配置

調整時區（按需）

# 調整系統 TimeZone
timedatectl set-timezone Asia/Shanghai

# 當前的 UTC 時間寫入硬件時鍾
timedatectl set-local-rtc 0

時間同步

master1節點去同步互聯網時間，其他節點與master1節點進行時間同步

chrony服務端節點啟動ntpd服務，其余與服務端同步時間的節點停用ntpd服務

# 更新時間
ntpdate cn.pool.ntp.org

# chrony
yum install -y chrony

## 配置 vi /etc/chrony.conf 時間同步服務器
#注意：注釋掉默認ntp服務器，我們此處使用阿里雲公網ntp服務器
server ntp.aliyun.com iburst
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp3.aliyun.com iburst
server ntp4.aliyun.com iburst
server ntp5.aliyun.com iburst
server ntp6.aliyun.com iburst
server ntp7.aliyun.com iburst

# Allow NTP client access from local network.
allow 192.168.10.0/24


## 其它節點配置master01作為同步服務器
vi /etc/chrony.conf
server 192.168.10.221 iburst

## 啟動服務
systemctl enable --now chronyd

# 檢查確認 [時間同步狀態 ^*表示已經同步]
chronyc sources

配置內核參數

cat > /etc/sysctl.d/99-k8s.conf <<-'EOF'
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.may_detach_mounts = 1
fs.inotify.max_user_watches=89100
fs.inotify.max_user_instances=8192
fs.file-max=52706963
fs.nr_open=52706963

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.netfilter.nf_conntrack_max=2310720
net.core.somaxconn = 16384
net.ipv4.ip_forward = 1
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
# net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_timestamps = 0
net.ipv4.neigh.default.gc_thresh1=1024
net.ipv4.neigh.default.gc_thresh1=2048
net.ipv4.neigh.default.gc_thresh1=4096
net.ipv6.conf.all.disable_ipv6=1
EOF

# 為了保證br_netfilter模塊加載，我們需要執行以下命令使參數生效
sysctl -p /etc/sysctl.d/99-k8s-cri.conf
sysctl --system && modprobe br_netfilter

tcp_tw_recycle 和 Kubernetes 的 NAT 沖突，必須關閉，否則會導致服務不通
- tcp_tw_recycle參數在linux內核4.12版本之后已經移除了tcp_tw_recycle參數
- 內核4.12以前版本時，需要添加 net.ipv4.tcp_tw_recycle=0參數
關閉 IPV6，防止觸發 docker BUG

配置資源限制

cat > /etc/security/limits.d/97-k8s.conf <<-'EOF'
*    soft    nofile    655360
*    hard    nofile    131072
*    soft    nproc    655350
*    hard    nproc    655350
*    soft    memlock    unlimited
*    hard    memlock    unlimited
EOF

加載模塊

ipvs模塊配置

kube-proxy開啟ipvs的前置條件

原文：https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md

參考：https://www.qikqiak.com/post/how-to-use-ipvs-in-kubernetes/

創建配置文件

內核 4.19+版本 nf_conntrack_ipv4 已改為 nf_conntrack ，4.18以下使用 nf_conntrack_ipv4即可

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

cat > /etc/modules-load.d/ipvs.conf <<-'EOF'
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack  # 4.18 改成這個nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

重新加載配置

# 加載內核配置
systemctl enable --now systemd-modules-load.service

檢查確認

# 檢查加載的模塊
lsmod | grep --color=auto -e ip_vs -e nf_conntrack
lsmod |grep -E "ip_vs|nf_conntrack"
# 或者
cut -f1 -d " "  /proc/modules | grep -e ip_vs -e nf_conntrack

創建軟件相關目錄

mkdir -p /ups/app/kubernetes/{bin,pki,log,cfg,manifests}

mkdir -p /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes

# 創建CNI插件目錄及配置文件目錄
mkdir -p /opt/cni/bin  /etc/cni/net.d

# etct 數據目錄和 wal 目錄
mkdir -p /ups/data/k8s/{etcd,wal}
chmod 700 /ups/data/k8s/etcd

安裝 CRI（Container Runtime Interface）組件

containerd組件配置（選一）

containerd 實現了 kubernetes 的 Container Runtime Interface (CRI) 接口，提供容器運行時核心功能，如鏡像管理、容器管理等，相比 dockerd 更加簡單、健壯和可移植。

加載模塊

cat> /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter

yum 安裝containerd

yum install -y containerd.io

# 配置 containerd
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

# 替換配置文件 
sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g"  /etc/containerd/config.toml 
sed -i '/containerd.runtimes.runc.options/a\ \ \ \ \ \ \ \ \ \ \ \ SystemdCgroup = true' /etc/containerd/config.toml 
sed -i "s#https://registry-1.docker.io#https://registry.cn-hangzhou.aliyuncs.com#g"  /etc/containerd/config.toml

## 使用 systemd cgroup 驅動程序
### 結合 runc 使用 systemd cgroup 驅動，在 /etc/containerd/config.toml 中設置

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

# 啟動
systemctl daemon-reload 
systemctl enable --now containerd

下載二進制文件

wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.17.0/crictl-v1.17.0-linux-amd64.tar.gz \
  https://github.com/opencontainers/runc/releases/download/v1.0.0-rc10/runc.amd64 \
  https://github.com/containernetworking/plugins/releases/download/v0.8.5/cni-plugins-linux-amd64-v0.8.5.tgz \
  https://github.com/containerd/containerd/releases/download/v1.3.3/containerd-1.3.3.linux-amd64.tar.gz

解壓

mkdir containerd
# 不包含 runc 二進制文件
tar -xvf containerd-1.3.3.linux-amd64.tar.gz -C containerd
tar -xvf crictl-v1.17.0-linux-amd64.tar.gz

mkdir cni-plugins
sudo tar -xvf cni-plugins-linux-amd64-v0.8.5.tgz -C cni-plugins

sudo mv runc.amd64 runc

# 包含了所有 Kubernetes 需要的二進制文件
tar -C / -xf cri-containerd-cni-1.4.3-linux-amd64.tar.gz

分發二進制文件到所有 worker 節點

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp containerd/bin/*  crictl  cni-plugins/*  runc  root@${node_ip}:/opt/k8s/bin
    ssh root@${node_ip} "chmod a+x /opt/k8s/bin/* && mkdir -p /etc/cni/net.d"
  done

創建和分發 containerd 配置文件

cat > /etc/containerd/containerd-config.toml<<EOF
version = 2
root = "${CONTAINERD_DIR}/root"
state = "${CONTAINERD_DIR}/state"

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2"
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/k8s/bin"
      conf_dir = "/etc/cni/net.d"
  [plugins."io.containerd.runtime.v1.linux"]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
EOF

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/containerd/ ${CONTAINERD_DIR}/{root,state}"
    scp containerd-config.toml root@${node_ip}:/etc/containerd/config.toml
  done

創建 containerd systemd unit 文件

cat > /usr/lib/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
# Environment="PATH=/opt/k8s/bin:/bin:/sbin:/usr/bin:/usr/sbin"
ExecStartPre=-/sbin/modprobe overlay
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStart=/usr/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

[Unit]
Description=Lightweight Kubernetes
Documentation=https://containerd.io
After=network-online.target

[Service]
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStartPre=-/bin/mkdir -p /run/k8s/containerd
ExecStart=/usr/local/bin/containerd \
         -c /apps/k8s/etc/containerd/config.toml \
         -a /run/k8s/containerd/containerd.sock \
         --state /apps/k8s/run/containerd \
         --root /apps/k8s/containerd 

KillMode=process
Delegate=yes
OOMScoreAdjust=-999
LimitNOFILE=1024000   # 決定容器里面文件打開數可以在這里設置
LimitNPROC=1024000
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

啟動 containerd 服務

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp containerd.service root@${node_ip}:/etc/systemd/system
    ssh root@${node_ip} "systemctl enable containerd && systemctl restart containerd"
  done

創建和分發 crictl 配置文件

crictl 是兼容 CRI 容器運行時的命令行工具，提供類似於 docker 命令的功能

cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

# 分發到所有 worker 節點
for node_ip in ${NODE_IPS[@]}; do
    echo ">>> ${node_ip}"
    scp crictl.yaml root@${node_ip}:/etc/crictl.yaml
done

鏡像管理

導入本地鏡像

# 1.3 前
ctr cri load image.tar

# 1.3 后
ctr -n=k8s.io image import pause-v3.2.tar

檢查確認導入的鏡像

ctr images list

crictl image list

docker配置（選一）

這里選用 docker 19.03.X 版本作為CRI 。可以只在 work|Node 節點上安裝

RHEL8|CentOS8 需要單獨安裝containerd

軟件部署

二進制包安裝

wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.15.tgz

tar -xf docker-19.03.15.tgz -C /usr/local/bin --no-same-owner --strip-components=1

# 配置服務文件
cat> /usr/lib/systemd/system/docker.service <<-'EOF'
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
  
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/local/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
  
[Install]
WantedBy=multi-user.target

EOF

yum源方式安裝

# 配置docker-ce源
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -ri -e 's/\$releasever/7/g' /etc/yum.repos.d/docker-ce.repo

# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# 或者直接配置repo文件
cat > /etc/yum.repos.d/docker-ce.repo <<-'EOF'
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF

yum clean all && yum makecache fast

# 查詢docker可用版本
yum list docker-ce.x86_64 --showduplicates | sort -r 

# 檢查依賴包是否已安裝
rpm -q --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" docker-ce-19.03.* containerd.io container-selinux

#安裝具體版本的docker
yum -y install docker-ce-19.03.15 docker-ce-cli-19.03.15

docker參數配置

mkdir -p /etc/docker
mkdir -p /ups/data/docker
cat > /etc/docker/daemon.json <<EOF
{
  "graph": "/ups/data/docker",
  "storage-driver": "overlay2",
  "insecure-registries": [ "registry.access.redhat.com" ],
  "registry-mirrors": [ 
    "https://hub-mirror.c.163.com",
    "https://docker.mirrors.ustc.edu.cn",
    "https://registry.docker-cn.com",
    "https://mirror.baidubce.com"
  ],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "max-concurrent-downloads": 10,
  "max-concurrent-uploads": 5,
  "live-restore": true,
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "2"
  }
}
EOF

不希望使用https的安全機制來訪問gcr.io 時，則可以添加--insecure-registry gcr.io命令行參數啟動docker 服務的方式，表示匿名下載
max-concurrent-downloads # 下載並發數
max-concurrent-uploads # 上傳並發數
max-size # 日志文件最大到多少切割
max-file # 日志文件保留個數
live-restore # 開啟這個參數，重啟docker服務不會影響容器的運行
native.cgroupdriver=systemd # k8s 推薦使用systemd

啟動服務

systemctl daemon-reload && systemctl enable --now docker  # 開機啟動並啟動服務

# 異常時檢查服務
journalctl -u docker

信息檢查

docker version
docker info
# kubelet 建議使用sytemd類型
docker info | grep "Cgroup Driver"

ps -elfH|grep docker

鏡像

拉取鏡像

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2

# 國內鏡像
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.0
# 國外鏡像
docker pull coredns/coredns:1.8.0
docker tag  coredns/coredns:1.8.0  k8s.gcr.io/coredns:1.8.0
docker rmi  coredns/coredns:1.8.0

打包鏡像

docker save -o coredns-1.7.0.tar k8s.gcr.io/coredns:1.7.0
docker save -o pause-3.2.tar k8s.gcr.io/pause:3.2

docker save -o calico-v3.16.6.tar calico/kube-controllers:v3.16.6 calico/node:v3.16.6 calico/pod2daemon-flexvol:v3.16.6 calico/cni:v3.16.6
docker save -o coredns-v1.8.0.tar coredns/coredns:1.8.0
docker save -o dashboard-v2.1.0.tar kubernetesui/dashboard:v2.1.0 kubernetesui/metrics-scraper:v1.0.6
docker save -o metrics-server-v0.4.1.tar k8s.gcr.io/metrics-server/metrics-server:v0.4.1

導入鏡像

for images in pause-v3.2.tar calico-v3.15.3.tar coredns-1.7.0.tar dashboard-v2.1.0.tar metrics-server-v0.4.1.tar ;do
  docker load -i $images
done

或使用aliyun鏡像

二進制包部署

下載

k8s軟件包

https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

打開頁面后選擇對應版本的Server Binaries下載

# server 軟件包(已包含work node所需軟件)
curl -sSL -C - -O https://dl.k8s.io/v1.20.5/kubernetes-server-linux-amd64.tar.gz

etcd軟件包

wget https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz
# or
curl -sSL -C - -O https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz

CNI插件

kubelet組件在啟動時，在命令行選項 --network-plugin=cni 來選擇CNI插件。它會自動搜索 --cni-bin-dir (default /opt/cni/bin)指定目錄下的網絡插件，並使用 --cni-conf-dir (default /etc/cni/net.d) 目錄下配置文件設置每個Pod的網絡。CNI配置文件引用的插件必須--cni-bin-dir目錄中。

新版k8s 不需要單獨安裝CNI, calico自帶有cni插件

wget https://github.com/containernetworking/plugins/releases/download/v0.9.0/cni-plugins-linux-amd64-v0.9.0.tgz

export CNI_VER='v0.9.0'
curl -sSL -C - -O https://github.com/containernetworking/plugins/releases/download/${CNI_VER}/cni-plugins-linux-amd64-${CNI_VER}.tgz

https://github.com/projectcalico/cni-plugin/releases/tag/v3.16.8
https://github.com/projectcalico/calicoctl/releases

部署

etcd

# 解包
tar -xf etcd-v3.4.13-linux-amd64.tar.gz --no-same-owner --strip-components=1 -C /ups/app/kubernetes/bin/ etcd-v3.4.13-linux-amd64/etcd{,ctl}

# 查看版本
etcdctl version

k8s

# 解包
tar -xf kubernetes-server-linux-amd64.tar.gz --no-same-owner --strip-components=3 -C /ups/app/kubernetes/bin/ kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy,adm}

## tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /ups/app/kubernetes/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy,-aggregator,adm} kubernetes/server/bin/{mounter,apiextensions-apiserver}

# 查看版本
kubectl version
kubectl version --client=true --short=true

--strip-components=N：在提取時從文件名中刪除 NUMBER 個前導目錄

CNI插件

mkdir -p /opt/cni/bin
tar -xf cni-plugins-linux-amd64-v0.9.0.tgz -C /opt/cni/bin/

TLS軟件(只在master01配置)

介紹

CFSSL是CloudFlare開源的一款PKI/TLS工具。 CFSSL 包含一個命令行工具和一個用於簽名，驗證並且捆綁TLS證書的 HTTP API 服務。使用Go語言編寫

部署cfssl工具

這里使用cfssl軟件配置所需的證書和私鑰文件

二進制部署

export TLS_BIN_DIR="/usr/local/bin"
curl -s -L -o ${TLS_BIN_DIR}/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 && \
curl -s -L -o ${TLS_BIN_DIR}/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 && \
curl -s -L -o ${TLS_BIN_DIR}/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmox +x ${TLS_BIN_DIR}/cfssl{,json,-certinfo}

源碼編譯

#go 環境部署
yum install go
vi ~/.bash_profile
GOBIN=/root/go/bin/
PATH=$PATH:$GOBIN:$HOME/bin
export PATH
go get  github.com/cloudflare/cfssl/cmd/cfssl
go get  github.com/cloudflare/cfssl/cmd/cfssljson

查看版本

#  查看版本
cfssl version

所需鏡像源網址

在安裝kubernetes時,默認的官方鏡像都存在gcr.io上,而在國內無法直接訪問gcr.io上的鏡像的。

使用阿里雲鏡像地址

registry.aliyuncs.com/google_containers

registry.cn-hangzhou.aliyuncs.com/google_containers

使用dockerhub下的`mirrorgooglecontainers`

# 要下載kube-proxy-amd64:v1.11.3這個鏡像,可以使用docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.11.3來進行下載,下載以后對鏡像重新打標簽

# 1、先pull下來
docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.11.3

# 2、重新打標簽
docker tag docker.io/mirrorgooglecontainers/kube-proxy-amd64:v1.11.3   k8s.gcr.io/kube-proxy-amd64:v1.11.3

# 3、查看鏡像，然后就可以直接使用這個鏡像了
docker images | grep k8s.gcr.io/kube-proxy-amd64

使用國內鏡像制作的鏡像

 https://github.com/zhangguanzhang/gcr.io

ETCD集群配置

kubernetes 使用 etcd 集群持久化存儲所有 API 對象、運行數據。

可以使用外etcd集群（即不部署在 kubernetes master節點），這里在kubernetes master節點上部署etcd集群【節點個數建議3，5，7....】。

生成證書和私鑰文件（master01)

etcd集群和kubernetes集群是2套不相關的證書

創建etcd證書

在master1上生成etcd證書，然后分發到其它master節點

證書簽名請求文件

cat > etcd-ca-csr.json <<-'EOF'
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GD",
      "L": "GZ",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

cat > etcd-crs.json <<-'EOF'
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GD",
      "L": "GZ",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

創建證書和私鑰

cd k8s-ha-install-manual-installation-v1.19.x.zip\k8s-ha-install\pki

# 生成CA 證書
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /ups/app/kubernetes/pki/etcd-ca

#生成客戶端證書
cfssl gencert \
   -ca=/ups/app/kubernetes/pki/etcd-ca.pem \
   -ca-key=/ups/app/kubernetes/pki/etcd-ca-key.pem \
   -config=ca-config.json \
   -hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.10.221,192.168.10.222,192.168.10.223,m01,m02,m03,k8s001,k8s002,k8s003 \
   -profile=etcd \
   etcd-csr.json | cfssljson -bare /ups/app/kubernetes/pki/etcd

分發證書文件

MasterNodes='k8s-master02 k8s-master03 m02 m03'
WorkNodes='k8s-node01 k8s-node02 n01 n02'

for NODE in $MasterNodes; do
  ping -c 1 $NODE >/dev/null 2>&1
  if [[ "$?" = "0" ]]; then
	ssh $NODE "mkdir -p /ups/app/kubernetes/pki"
	for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do
		scp /ups/app/kubernetes/pki/${FILE} $NODE:/ups/app/kubernetes/pki/${FILE}
	done
  fi
done

配置集群

master節點etcd配置文件

注意修改主機名和IP地址

master1節點

cat > /ups/app/kubernetes/cfg/etcd.config.yml<<EOF
name: 'k8s-master01'
data-dir: /ups/data/k8s/etcd
wal-dir: /ups/data/k8s/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.10.221:2380'
listen-client-urls: 'https://192.168.10.221:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.10.221:2380'
advertise-client-urls: 'https://192.168.10.221:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.10.221:2380,k8s-master02=https://192.168.10.222:2380,k8s-master03=https://192.168.10.223:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/ups/app/kubernetes/pki/etcd.pem'
  key-file: '/ups/app/kubernetes/pki/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/ups/app/kubernetes/pki/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/ups/app/kubernetes/pki/etcd.pem'
  key-file: '/ups/app/kubernetes/pki/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/ups/app/kubernetes/pki/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

etcd-v3.4+版本中，注意log-outputs是一個切片類型
--cert-file、--key-file：etcd server 與 client 通信時使用的證書和私鑰
--trusted-ca-file：簽名 client 證書的 CA 證書，用於驗證 client 證書
--peer-cert-file、--peer-key-file：etcd 與 peer 通信使用的證書和私鑰
--peer-trusted-ca-file：簽名 peer 證書的 CA 證書，用於驗證 peer 證書

master2節點

cat > /ups/app/kubernetes/cfg/etcd.config.yml<<EOF
name: 'k8s-master02'
data-dir: /ups/data/k8s/etcd
wal-dir: /ups/data/k8s/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.10.222:2380'
listen-client-urls: 'https://192.168.10.222:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.10.222:2380'
advertise-client-urls: 'https://192.168.10.222:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.10.221:2380,k8s-master02=https://192.168.10.222:2380,k8s-master03=https://192.168.10.223:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/ups/app/kubernetes/pki/etcd.pem'
  key-file: '/ups/app/kubernetes/pki/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/ups/app/kubernetes/pki/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/ups/app/kubernetes/pki/etcd.pem'
  key-file: '/ups/app/kubernetes/pki/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/ups/app/kubernetes/pki/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

master3節點

cat > /ups/app/kubernetes/cfg/etcd.config.yml<<EOF
name: 'k8s-master03'
data-dir: /ups/data/k8s/etcd
wal-dir: /ups/data/k8s/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.10.223:2380'
listen-client-urls: 'https://192.168.10.223:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.10.223:2380'
advertise-client-urls: 'https://192.168.10.223:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.10.221:2380,k8s-master02=https://192.168.10.222:2380,k8s-master03=https://192.168.10.223:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/ups/app/kubernetes/pki/etcd.pem'
  key-file: '/ups/app/kubernetes/pki/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/ups/app/kubernetes/pki/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/ups/app/kubernetes/pki/etcd.pem'
  key-file: '/ups/app/kubernetes/pki/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/ups/app/kubernetes/pki/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

配置systemd unit文件

cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/ups/app/kubernetes/bin/etcd --config-file=/ups/app/kubernetes/cfg/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF

啟動服務

systemctl daemon-reload; systemctl enable --now etcd
systemctl status etcd -l

systemctl daemon-reload && systemctl restart etcd

# 若服務啟動異常，檢查原因
journalctl -u etcd

驗證集群

etcdctl接口

export ETCDCTL_API=3
etcdctl --endpoints="192.168.10.223:2379,192.168.10.222:2379,192.168.10.221:2379" \
  --cacert=/ups/app/kubernetes/pki/etcd-ca.pem \
  --cert=/ups/app/kubernetes/pki/etcd.pem \
  --key=/ups/app/kubernetes/pki/etcd-key.pem \
  endpoint status \
  --write-out=table
  
etcdctl --endpoints="192.168.10.223:2379,192.168.10.222:2379,192.168.10.221:2379" \
  --cacert=/ups/app/kubernetes/pki/etcd-ca.pem \
  --cert=/ups/app/kubernetes/pki/etcd.pem \
  --key=/ups/app/kubernetes/pki/etcd-key.pem \
  endpoint health -w table

etcdctl --endpoints="192.168.10.223:2379,192.168.10.222:2379,192.168.10.221:2379" \
  --cacert=/ups/app/kubernetes/pki/etcd-ca.pem \
  --cert=/ups/app/kubernetes/pki/etcd.pem \
  --key=/ups/app/kubernetes/pki/etcd-key.pem \
  member list -w table

curl命令獲取

curl http://127.0.0.1:2379/v2/members|jq

高可用軟件配置

HAProxy配置文件

Master配置HAProxy配置參數文件一樣

cat >/etc/haproxy/haproxy.cfg<<EOF
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

listen stats
  bind    *:8666
  mode    http
  stats   enable
  stats   hide-version
  stats   uri       /stats
  stats   refresh   30s
  stats   realm     Haproxy\ Statistics
  stats   auth      admin:admin

frontend k8s-master
  bind 0.0.0.0:8443
  bind 127.0.0.1:8443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01    192.168.10.221:6443  check
  server k8s-master02    192.168.10.222:6443  check
  server k8s-master03    192.168.10.223:6443  check
EOF

Nginx代理（可選）

配置文件

cat > kube-nginx.conf <<EOF
worker_processes 1;
events {
    worker_connections  1024;
}
stream {
    upstream backend {
        hash $remote_addr consistent;
        server 192.168.10.221:6443        max_fails=3 fail_timeout=30s;
        server 192.168.10.222:6443        max_fails=3 fail_timeout=30s;
        server 192.168.10.223:6443        max_fails=3 fail_timeout=30s;
    }
    server {
        listen *:16443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF

cat > /etc/nginx/nginx.conf << "EOF"
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

# 四層負載均衡，為Master apiserver組件提供負載均衡
stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
       server 192.168.10.221:6443;   # Master1 APISERVER IP:PORT
       server 192.168.10.222:6443;   # Master2 APISERVER IP:PORT
       server 192.168.10.223:6443;   # Master2 APISERVER IP:PORT
    }
    
    server {
       listen 6443;
       proxy_pass k8s-apiserver;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}
EOF

服務啟動文件

cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

配置KeepAlived

注意每個節點的state、proority、IP和網卡

各 Master 節點的 mcast_src_ip
權重參數 priority （值越大優先級越高）
virtual_router_id ：整個區域內必須值唯一（廣播）
interface：網卡設備名稱
state ： MASTER|BACKUP 模式

master1節點

cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3
    rise 2
}
vrrp_instance VI_1 {
    state MASTER
    interface ens32
    mcast_src_ip 192.168.10.221
    virtual_router_id 51
    priority 200
    advert_int 2
    authentication {
      auth_type PASS
      auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
      192.168.10.225
    }
    track_script {
      chk_apiserver
} }
EOF

master2節點

cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens32
    mcast_src_ip 192.168.10.222
    virtual_router_id 51
    priority 150
    advert_int 2
    authentication {
      auth_type PASS
      auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
      192.168.10.225
    }
    track_script {
      chk_apiserver
} }
EOF

master3節點

cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens32
    mcast_src_ip 192.168.10.223
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
      auth_type PASS
      auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
      192.168.10.225
    }
    track_script {
      chk_apiserver
} }
EOF

健康檢查配置

cat > /etc/keepalived/check_apiserver.sh <<-'EOF'
#!/bin/bash
err=0
for k in $(seq 1 5)
do
    check_code=$(pgrep kube-apiserver)
    if [[ $check_code == "" ]]; then
      err=$(expr $err + 1)
      sleep 5
      continue
    else
      err=0
      break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
EOF

# 改進,使用接口的方式檢查服務健康情況
cat > check_apiserver.sh<<EOF
#!/bin/sh

err=0 
for k in $(seq 1 5); do
  check_code=$(curl -k -s https://127.0.0.1:6443/healthz)
  if [[ $check_code != "ok" ]]; then
    err=$(expr $err + 1)
    sleep 5
    continue
  else
    err=0
    break
  fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
EOF

啟動服務

systemctl enable --now haproxy; systemctl status haproxy -l
systemctl enable --now keepalived; systemctl status keepalived -l

驗證

# 檢查狀態
systemctl status haproxy keepalived -l

# ping通VIP
ping 192.168.10.225

# 8443監聽端口
netstat -tnlp|grep -v sshd

權限認證文件配置

生成kubernetes證書

kubernetes系統的各組件需要使用TLS證書對通信進行加密，每個k8s集群都需要有獨立的CA證書體系。

生成CA證書

CA 策略配置文件

cat > ca-config.json <<-'EOF'
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "expiry": "876000h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      },
      "etcd": {
        "expiry": "876000h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      },
      "server": {
        "expiry": "876000h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },
      "client": {
        "expiry": "876000h",
        "usages": [
          "signing",
          "key encipherment",
          "client auth"
        ]
      },
      "peer": {
        "expiry": "876000h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      }
    }
  }
}
EOF

證書簽名請求文件

cat > ca-csr.json <<-'EOF'
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GD",
      "L": "GZ",
      "O": "Kubernetes",
      "OU": "system"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

生成證書和私鑰

cd k8s-ha-install-manual-installation-v1.19.x.zip\k8s-ha-install\pki

cfssl gencert -initca ca-csr.json | cfssljson -bare /ups/app/kubernetes/pki/ca

生成apiserver證書

簽名請求文件

cat > apiserver-csr.json <<-'EOF'
{
  "CN": "kube-apiserver",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GD",
      "L": "GZ",
      "O": "Kubernetes",
      "OU": "system"
    }
  ]
}
EOF

客戶端證書和私鑰

cfssl gencert -ca=/ups/app/kubernetes/pki/ca.pem \
  -ca-key=/ups/app/kubernetes/pki/ca-key.pem \
  -config=ca-config.json \
  -hostname=10.96.0.1,192.168.10.225,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.10.221,192.168.10.222,192.168.10.223,k8s-master01,k8s-master02,k8s-master03,m01,m02,m03 \
  -profile=kubernetes apiserver-csr.json | cfssljson -bare /ups/app/kubernetes/pki/apiserver

10.96.0. 1：由 kube-apiserver 指定的 service-cluster-ip-range 網段的第一個IP，如 10.96.0.1
192.168.10.225 ：kubernetes 服務的服務IP（即VIP)
hosts 字段指定授權使用該證書的 IP 和域名列表，這里列出了 master 節點 IP、kubernetes 服務的 IP 和域名

生成token文件（忽略)

cat > /ups/app/kubernetes/cfg/token.csv <<-'EOF'
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

已棄用

生成聚合證書

證書簽名請求文件

cat > front-proxy-ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF

cat > front-proxy-client-csr.json <<EOF
{
  "CN": "front-proxy-client",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF

生成客戶端證書和私鑰

cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /ups/app/kubernetes/pki/front-proxy-ca

cfssl gencert \
  -ca=/ups/app/kubernetes/pki/front-proxy-ca.pem \
  -ca-key=/ups/app/kubernetes/pki/front-proxy-ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /ups/app/kubernetes/pki/front-proxy-client

生成controller-manager證書

證書簽名請求文件

cat > manager-csr.json <<-'EOF'
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GD",
      "L": "GZ",
      "O": "system:kube-controller-manager",
      "OU": "system"
    }
  ]
}
EOF

CN 和 O 均為 system:kube-controller-manager，kubernetes 內置的 ClusterRoleBindings system:kube-controller-manager 賦予 kube-controller-manager 工作所需的權限。

客戶端證書和私鑰

cfssl gencert \
   -ca=/ups/app/kubernetes/pki/ca.pem \
   -ca-key=/ups/app/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   manager-csr.json | cfssljson -bare /ups/app/kubernetes/pki/controller-manager

生成scheduler證書

證書簽名請求文件

cat> scheduler-csr.json <<-'EOF'
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GD",
      "L": "GZ",
      "O": "system:kube-scheduler",
      "OU": "system"
    }
  ]
}
EOF

CN 和 O 均為 system:kube-scheduler，kubernetes 內置的 ClusterRoleBindings system:kube-scheduler 將賦予 kube-scheduler 工作所需的權限

客戶端證書和私鑰

cfssl gencert \
   -ca=/ups/app/kubernetes/pki/ca.pem \
   -ca-key=/ups/app/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   scheduler-csr.json | cfssljson -bare /ups/app/kubernetes/pki/scheduler

生成admin證書

證書簽名請求文件

cat> admin-csr.json <<-'EOF'
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GD",
      "L": "GZ",
      "O": "system:masters",
      "OU": "system"
    }
  ]
}
EOF

O: system:masters：kube-apiserver 收到使用該證書的客戶端請求后，為請求添加組（Group）認證標識 system:masters；
預定義的 ClusterRoleBinding cluster-admin 將 Group system:masters 與 Role cluster-admin 綁定，該 Role 授予操作集群所需的最高權限；
該證書只會被 kubectl 當做 client 證書使用，所以 hosts 字段為空；

客戶端證書和私鑰

cfssl gencert \
  -ca=/ups/app/kubernetes/pki/ca.pem \
  -ca-key=/ups/app/kubernetes/pki/ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare /ups/app/kubernetes/pki/admin

生成kube-proxy證書(跳過)

證書簽名請求文件

cat> kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GD",
      "L": "GZ",
      "O": "system:kube-proxy",
      "OU": "system"
    }
  ]
}
EOF

客戶端證書和私鑰

檢查確認證書

openssl命令

openssl x509 -noout -text -in /ups/app/kubernetes/pki/etcd.pem

確認Issuer字段的內容和etcd-ca-csr.json一致
確認Subject字段的內容和etcd-csr.json一致；
確認X509v3 Subject Alternative Name字段的內容和etcd-csr.json一致；
確認X509v3 Key Usage、Extended Key Usage字段的內容和ca-config.json中kubernetes-profile一致

使用cfss-certinfo命令

cfssl-certinfo -cert /ups/app/kubernetes/pki/etcd.pem

創建kubeconfig文件

創建controller-manager.kubeconfig 文件

# 設置一個集群項
kubectl config set-cluster kubernetes \
  --certificate-authority=/ups/app/kubernetes/pki/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.225:8443 \
  --kubeconfig=/ups/app/kubernetes/cfg/controller-manager.kubeconfig

# 設置一個用戶項
kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=/ups/app/kubernetes/pki/controller-manager.pem \
  --client-key=/ups/app/kubernetes/pki/controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=/ups/app/kubernetes/cfg/controller-manager.kubeconfig
  
# 設置上下文環境
kubectl config set-context system:kube-controller-manager@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-controller-manager \
  --kubeconfig=/ups/app/kubernetes/cfg/controller-manager.kubeconfig

# 設置為默認環境
kubectl config use-context system:kube-controller-manager@kubernetes \
  --kubeconfig=/ups/app/kubernetes/cfg/controller-manager.kubeconfig

--server：若非高可用環境，使用master01的IP和端口配置(--server=https://192.168.10.221:6443)

創建scheduler.kubeconfig 文件

kube-scheduler 使用 kubeconfig 文件訪問 apiserver，該文件提供了 apiserver 地址、嵌入的 CA 證書和 kube-scheduler 證書

kubectl config set-cluster kubernetes \
  --certificate-authority=/ups/app/kubernetes/pki/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.225:8443 \
  --kubeconfig=/ups/app/kubernetes/cfg/scheduler.kubeconfig
  
kubectl config set-credentials system:kube-scheduler \
  --client-certificate=/ups/app/kubernetes/pki/scheduler.pem \
  --client-key=/ups/app/kubernetes/pki/scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=/ups/app/kubernetes/cfg/scheduler.kubeconfig
  
kubectl config set-context system:kube-scheduler@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-scheduler \
  --kubeconfig=/ups/app/kubernetes/cfg/scheduler.kubeconfig
  
kubectl config use-context system:kube-scheduler@kubernetes \
  --kubeconfig=/ups/app/kubernetes/cfg/scheduler.kubeconfig

創建admin.kubeconfig 文件

kubeconfig 為 kubectl 的配置文件，包含訪問 apiserver 的所有信息，如 apiserver 地址、CA 證書和自身使用的證書

# 設置集群參數
kubectl config set-cluster kubernetes \
  --certificate-authority=/ups/app/kubernetes/pki/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.225:8443 \
  --kubeconfig=/ups/app/kubernetes/cfg/admin.kubeconfig

# 設置用戶項
kubectl config set-credentials kubernetes-admin \
  --client-certificate=/ups/app/kubernetes/pki/admin.pem \
  --client-key=/ups/app/kubernetes/pki/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=/ups/app/kubernetes/cfg/admin.kubeconfig

# 設置上下文參數  
kubectl config set-context kubernetes-admin@kubernetes \
  --cluster=kubernetes \
  --user=kubernetes-admin \
  --kubeconfig=/ups/app/kubernetes/cfg/admin.kubeconfig

# 設置默認上下文
kubectl config use-context kubernetes-admin@kubernetes \
  --kubeconfig=/ups/app/kubernetes/cfg/admin.kubeconfig

--certificate-authority：驗證 kube-apiserver 證書的根證書；
--client-certificate、--client-key：剛生成的 admin 證書和私鑰，與 kube-apiserver https 通信時使用；
--embed-certs=true：將 ca.pem 和 admin.pem 證書內容嵌入到生成的 kubectl.kubeconfig 文件中(否則，寫入的是證書文件路徑，后續拷貝 kubeconfig 到其它機器時，還需要單獨拷貝證書文件，不方便。)；
--server：指定 kube-apiserver 的地址。如果使用高可用集群環境（如：lb，keepalive+haproxy|nginx），使用VIP地址和端口(${CLUSTER_VIP}:8443)；否則使用第一個master 的IP地址和端口(${MASTER_IPS[0]}:6443)

創建ServiceAccount Key

openssl genrsa -out /ups/app/kubernetes/pki/sa.key 2048

openssl rsa -in /ups/app/kubernetes/pki/sa.key -pubout -out /ups/app/kubernetes/pki/sa.pub

分發文件

# 復制/ups/app/kubernetes/pki目錄下證書和私鑰文件到其他master節點上
for NODE in k8s-master02 k8s-master03; do
  for FILE in $(ls /ups/app/kubernetes/pki | grep -v etcd);do
    scp /ups/app/kubernetes/pki/${FILE} $NODE:/ups/app/kubernetes/pki/${FILE}
  done
  # 復制kubeconfig文件到其他master節點
  for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do
    scp /ups/app/kubernetes/cfg/${FILE} $NODE:/ups/app/kubernetes/cfg/${FILE};
  done
done

Kubernetes組件配置（Master)

k8s service網段為10.96.0.0/12，該網段不能和宿主機的網段、Pod網段的重復
Pod網段為 172.16.0.0/16

Master節點必備組件

kube-apiserver
kube-scheduler
kube-controller-manager

kube-apiserver、kube-scheduler 和 kube-controller-manager 均以多實例模式運行

kube-scheduler 和 kube-controller-manager 會自動選舉產生一個 leader 實例，其它實例處於阻塞模式，當 leader 掛了后，重新選舉產生新的 leader，從而保證服務可用性

注意：如果三台Master節點僅僅作為集群管理節點的話，那么則無需部署docker、kubelet、kube-proxy組件；但是如果后期要部署mertics-server、istio組件服務時會出現無法運行的情況，所以還是建議master節點也部署docker、kubelet、kube-proxy組件

kube-apiserver 組件配置

創建kube-apiserver service

所有Master節點創建kube-apiserver service

說明

--advertise-address：apiserver 對外通告的 IP（kubernetes 服務后端節點 IP）；

--default-*-toleration-seconds：設置節點異常相關的閾值；

--max-*-requests-inflight：請求相關的最大閾值；

--etcd-*：訪問 etcd 的證書和 etcd 服務器地址；

--bind-address： https 監聽的 IP，不能為 127.0.0.1，否則外界不能訪問它的安全端口 6443；

--secret-port：https 監聽端口；

--insecure-port=0：關閉監聽 http 非安全端口(8080)；

--tls-*-file：指定 apiserver 使用的證書、私鑰和 CA 文件；

--audit-*：配置審計策略和審計日志文件相關的參數；

--client-ca-file：驗證 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)請求所帶的證書；

--enable-bootstrap-token-auth：啟用 kubelet bootstrap 的 token 認證；

--requestheader-*：kube-apiserver 的 aggregator layer 相關的配置參數，proxy-client & HPA 需要使用；

--requestheader-client-ca-file：用於簽名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的證書；在啟用了 metric aggregator 時使用；

--requestheader-allowed-names：不能為空，值為逗號分割的 --proxy-client-cert-file 證書的 CN 名稱，這里設置為 "aggregator"；

--service-account-key-file：簽名 ServiceAccount Token 的公鑰文件，kube-controller-manager 的 --service-account-private-key-file 指定私鑰文件，兩者配對使用；

--runtime-config=api/all=true：啟用所有版本的 APIs，如 autoscaling/v2alpha1；

--authorization-mode=Node,RBAC、--anonymous-auth=false：開啟 Node 和 RBAC 授權模式，拒絕未授權的請求；

--enable-admission-plugins：啟用一些默認關閉的 plugins；

--allow-privileged：運行執行 privileged 權限的容器；

--apiserver-count=3：指定 apiserver 實例的數量；

--event-ttl：指定 events 的保存時間；

--kubelet-*：如果指定，則使用 https 訪問 kubelet APIs；需要為證書對應的用戶(上面 kubernetes*.pem 證書的用戶為 kubernetes) 用戶定義 RBAC 規則，否則訪問 kubelet API 時提示未授權；

--proxy-client-*：apiserver 訪問 metrics-server 使用的證書；

--service-cluster-ip-range：指定 Service Cluster IP 地址段；

--service-node-port-range：指定 NodePort 的端口范圍；

如果 kube-apiserver 機器沒有運行 kube-proxy，則還需要添加 --enable-aggregator-routing=true 參數；

關於 --requestheader-XXX 相關參數，參考：

https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md

https://docs.bitnami.com/kubernetes/how-to/configure-autoscaling-custom-metrics/

注意：

--requestheader-client-ca-file 指定的 CA 證書，必須具有 client auth and server auth；

如果 --requestheader-allowed-names 不為空,且 --proxy-client-cert-file 證書的 CN 名稱不在 allowed-names 中，則后續查看 node 或 pods 的 metrics 失敗，提示：
$ kubectl top nodes
Error from server (Forbidden): nodes.metrics.k8s.io is forbidden: User "aggregator" cannot list resource "nodes" in API group "metrics.k8s.io" at the cluster scope

mastet01

cat> /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/ups/app/kubernetes/bin/kube-apiserver \\
  --advertise-address=192.168.10.221 \\
  --allow-privileged=true  \\
  --authorization-mode=Node,RBAC  \\
  --bind-address=0.0.0.0  \\
  --client-ca-file=/ups/app/kubernetes/pki/ca.pem  \\
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
  --enable-bootstrap-token-auth=true  \\
  --enable-aggregator-routing=true \\
  --etcd-cafile=/ups/app/kubernetes/pki/etcd-ca.pem  \\
  --etcd-certfile=/ups/app/kubernetes/pki/etcd.pem  \\
  --etcd-keyfile=/ups/app/kubernetes/pki/etcd-key.pem  \\
  --etcd-servers=https://192.168.10.221:2379,https://192.168.10.222:2379,https://192.168.10.223:2379 \\
  --insecure-port=0  \\
  --kubelet-client-certificate=/ups/app/kubernetes/pki/apiserver.pem  \\
  --kubelet-client-key=/ups/app/kubernetes/pki/apiserver-key.pem  \\
  --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
  --logtostderr=false  \\
  --log-dir=/ups/app/kubernetes/log \\
  --proxy-client-cert-file=/ups/app/kubernetes/pki/front-proxy-client.pem  \\
  --proxy-client-key-file=/ups/app/kubernetes/pki/front-proxy-client-key.pem  \\
  --requestheader-allowed-names=aggregator  \\
  --requestheader-client-ca-file=/ups/app/kubernetes/pki/front-proxy-ca.pem  \\
  --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
  --requestheader-group-headers=X-Remote-Group  \\
  --requestheader-username-headers=X-Remote-User \\
  --secure-port=6443  \\
  --service-account-key-file=/ups/app/kubernetes/pki/sa.pub  \\
  --service-cluster-ip-range=10.96.0.0/12  \\
  --service-node-port-range=30000-32767  \\
  --tls-cert-file=/ups/app/kubernetes/pki/apiserver.pem  \\
  --tls-private-key-file=/ups/app/kubernetes/pki/apiserver-key.pem  \\
  --v=2
  # --token-auth-file=/ups/app/kubernetes/cfg/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

EOF

V1.20.X 中需新增以下2項參數項

--service-account-signing-key-file=/ups/app/kubernetes/pki/sa.key

--service-account-issuer=https://kubernetes.default.svc.cluster.local

master2

cat> /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/ups/app/kubernetes/bin/kube-apiserver \\
  --advertise-address=192.168.10.222 \\
  --allow-privileged=true  \\
  --authorization-mode=Node,RBAC  \\
  --bind-address=0.0.0.0  \\
  --client-ca-file=/ups/app/kubernetes/pki/ca.pem  \\
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
  --enable-bootstrap-token-auth=true  \\
  --enable-aggregator-routing=true \\
  --etcd-cafile=/ups/app/kubernetes/pki/etcd-ca.pem  \\
  --etcd-certfile=/ups/app/kubernetes/pki/etcd.pem  \\
  --etcd-keyfile=/ups/app/kubernetes/pki/etcd-key.pem  \\
  --etcd-servers=https://192.168.10.221:2379,https://192.168.10.222:2379,https://192.168.10.223:2379 \\
  --insecure-port=0  \\
  --kubelet-client-certificate=/ups/app/kubernetes/pki/apiserver.pem  \\
  --kubelet-client-key=/ups/app/kubernetes/pki/apiserver-key.pem  \\
  --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
  --logtostderr=false  \\
  --log-dir=/ups/app/kubernetes/log \\
  --proxy-client-cert-file=/ups/app/kubernetes/pki/front-proxy-client.pem  \\
  --proxy-client-key-file=/ups/app/kubernetes/pki/front-proxy-client-key.pem  \\
  --requestheader-allowed-names=aggregator  \\
  --requestheader-client-ca-file=/ups/app/kubernetes/pki/front-proxy-ca.pem  \\
  --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
  --requestheader-group-headers=X-Remote-Group  \\
  --requestheader-username-headers=X-Remote-User \\
  --secure-port=6443  \\
  --service-account-key-file=/ups/app/kubernetes/pki/sa.pub  \\
  --service-cluster-ip-range=10.96.0.0/12  \\
  --service-node-port-range=30000-32767  \\
  --tls-cert-file=/ups/app/kubernetes/pki/apiserver.pem  \\
  --tls-private-key-file=/ups/app/kubernetes/pki/apiserver-key.pem  \\
  --v=2
  # --token-auth-file=/ups/app/kubernetes/cfg/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

EOF

master3

cat> /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/ups/app/kubernetes/bin/kube-apiserver \\
  --advertise-address=192.168.10.223 \\
  --allow-privileged=true  \\
  --authorization-mode=Node,RBAC  \\
  --bind-address=0.0.0.0  \\
  --client-ca-file=/ups/app/kubernetes/pki/ca.pem  \\
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
  --enable-bootstrap-token-auth=true  \\
  --enable-aggregator-routing=true \\
  --etcd-cafile=/ups/app/kubernetes/pki/etcd-ca.pem  \\
  --etcd-certfile=/ups/app/kubernetes/pki/etcd.pem  \\
  --etcd-keyfile=/ups/app/kubernetes/pki/etcd-key.pem  \\
  --etcd-servers=https://192.168.10.221:2379,https://192.168.10.222:2379,https://192.168.10.223:2379 \\
  --insecure-port=0  \\
  --kubelet-client-certificate=/ups/app/kubernetes/pki/apiserver.pem  \\
  --kubelet-client-key=/ups/app/kubernetes/pki/apiserver-key.pem  \\
  --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
  --logtostderr=false  \\
  --log-dir=/ups/app/kubernetes/log \\
  --proxy-client-cert-file=/ups/app/kubernetes/pki/front-proxy-client.pem  \\
  --proxy-client-key-file=/ups/app/kubernetes/pki/front-proxy-client-key.pem  \\
  --requestheader-allowed-names=aggregator  \\
  --requestheader-client-ca-file=/ups/app/kubernetes/pki/front-proxy-ca.pem  \\
  --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
  --requestheader-group-headers=X-Remote-Group  \\
  --requestheader-username-headers=X-Remote-User \\
  --secure-port=6443  \\
  --service-account-key-file=/ups/app/kubernetes/pki/sa.pub  \\
  --service-cluster-ip-range=10.96.0.0/12  \\
  --service-node-port-range=30000-32767  \\
  --tls-cert-file=/ups/app/kubernetes/pki/apiserver.pem  \\
  --tls-private-key-file=/ups/app/kubernetes/pki/apiserver-key.pem  \\
  --v=2
  # --token-auth-file=/ups/app/kubernetes/cfg/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

EOF

啟動kube-apiserver服務

systemctl daemon-reload && systemctl enable --now kube-apiserver
systemctl status kube-apiserver -l

檢查服務

驗證

curl --insecure https://192.168.10.221:6443/
curl --insecure https://192.168.10.225:8443/

檢查集群狀態

$ kubectl cluster-info
Kubernetes master is running at https://172.27.138.251:6443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl get all --all-namespaces
NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.96.0.1   <none>        443/TCP   3m53s

$ kubectl get componentstatuses
NAME                 AGE
controller-manager   <unknown>
scheduler            <unknown>
etcd-0               <unknown>
etcd-2               <unknown>
etcd-1               <unknown>

$ kubectl get cs -o yaml

檢查 kube-apiserver 監聽的端口

netstat -lnpt|grep kube

查看集群組件狀態

kubectl cluster-info
kubectl get componentstatuses
kubectl get all --all-namespaces

kube-controller-manager組件配置

啟動后將通過競爭選舉機制產生一個 leader 節點，其它節點為阻塞狀態。當 leader 節點不可用時，阻塞的節點將再次進行選舉產生新的 leader 節點，從而保證服務的可用性

kube-controller-manager 在以下兩種情況下使用證書：

與 kube-apiserver 的安全端口通信;
在安全端口(https，10252) 輸出 prometheus 格式的 metrics

創建kube-controller-manager service

所有Master節點配置kube-controller-manager service

cat >  /usr/lib/systemd/system/kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target 
After=kube-apiserver.service
Requires=kube-apiserver.service

[Service]
ExecStart=/ups/app/kubernetes/bin/kube-controller-manager \\
  --address=0.0.0.0 \\
  --allocate-node-cidrs=true \\
  --cluster-cidr=172.16.0.0/16 \\
  --cluster-signing-cert-file=/ups/app/kubernetes/pki/ca.pem \\
  --cluster-signing-key-file=/ups/app/kubernetes/pki/ca-key.pem \\
  --controllers=*,bootstrapsigner,tokencleaner \\
  --kubeconfig=/ups/app/kubernetes/cfg/controller-manager.kubeconfig \\
  --leader-elect=true \\
  --logtostderr=false \\
  --log-dir=/ups/app/kubernetes/log \\
  --node-cidr-mask-size=24 \\
  --node-monitor-grace-period=40s \\
  --node-monitor-period=5s \\
  --pod-eviction-timeout=2m0s \\
  --requestheader-client-ca-file=/ups/app/kubernetes/pki/front-proxy-ca.pem \\
  --root-ca-file=/ups/app/kubernetes/pki/ca.pem \\
  --service-account-private-key-file=/ups/app/kubernetes/pki/sa.key \\
  --use-service-account-credentials=true \\
  --v=2
  # --cluster-signing-duration=876000h0m0s \\

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF

啟動kube-controller-manager 服務

所有Master節點啟動kube-controller-manager

systemctl daemon-reload 
systemctl enable --now kube-controller-manager
systemctl status kube-controller-manager -l

# 異常檢查原因
journalctl -u kube-apiserver

檢查

檢查服務監聽端口

kube-controller-manager 監聽 10252 端口，接收 http 請求
kube-controller-manager 監聽 10257 端口，接收 https 請求

netstat -lnpt | grep kube-control

查看輸出的 metrics

在 kube-controller-manager 節點上執行

curl -s --cacert /ups/app/kubernetes/pki/ca.pem \
  --cert /ups/app/kubernetes/pki/admin.pem \
  --key /ups/app/kubernetes/pki/admin-key.pem https://192.168.10.221:10252/metrics |head

檢查當前的leader

kubectl get endpoints kube-controller-manager --namespace=kube-system  -o yaml

kube-scheduler組件配置

kube-scheduler 在以下兩種情況下使用該證書：

與 kube-apiserver 的安全端口通信;
在安全端口(https，10251) 輸出 prometheus 格式的 metrics

創建kube-scheduler service

所有Master節點配置kube-scheduler service

cat > /usr/lib/systemd/system/kube-scheduler.service <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
After=kube-apiserver.service
Requires=kube-apiserver.service

[Service]
ExecStart=/ups/app/kubernetes/bin/kube-scheduler \\
  --address=0.0.0.0 \\
  --kubeconfig=/ups/app/kubernetes/cfg/scheduler.kubeconfig \\
  --leader-elect=true \\
  --logtostderr=false \\
  --log-dir=/ups/app/kubernetes/log \\
  --v=2
  # --secure-port=10259

Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF

--kubeconfig：指定 kubeconfig 文件路徑，kube-scheduler 使用它連接和驗證 kube-apiserver
--leader-elect=true：集群運行模式，啟用選舉功能；被選為 leader 的節點負責處理工作，其它節點為阻塞狀態

啟動kube-scheduler 服務

systemctl daemon-reload 
systemctl enable --now kube-scheduler
systemctl status kube-scheduler -l

# 異常時檢查
journalctl -u kube-scheduler

檢查

查看輸出的 metrics

在 kube-scheduler 節點上執行

kube-scheduler 監聽 10251 和 10259 端口：

10251：接收 http 請求，非安全端口，不需要認證授權
10259：接收 https 請求，安全端口，需要認證授權
兩個接口都對外提供 /metrics 和 /healthz 的訪問

 netstat -lnpt |grep kube-sched

curl -s http://192.168.10.221:10251/metrics |head

curl -s --cacert /ups/app/kubernetes/pki/ca.pem \
  --cert /ups/app/kubernetes/pki/admin.pem \
  --key /ups/app/kubernetes/pki/admin-key.pem \
  https://192.168.10.221:10259/metrics |head

查看當前的 leader

kubectl get endpoints kube-scheduler --namespace=kube-system  -o yaml

TLS Bootstrapping配置

在master1創建bootstrap

注意：如果不是高可用集群，192.168.10.225:8443改為master01的地址，8443改為apiserver的端口，默認是6443

創建bootstrap-kubelet.kubeconfig文件

cd /root/k8s-ha-install/bootstrap

kubectl config set-cluster kubernetes \
  --certificate-authority=/ups/app/kubernetes/pki/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.225:8443 \
  --kubeconfig=/ups/app/kubernetes/cfg/bootstrap-kubelet.kubeconfig

kubectl config set-credentials tls-bootstrap-token-user \
  --token=c8ad9c.2e4d610cf3e7426e \
  --kubeconfig=/ups/app/kubernetes/cfg/bootstrap-kubelet.kubeconfig

c8ad9c.2e4d610cf3e7426e是生成的隨機序列，可通過以下命令生成，把生成的token寫入bootstrap.secret.yaml文件中。建議自行修改
```
echo "$(head -c 6 /dev/urandom | md5sum | head -c 6)"."$(head -c 16 /dev/urandom | md5sum | head -c 16)"
c8ad9c.2e4d610cf3e7426e
```
- token 格式： [a-z0-9]{6}.[a-z0-9]
  - 第一部分是token_id, 它是一種公開信息，用於引用令牌並確保不會泄露認證所使用的秘密信息
  - 第二部分是“令牌秘密（Token Secret）”，它應該被共享給受信的第三方

kubectl config set-context tls-bootstrap-token-user@kubernetes \
  --cluster=kubernetes \
  --user=tls-bootstrap-token-user \
  --kubeconfig=/ups/app/kubernetes/cfg/bootstrap-kubelet.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes \
  --kubeconfig=/ups/app/kubernetes/cfg/bootstrap-kubelet.kubeconfig

配置kubectl登陸認證文件

mkdir -p /root/.kube
cp -i /ups/app/kubernetes/cfg/admin.kubeconfig /root/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

創建secret配置文件

若修改token，記住同時修改下面的文件並記錄后續使用

cat> bootstrap.secret.yaml<<-'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-c8ad9c
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: c8ad9c
  token-secret: 2e4d610cf3e7426e
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-certificate-rotation
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver

EOF

k8s集群創建資源

kubectl create -f bootstrap.secret.yaml

查看secret

kubectl get secret bootstrap-token-c8ad9c -n kube-system -oyaml

輸出token-id, token-secret都是經過bas464加密

base64解密查看token-id 和 token-secret

echo "YzhhZDLj" | base64 -d

echo "MmU0ZDYxMGNmM2U3NDI2ZQ==" |base64 -d

得到的結果與上面bootstrap.secret.yaml 文件內容中一致

配置Kubectl

kubectl 使用 https 協議與 kube-apiserver 進行安全通信，kube-apiserver 對 kubectl 請求包含的證書進行認證和授權。

kubectl 后續用於集群管理，所以這里創建具有最高權限的 admin 證書。

# 把master01節點上的admin.kubeconfig分發到其他節點
for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02; do
  ssh $NODE "mkdir -p $HOME/.kube"
  scp /ups/app/kubernetes/cfg/admin.kubeconfig $NODE:$HOME/.kube/config
  ssh $NODE "chmod 660 $HOME/.kube/config"
done

tab鍵命令補全

echo "source <(kubectl completion bash)" >> ~/.bash_profile
source ~/.bash_profile
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)

Worker|Node 節點配置

Worker 節點組件

containerd | docker
kubelet
kube-proxy
calico

同步證書和私鑰文件

把master1上的證書復制到Node節點

for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02; do
  ssh $NODE mkdir -p /ups/app/kubernetes/pki
  
  scp etcd-ca.pem etcd.pem etcd-key.pem ca.pem ca-key.pem front-proxy-ca.pem $NODE:/ups/app/kubernetes/pki/
  scp /ups/app/kubernetes/cfg/bootstrap-kubelet.kubeconfig $NODE:/ups/app/kubernetes/cfg/
done

kubelet組件配置

注意：從v1.19.X +版本開始，master節點建議都啟動kubelet服務，然后通過配置污點的方式讓master節點不運行Pod

kubelet 運行在每個 worker 節點上，接收 kube-apiserver 發送的請求，管理 Pod 容器，執行交互式命令，如 exec、run、logs 等。

kubelet 啟動時自動向 kube-apiserver 注冊節點信息，內置的 cadvisor 統計和監控節點的資源使用情況。

為確保安全，部署時關閉了 kubelet 的非安全 http 端口，對請求進行認證和授權，拒絕未授權的訪問(如 apiserver、heapster 的請求)

創建kubelet service

所有節點配置kubelet service（Master節點不部署Pod也可無需配置）

cat > /usr/lib/systemd/system/kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/ups/app/kubernetes/bin/kubelet \\
  --bootstrap-kubeconfig=/ups/app/kubernetes/cfg/bootstrap-kubelet.kubeconfig \\
  --cni-conf-dir=/etc/cni/net.d \\
  --cni-bin-dir=/opt/cni/bin \\
  --config=/ups/app/kubernetes/cfg/kubelet-conf.yml \\
  --image-pull-progress-deadline=30m \\
  --kubeconfig=/ups/app/kubernetes/cfg/kubelet.kubeconfig \\
  --logtostderr=false \\
  --log-dir=/ups/app/kubernetes/log \\
  --network-plugin=cni \\
  --node-labels=node.kubernetes.io/node='' \\
  --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2 \\
  --v=2

Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

注意：使用非docker作為容器運行時的情況，需要添加以下配置項

--container-runtime 參數為 remote，

設置 --container-runtime-endpoint 為對應的容器運行時的監聽地址

示例：使用containerd
--container-runtime=remote
--runtime-request-timeout=30m
--container-runtime-endpoint=unix:///run/containerd/containerd.sock

配置10-kubelet.conf

cat > /etc/systemd/system/kubelet.service.d/10-kubelet.conf <<-'EOF'
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/ups/app/kubernetes/cfg/bootstrap-kubelet.kubeconfig --kubeconfig=/ups/app/kubernetes/cfg/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/ups/app/kubernetes/cfg/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' --image-pull-progress-deadline=30m "
ExecStart=
ExecStart=/ups/app/kubernetes/bin/kubelet --logtostderr=false --log-dir=/ups/app/kubernetes/log $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS
EOF

bootstrap-kubeconfig：首次啟動時向apiserver申請證書
kubeconfig：通過bootstrap自動生成kubelet.kubeconfig文件，用於連接apiserver
pod-infra-container-image：管理Pod網絡容器的鏡像

配置kubelet-conf.yml

注意：

如果更改了k8s的service網段，需要更改kubelet-conf.yml 的clusterDNS:配置，改成k8s Service網段的第十個地址，比如10.96.0.10（k8s的service網段開始設置的是10.96.0.0/12）

cgroupDriver 改成 systemd，必須與 docker 配置文件 /etc/docker/daemon.json 中 "exec-opts": ["native.cgroupdriver=systemd"]配置一致。

cat > /ups/app/kubernetes/cfg/kubelet-conf.yml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /ups/app/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /ups/app/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF

kubeadm打印默認配置

kubeadm config print init-defaults --help

This command prints objects such as the default init configuration that is used for 'kubeadm init'.

Note that sensitive values like the Bootstrap Token fields are replaced with placeholder values like {"abcdef.0123456789abcdef" "" "nil" [] []} in order to pass validation but
not perform the real computation for creating a token.

Usage:
kubeadm config print init-defaults [flags]

Flags:
--component-configs strings A comma-separated list for component config API objects to print the default values for. Available values: [KubeProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed.
-h, --help help for init-defaults

Global Flags:
--add-dir-header If true, adds the file directory to the header of the log messages
--kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf")
--log-file string If non-empty, use this log file
--log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--one-output If true, only write logs to their native severity level (vs also writing to each lower severity level
--rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem.
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files
-v, --v Level number for the log level verbosity

kubeadm config print init-defaults --component-configs KubeletConfiguration

啟動kubelet

所有節點啟動kubelet

systemctl daemon-reload
systemctl enable --now kubelet

systemctl status kubelet -l

# 若啟動失敗，檢查原因
journalctl -xefu kubelet

此時系統日志/var/log/messages, 顯示只有如下信息為正常
Unable to update cni config: no networks found in /etc/cni/net.d

查看集群狀態

kubectl get node
kubectl get csr

Kube-Proxy組件配置

創建kube-proxy.kubeconfig文件（在master1節點上）

# 創建賬號
kubectl -n kube-system create serviceaccount kube-proxy
# 角色綁定
kubectl create clusterrolebinding system:kube-proxy \
  --clusterrole system:node-proxier \
  --serviceaccount kube-system:kube-proxy

SECRET=$(kubectl -n kube-system get sa/kube-proxy \
  --output=jsonpath='{.secrets[0].name}')
JWT_TOKEN=$(kubectl -n kube-system get secret/${SECRET} \
  --output=jsonpath='{.data.token}' | base64 -d)

kubectl config set-cluster kubernetes \
  --certificate-authority=/ups/app/kubernetes/pki/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.225:8443 \
  --kubeconfig=/ups/app/kubernetes/cfg/kube-proxy.kubeconfig

kubectl config set-credentials kubernetes \
  --token=${JWT_TOKEN} \
  --kubeconfig=/ups/app/kubernetes/cfg/kube-proxy.kubeconfig

kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=kubernetes \
  --kubeconfig=/ups/app/kubernetes/cfg/kube-proxy.kubeconfig

kubectl config use-context kubernetes \
  --kubeconfig=/ups/app/kubernetes/cfg/kube-proxy.kubeconfig

創建kube-proxy system unit文件

cat > /usr/lib/systemd/system/kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/ups/app/kubernetes/bin/kube-proxy \\
  --config=/ups/app/kubernetes/cfg/kube-proxy.yaml \\
  --logtostderr=false \\
  --log-dir=/ups/app/kubernetes/log \\
  --v=2

Restart=always
RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

創建配置文件

如果更改了集群Pod的網段，需要更改kube-proxy/kube-proxy.conf的clusterCIDR: 172.16.0.0/12參數為pod的網段。

cat > /ups/app/kubernetes/cfg/kube-proxy.conf <<EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /ups/app/kubernetes/cfg/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/16
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
EOF

分發配置文件

for NODE in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02; do
    scp /ups/app/kubernetes/cfg/kube-proxy.kubeconfig $NODE:/ups/app/kubernetes/cfg/kube-proxy.kubeconfig
	scp kube-proxy/kube-proxy.conf $NODE:/ups/app/kubernetes/cfg/kube-proxy.conf
	scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service
done

啟動kube-proxy

systemctl daemon-reload
systemctl enable --now kube-proxy

systemctl status kube-proxy -l

安裝插件

安裝calico網絡插件(master01)

Calico 是一款純 Layer 3 的數據中心網絡方案
kubernetes 要求集群內各節點(包括 master 節點)能通過 Pod 網段互聯互通。
calico 使用 IPIP 或 BGP 技術（默認為 IPIP）為各節點創建一個可以互通的 Pod 網絡。

安裝文檔

https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico-with-kubernetes-api-datastore-more-than-50-nodes

將calico安裝到etcd中

下載默認配置文件

# 最新版本
curl https://docs.projectcalico.org/manifests/calico-etcd.yaml -o calico-etcd.yaml

# 指定版本
curl https://docs.projectcalico.org/archive/v3.15/manifests/calico-etcd.yaml -O

curl https://docs.projectcalico.org/archive/v3.16/manifests/calico-etcd.yaml -o calico-etcd-v3.16.yaml

wget https://docs.projectcalico.org/v3.16.6/manifests/calico.yaml -O calico.yaml

默認yaml文件內容

---
# Source: calico/templates/calico-etcd-secrets.yaml
# The following contains k8s Secrets for use with a TLS enabled etcd cluster.
# For information on populating Secrets, see http://kubernetes.io/docs/user-guide/secrets/
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: calico-etcd-secrets
  namespace: kube-system
data:
  # Populate the following with etcd TLS configuration if desired, but leave blank if
  # not using TLS for etcd.
  # The keys below should be uncommented and the values populated with the base64
  # encoded contents of each file that would be associated with the TLS data.
  # Example command for encoding a file contents: cat <file> | base64 -w 0
  # etcd-key: null
  # etcd-cert: null
  # etcd-ca: null
---
# Source: calico/templates/calico-config.yaml
# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"
  # If you're using TLS enabled etcd uncomment the following.
  # You must also populate the Secret below with these files.
  etcd_ca: ""   # "/calico-secrets/etcd-ca"
  etcd_cert: "" # "/calico-secrets/etcd-cert"
  etcd_key: ""  # "/calico-secrets/etcd-key"
  # Typha is disabled.
  typha_service_name: "none"
  # Configure the backend to use.
  calico_backend: "bird"

  # Configure the MTU to use for workload interfaces and tunnels.
  # By default, MTU is auto-detected, and explicitly setting this field should not be required.
  # You can override auto-detection by providing a non-zero value.
  veth_mtu: "0"

  # The CNI network configuration to install on each node. The special
  # values in this config will be automatically populated.
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "log_file_path": "/var/log/calico/cni/cni.log",
          "etcd_endpoints": "__ETCD_ENDPOINTS__",
          "etcd_key_file": "__ETCD_KEY_FILE__",
          "etcd_cert_file": "__ETCD_CERT_FILE__",
          "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
          "mtu": __CNI_MTU__,
          "ipam": {
              "type": "calico-ipam"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        },
        {
          "type": "bandwidth",
          "capabilities": {"bandwidth": true}
        }
      ]
    }

---
# Source: calico/templates/calico-kube-controllers-rbac.yaml

# Include a clusterrole for the kube-controllers component,
# and bind it to the calico-kube-controllers serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  # Pods are monitored for changing labels.
  # The node controller monitors Kubernetes nodes.
  # Namespace and serviceaccount labels are used for policy.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
      - serviceaccounts
    verbs:
      - watch
      - list
      - get
  # Watch for changes to Kubernetes NetworkPolicies.
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system
---

---
# Source: calico/templates/calico-node-rbac.yaml
# Include a clusterrole for the calico-node DaemonSet,
# and bind it to the calico-node serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
  # The CNI plugin needs to get pods, nodes, and namespaces.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      # Used to discover service IPs for advertisement.
      - watch
      - list
  # Pod CIDR auto-detection on kubeadm needs access to config maps.
  - apiGroups: [""]
    resources:
      - configmaps
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      # Needed for clearing NodeNetworkUnavailable flag.
      - patch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: calico-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-node
subjects:
- kind: ServiceAccount
  name: calico-node
  namespace: kube-system

---
# Source: calico/templates/calico-node.yaml
# This manifest installs the calico-node container, as well
# as the CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: calico-node
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      hostNetwork: true
      tolerations:
        # Make sure calico-node gets scheduled on all nodes.
        - effect: NoSchedule
          operator: Exists
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      serviceAccountName: calico-node
      # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
      # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
      terminationGracePeriodSeconds: 0
      priorityClassName: system-node-critical
      initContainers:
        # This container installs the CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: docker.io/calico/cni:v3.18.1
          command: ["/opt/cni/bin/install"]
          envFrom:
          - configMapRef:
              # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
              name: kubernetes-services-endpoint
              optional: true
          env:
            # Name of the CNI config file to create.
            - name: CNI_CONF_NAME
              value: "10-calico.conflist"
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
            # The location of the etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # CNI MTU Config variable
            - name: CNI_MTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Prevents the container from sleeping forever.
            - name: SLEEP
              value: "false"
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
            - mountPath: /calico-secrets
              name: etcd-certs
          securityContext:
            privileged: true
        # Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes
        # to communicate with Felix over the Policy Sync API.
        - name: flexvol-driver
          image: docker.io/calico/pod2daemon-flexvol:v3.18.1
          volumeMounts:
          - name: flexvol-driver-host
            mountPath: /host/driver
          securityContext:
            privileged: true
      containers:
        # Runs calico-node container on each Kubernetes node. This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: docker.io/calico/node:v3.18.1
          envFrom:
          - configMapRef:
              # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
              name: kubernetes-services-endpoint
              optional: true
          env:
            # The location of the etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Location of the CA certificate for etcd.
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key
            # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            # Set noderef for node controller.
            - name: CALICO_K8S_NODE_REF
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Choose the backend to use.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Always"
            # Enable or Disable VXLAN on the default IP pool.
            - name: CALICO_IPV4POOL_VXLAN
              value: "Never"
            # Set MTU for tunnel device used if ipip is enabled
            - name: FELIX_IPINIPMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Set MTU for the VXLAN tunnel device.
            - name: FELIX_VXLANMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Set MTU for the Wireguard tunnel device.
            - name: FELIX_WIREGUARDMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            # - name: CALICO_IPV4POOL_CIDR
            #   value: "192.168.0.0/16"
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            # Set Felix logging to "info"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            - name: FELIX_HEALTHENABLED
              value: "true"
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 250m
          livenessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-live
              - -bird-live
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-ready
              - -bird-ready
            periodSeconds: 10
          volumeMounts:
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /run/xtables.lock
              name: xtables-lock
              readOnly: false
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
            - mountPath: /var/lib/calico
              name: var-lib-calico
              readOnly: false
            - mountPath: /calico-secrets
              name: etcd-certs
            - name: policysync
              mountPath: /var/run/nodeagent
            # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
            # parent directory.
            - name: sysfs
              mountPath: /sys/fs/
              # Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.
              # If the host is known to mount that filesystem already then Bidirectional can be omitted.
              mountPropagation: Bidirectional
            - name: cni-log-dir
              mountPath: /var/log/calico/cni
              readOnly: true
      volumes:
        # Used by calico-node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
        - name: sysfs
          hostPath:
            path: /sys/fs/
            type: DirectoryOrCreate
        # Used to install CNI.
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
        # Used to access CNI logs.
        - name: cni-log-dir
          hostPath:
            path: /var/log/calico/cni
        # Mount in the etcd TLS secrets with mode 400.
        # See https://kubernetes.io/docs/concepts/configuration/secret/
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets
            defaultMode: 0400
        # Used to create per-pod Unix Domain Sockets
        - name: policysync
          hostPath:
            type: DirectoryOrCreate
            path: /var/run/nodeagent
        # Used to install Flex Volume Driver
        - name: flexvol-driver-host
          hostPath:
            type: DirectoryOrCreate
            path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system

---
# Source: calico/templates/calico-kube-controllers.yaml
# See https://github.com/projectcalico/kube-controllers
apiVersion: apps/v1
kind: Deployment
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  # The controllers can only have a single active instance.
  replicas: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      serviceAccountName: calico-kube-controllers
      priorityClassName: system-cluster-critical
      # The controllers must run in the host network namespace so that
      # it isn't governed by policy that would prevent it from working.
      hostNetwork: true
      containers:
        - name: calico-kube-controllers
          image: docker.io/calico/kube-controllers:v3.18.1
          env:
            # The location of the etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Location of the CA certificate for etcd.
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key
            # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            # Choose which controllers to run.
            - name: ENABLED_CONTROLLERS
              value: policy,namespace,serviceaccount,workloadendpoint,node
          volumeMounts:
            # Mount in the etcd TLS secrets.
            - mountPath: /calico-secrets
              name: etcd-certs
          readinessProbe:
            exec:
              command:
              - /usr/bin/check-status
              - -r
      volumes:
        # Mount in the etcd TLS secrets with mode 400.
        # See https://kubernetes.io/docs/concepts/configuration/secret/
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets
            defaultMode: 0400

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system

---

# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers

---
# Source: calico/templates/calico-typha.yaml

---
# Source: calico/templates/configure-canal.yaml

---
# Source: calico/templates/kdd-crds.yaml

修改配置

sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.10.221:2379,https://192.168.10.222:2379,https://192.168.10.223:2379"#g' calico-etcd.yaml

ETCD_CA=`cat /ups/app/kubernetes/pki/etcd-ca.pem | base64 | tr -d '\n'`
ETCD_CERT=`cat /ups/app/kubernetes/pki/etcd.pem | base64 | tr -d '\n'`
ETCD_KEY=`cat /ups/app/kubernetes/pki/etcd-key.pem | base64 | tr -d '\n'`

sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml


sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml

# 更改此處為自己的pod網段
POD_SUBNET="172.16.0.0/16"

sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${POD_SUBNET}"'@g' calico-etcd.yaml

CALICO_IPV4POOL_IPIP 為 Never 使用 BGP 模式
- 它會以daemonset方式安裝在所有node主機，每台主機啟動一個bird(BGP client)，它會將calico網絡內的所有node分配的ip段告知集群內的主機，並通過本機的網卡eth0或者ens33轉發數據
cni插件默認安裝目錄

創建資源

kubectl apply -f calico-etcd.yaml

檢查Pod狀態

# 檢查Pod狀態
kubectl get pod -n kube-system
kubectl get po -n kube-system -owide
kubectl get pods -A

# 查看狀態信息
kubectl describe po calico-node-k7cff -n kube-system
# 查看容器日志
kubectl logs -f calico-node-k7cff -n kube-system

安裝CoreDNS

服務發現插件

下載默認配置文件

# 軟件地址
https://github.com/coredns/deployment/tree/master/kubernetes

wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed -O coredns.yaml.template

修改yaml文件
1. 將 kubernetes CLUSTER_DOMAIN REVERSE_CIDRS 改成 kubernetes cluster.local in-addr.arpa ip6.arpa
2. 將 forward . UPSTREAMNAMESERVER 改成 forward . /etc/resolv.conf
3. 將 clusterIP: CLUSTER_DNS_IP 改成 k8s service 網段的第10個IP 地址。例如：clusterIP：10.96.0.10（kubelet配置文件中的clusterDNS）

創建配置文件

如果更改了k8s service的網段需要將coredns的serviceIP改成k8s service網段的第十個IP

cat > coredns.yaml <<-'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  # replicas: not specified here:
  # 1. Default is 1.
  # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
           - weight: 100
             podAffinityTerm:
               labelSelector:
                 matchExpressions:
                   - key: k8s-app
                     operator: In
                     values: ["kube-dns"]
               topologyKey: kubernetes.io/hostname
      containers:
      - name: coredns
        image: registry.cn-beijing.aliyuncs.com/dotbalo/coredns:1.7.0
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.10
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP
EOF

如果更改了k8s service的網段需要將coredns的serviceIP改成k8s service網段的第十個IP。如：clusterIP: 10.96.0.10
```
sed -i "s#10.96.0.10#10.96.0.10#g" CoreDNS/coredns.yaml
```

集群中創建資源

kubectl apply -f coredns.yaml

檢查確認

# 查看狀態
kubectl get po -n kube-system -l k8s-app=kube-dns
kubectl logs -f coredns-6ccb5d565f-nsfq4 -n kube-system
kubectl get pods -n kube-system -o wide

安裝最新版本

git clone https://github.com/coredns/deployment.git

# 下載所需鏡像
for img in $(awk '/image:/{print $NF}' calico-etcd.yaml); do echo -e "pulling $img ------\n";docker pull $img; done

cd deployment/kubernetes
# 安裝或升級 coredns 版本
## deploy.sh通過提供-s選項，部署腳本將跳過ConfigMap從kube-dns到CoreDNS的轉換。
## -i 選項指定 k8s service 的網段的第10個IP地址
./deploy.sh -s -i 10.96.0.10 | kubectl apply -f -
./deploy.sh -i 10.96.0.10 -r "10.96.0.10/12" -s -t coredns.yaml.sed | kubectl apply -f -

# 安裝並替換 kube-dns
./deploy.sh | kubectl apply -f -
kubectl delete --namespace=kube-system deployment kube-dns

# 將 coredns 回滾到 kube-dns
./rollback.sh | kubectl apply -f -
kubectl delete --namespace=kube-system deployment coredns

# 查看狀態
kubectl get po -n kube-system -l k8s-app=kube-dns

升級最新版coredns

查看當前版本

 kubectl get pod -n kube-system coredns-867bfd96bd-f8ffj -oyaml|grep image
            f:image: {}
            f:imagePullPolicy: {}
    image: coredns/coredns:1.7.0
    imagePullPolicy: IfNotPresent
  - image: coredns/coredns:1.7.0
    imageID: ""

備份原來的cm、deploy、clusterrole、clusterrolebinding

[root@k8s-master01 ~]# kubectl get cm -n kube-system coredns -oyaml > coredns-config.yaml
[root@k8s-master01 ~]# kubectl get deploy -n kube-system coredns -oyaml > coredns-controllers.yaml
[root@k8s-master01 ~]# kubectl get clusterrole system:coredns -oyaml > coredns-clusterrole.yaml
[root@k8s-master01 ~]# kubectl get clusterrolebinding  system:coredns -oyaml > coredns-clusterrolebinding.yaml

升級

git clone https://github.com/coredns/deployment.git

cd deployment/kubernetes/
./deploy.sh -s | kubectl apply -f -

檢查確認

kubectl get pod -n kube-system coredns-867bfd96bd-f8ffj -oyaml|grep image

安裝Metrics Server

在新版的Kubernetes中系統資源的采集均使用Metrics-server，可以通過Metrics采集節點和Pod的內存、磁盤、CPU和網絡的使用率

注意：

當使用非默認證書文件的路徑/etc/kubernetes/pki,需要更新 metrics-server-v0.4.1.yaml 文件為自己的路徑如下：

拉取代碼

https://github.com/kubernetes-sigs/metrics-server/releases

在線安裝

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml

components.yaml 文件內容

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        image: k8s.gcr.io/metrics-server/metrics-server:v0.4.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          periodSeconds: 10
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

利用自定義配置文件創建資源

cd k8s-ha-install/metrics-server-0.4.x/
kubectl apply -f comp.yaml

kubectl get pod -n kube-system -l k8s-app=metrics-server
kubectl get pod -n kube-system

注意：comp.yaml配置內容

檢查確認

# 等待metrics server啟動然后查看狀態
kubectl  top node

kubectl get pod -n kube-system -l k8s-app=metrics-server

查看輸出metric

# 查看 metrics-server 輸出的 metrics
kubectl get --raw https://192.168.10.221:6443/apis/metrics.k8s.io/v1beta1/nodes | jq .
kubectl get --raw https://192.168.10.221:6443/apis/metrics.k8s.io/v1beta1/pods | jq .
kubectl get --raw https://192.168.10.221:6443/apis/metrics.k8s.io/v1beta1/nodes/<node-name> | jq .
kubectl get --raw https://192.168.10.221:6443/apis/metrics.k8s.io/v1beta1/namespace/<namespace-name>/pods/<pod-name> | jq .

metric api 接口

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes |jq

安裝Dashboard

Dashboard用於展示集群中的各類資源，同時也可以通過Dashboard實時查看Pod的日志和在容器中執行一些命令等

cd dashboard/
kubectl apply -f .

安裝最新版

# 官方GitHub地址：https://github.com/kubernetes/dashboard
# 可以在官方dashboard查看到最新版dashboard
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml -O dashboard.yaml

curl https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml -o kube-dashboard-v2.2.0.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml

# 創建管理員用戶vim admin.yaml
cat > dashboard-admin.yaml <<-'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding 
metadata: 
  name: admin-user
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
EOF
# 安裝
kubectl apply -f dashboard-admin.yaml -n kube-system

登錄

# 更改dashboard的svc為NodePort (將ClusterIP更改為NodePort)
kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard

# 查看端口號
kubectl get svc kubernetes-dashboard -n kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.96.77.112   <none>        443:30902/TCP   4m10s

# 根據上面得到的實例端口號，通過任意安裝了kube-proxy的宿主機或者VIP的IP+端口即可訪問到dashboard
如瀏覽器中打開web訪問：https://192.168.10.225:30902/ 並使用token登錄

#### 瀏覽器問題
谷歌瀏覽器（Chrome）啟動文件中加入啟動參數，用於解決無法訪問Dashboard的問題
--test-type --ignore-certificate-errors

# 獲取token
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

用token的 kubeconfig文件登陸 dashboard

# 創建登陸token
kubectl create sa dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

ADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}')
DASHBOARD_LOGIN_TOKEN=$(kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}')

# 設置集群參數
kubectl config set-cluster kubernetes \
  --certificate-authority=/ups/app/kubernetes/pki/ca.pem \
  --embed-certs=true \
  --server=https://192.168.10.225:8443 \
  --kubeconfig=dashboard.kubeconfig
 
# 設置客戶端認證參數，使用上面創建的 Token
kubectl config set-credentials dashboard_user \
  --token=${DASHBOARD_LOGIN_TOKEN} \
  --kubeconfig=dashboard.kubeconfig
 
# 設置上下文參數
kubectl config set-context default \
  --cluster=kubernetes \
  --user=dashboard_user \
  --kubeconfig=dashboard.kubeconfig
 
# 設置默認上下文
kubectl config use-context default --kubeconfig=dashboard.kubeconfig

安裝 kube-prometheus

項目地址

軟件下載

git clone https://github.com/coreos/kube-prometheus.git

安裝配置

cd kube-prometheus/
find . -name "*.yaml" -exec grep 'image: ' {} \;|awk '{print $NF}'|sort|uniq
find . -name "*.yaml" -exec grep 'quay.io' {} \;|awk '{print $NF}'|sort|uniq

# 使用中科大的 Registry
 sed -i -e 's#quay.io#quay.mirrors.ustc.edu.cn#g' manifests/*.yaml manifests/setup/*.yaml    
 
# 安裝 prometheus-operator
kubectl apply -f manifests/setup   
 
# 安裝 promethes metric adapter
kubectl apply -f manifests/

檢查運行狀態

kubectl get pods -n monitoring

安裝traefik

服務暴露用插件

創建命名空間

kubectl create ns ingress-traefik

創建CRD 資源

在 traefik v2.0 版本后，開始使用 CRD（Custom Resource Definition）來完成路由配置等，所以需要提前創建 CRD 資源

cat > traefik-crd.yaml <<-'EOF'
## IngressRoute
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
---
## IngressRouteTCP
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
---
## Middleware
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
---
## TraefikService
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
 
---
## TraefikTLSStore
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore
 
---
## IngressRouteUDP
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.us 
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
singular: ingressrouteudp
EOF

# 創建資源
kubectl apply -f traefik-crd.yaml
# 查看crd資源
kubectl get crd | grep traefik

安裝helm

Helm 是 Kubernetes 的包管理器。使用Helm 能夠從Chart repository（Helm應用倉庫）快速查找、下載安裝軟件包並通過與K8s API Server交互構建應用。

架構圖

組成

Charts: Helm使用的打包格式，一個Chart包含了一組K8s資源集合的描述文件。Chart有特定的文件目錄結構，如果開發者想自定義一個新的 Chart，只需要使用Helm create命令生成一個目錄結構即可進行開發。
Release: 通過Helm將Chart部署到 K8s集群時創建的特定實例，包含了部署在容器集群內的各種應用資源。
Tiller: Helm 2.x版本中，Helm采用Client/Server的設計，Tiller就是Helm的Server部分，需要具備集群管理員權限才能安裝到K8s集群中運行。Tiller與Helm client進行交互，接收client的請求，再與K8s API Server通信，根據傳遞的Charts來生成Release。而在最新的Helm 3.x中，據說是為了安全性考慮移除了Tiller。
Chart Repository: Helm Chart包倉庫，提供了很多應用的Chart包供用戶下載使用，官方倉庫的地址是https://hub.helm.sh。

Helm的任務是在倉庫中查找需要的Chart，然后將Chart以Release的形式安裝到K8S集群中。

下載地址

https://github.com/helm/helm/releases/tag/v3.5.3

https://get.helm.sh/helm-v3.5.3-linux-amd64.tar.gz

配置

#  helm
wget https://get.helm.sh/helm-v3.4.1-linux-amd64.tar.gz
tar -zxvf helm-v3.4.1-linux-amd64.tar.gz
cd linux-amd64/
cp helm /usr/local/bin
chmod a+x /usr/local/bin/helm

# 安裝 Tiller
helm init --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.16.6  --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

通過 kubectl get po -n kube-system 來查看 tiller 的安裝情況

k8s集群驗證

Pod必須能解析Service
Pod必須能解解析跨namespace的Service
每個節點都必須能訪問集群的kubernetes的SVC(443端口)和kube-dns的svc(53端口)
Pod於pod之間要能通信：
- 同namespace間
- 跨namespace間
- 跨機器能通信

創建一個busybox的pod

cat> busybox.yaml<<EOF
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
  - sleep
  - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF

kubectl apply -f busybox.yaml

首先查看po是否安裝成功

kubectl get po

查看svc是否正常

kubectl get svc

查看Pod是否能能解析Service

# 查看Pod是否能能解析Service
kubectl exec  busybox -n default -- nslookup kubernetes

# 查看Pod是否能解析跨namespace的Service
kubectl exec  busybox -n default -- nslookup kube-dns.kube-system

設置污點

污點(Taint)的組成

key=value:effect

每個污點有一個key和value作為污點的標簽，其中value可以為空，effect描述污點的作用。當前taint effect支持如下三個選項：

NoSchedule：表示k8s將不會將Pod調度到具有該污點的Node上
PreferNoSchedule：表示k8s將盡量避免將Pod調度到具有該污點的Node上
NoExecute：表示k8s將不會將Pod調度到具有該污點的Node上，同時會將Node上已經存在的Pod驅逐出去

污點設置和去除

# 設置污點
kubectl taint nodes node1 key1=value1:NoSchedule

# 去除污點
kubectl taint nodes node1 key1:NoSchedule-

示例

Kubernetes集群中總共有3個master節點，節點的名稱分別為k8s-master01、k8s-master02、k8s-master03。為了保證集群的穩定性，同時提高master節點的利用率，我們將其中一個節點設置為node-role.kubernetes.io/master:NoSchedule，另外兩個節點設置為node-role.kubernetes.io/master:PreferNoSchedule，這樣保證3個節點中的1個無論在任何情況下都將不運行業務Pod，而另外2個載集群資源充足的情況下盡量不運行業務Pod

kubectl taint nodes m01 node-role.kubernetes.io/master=:NoSchedule

kubectl taint nodes m02 node-role.kubernetes.io/master=:PreferNoSchedule

kubectl taint nodes m03 node-role.kubernetes.io/master=:PreferNoSchedule

問題

calico-kube-controllers啟動失敗

錯誤信息

kubectl logs calico-kube-controllers-8599495c57-bnqgp -n kube-system
# -- 輸出日志
[FATAL][1] main.go 105: Failed to start error=failed to build Calico client: could not initialize etcdv3 client: open /calico-secrets/etcd-cert: permission denied

處理

參考文檔

修改配置文件

# 修改 calico-etcd.yaml 中
defaultMode: 0400 修改成 defaultMode: 0040

重新應用資源

kubectl apply -f calico-etcd.yaml

檢查確認

kubectl get po -A -owide

附錄

簽名證書

https://guisu.blog.csdn.net/article/details/95459273

基礎概念

CA(Certification Authority)證書，指的是權威機構給我們頒發的證書。

密鑰就是用來加解密用的文件或者字符串。密鑰在非對稱加密的領域里，指的是私鑰和公鑰，他們總是成對出現，其主要作用是加密和解密。常用的加密強度是2048bit。

RSA即非對稱加密算法。非對稱加密有兩個不一樣的密碼，一個叫私鑰，另一個叫公鑰，用其中一個加密的數據只能用另一個密碼解開，用自己的都解不了，也就是說用公鑰加密的數據只能由私鑰解開。

證書的編碼格式

PEM(Privacy Enhanced Mail)，通常用於數字證書認證機構（Certificate Authorities，CA），擴展名為.pem, .crt, .cer, 和 .key。內容為Base64編碼的ASCII碼文件，有類似"-----BEGIN CERTIFICATE-----" 和 "-----END CERTIFICATE-----"的頭尾標記。服務器認證證書，中級認證證書和私鑰都可以儲存為PEM格式（認證證書其實就是公鑰）。Apache和nginx等類似的服務器使用PEM格式證書。

DER(Distinguished Encoding Rules)，與PEM不同之處在於其使用二進制而不是Base64編碼的ASCII。擴展名為.der，但也經常使用.cer用作擴展名，所有類型的認證證書和私鑰都可以存儲為DER格式。Java使其典型使用平台。

證書簽名請求CSR

CSR(Certificate Signing Request)，它是向CA機構申請數字×××書時使用的請求文件。在生成請求文件前，我們需要准備一對對稱密鑰。私鑰信息自己保存，請求中會附上公鑰信息以及國家，城市，域名，Email等信息，CSR中還會附上簽名信息。當我們准備好CSR文件后就可以提交給CA機構，等待他們給我們簽名，簽好名后我們會收到crt文件，即證書。

注意：CSR並不是證書。而是向權威證書頒發機構獲得簽名證書的申請。

把CSR交給權威證書頒發機構,權威證書頒發機構對此進行簽名,完成。保留好CSR,當權威證書頒發機構頒發的證書過期的時候,你還可以用同樣的CSR來申請新的證書,key保持不變.

數字簽名

數字簽名就是"非對稱加密+摘要算法"，其目的不是為了加密，而是用來防止他人篡改數據。

其核心思想是：比如A要給B發送數據，A先用摘要算法得到數據的指紋，然后用A的私鑰加密指紋，加密后的指紋就是A的簽名，B收到數據和A的簽名后，也用同樣的摘要算法計算指紋，然后用A公開的公鑰解密簽名，比較兩個指紋，如果相同，說明數據沒有被篡改，確實是A發過來的數據。假設C想改A發給B的數據來欺騙B，因為篡改數據后指紋會變，要想跟A的簽名里面的指紋一致，就得改簽名，但由於沒有A的私鑰，所以改不了，如果C用自己的私鑰生成一個新的簽名，B收到數據后用A的公鑰根本就解不開。

常用的摘要算法有MD5、SHA1、SHA256。

使用私鑰對需要傳輸的文本的摘要進行加密，得到的密文即被稱為該次傳輸過程的簽名。

數字證書和公鑰

數字證書則是由證書認證機構（CA）對證書申請者真實身份驗證之后，用CA的根證書對申請人的一些基本信息以及申請人的公鑰進行簽名（相當於加蓋發證書機構的公章）后形成的一個數字文件。實際上，數字證書就是經過CA認證過的公鑰，除了公鑰，還有其他的信息，比如Email，國家，城市，域名等。

證書類型分類

client certificate：用於服務端認證客戶端（例如etcdctl、etcd proxy、fleetctl、docker 客戶端等等）
server certificate: 服務端使用，客戶端以此驗證服務端身份（例如 docker服務端、kube-apiserver 等等）
peer certificate: 雙向證書，用於etcd 集群成員間通信

證書分類

服務器證書 :server cert
客戶端證書 : client cert
對等證書 : peer cert (表示既是server cert又是client cert)

在kubernetes 集群中需要的證書

etcd 節點需要標識自己服務的server cert，也需要client cert與etcd集群其他節點交互，當然可以分別指定2個證書，也可以使用一個對等證書
master 節點需要標識 apiserver服務的server cert，也需要client cert連接etcd集群，這里也使用一個對等證書
kubectl calico kube-proxy 只需要client cert，因此證書請求中 hosts 字段可以為空
kubelet證書比較特殊，(自動生成) 它由node節點TLS BootStrap向apiserver請求，由master節點的controller-manager 自動簽發，包含一個client cert 和一個server cert

工具使用方法

生成證書和私鑰

# cfssl gencert --help
	cfssl gencert -- generate a new key and signed certificate

Usage of gencert:
    Generate a new key and cert from CSR:
        cfssl gencert -initca CSRJSON
        cfssl gencert -ca cert -ca-key key [-config config] [-profile profile] [-hostname hostname] CSRJSON
        cfssl gencert -remote remote_host [-config config] [-profile profile] [-label label] [-hostname hostname] CSRJSON

    Re-generate a CA cert with the CA key and CSR:
        cfssl gencert -initca -ca-key key CSRJSON

    Re-generate a CA cert with the CA key and certificate:
        cfssl gencert -renewca -ca cert -ca-key key

Arguments:
        CSRJSON:    JSON file containing the request, use '-' for reading JSON from stdin

Flags:
  -initca=false: initialise new CA
  -remote="": remote CFSSL server
  -ca="": CA used to sign the new certificate
  -ca-key="": CA private key
  -config="": path to configuration file
  -hostname="": Hostname for the cert, could be a comma-separated hostname list
  -profile="": signing profile to use
  -label="": key label to use in remote CFSSL server

cfssljson

從cfssl和multirootca程序獲取JSON輸出，並將證書，密鑰，CSR和捆綁寫入文件

# cfssljson --help
Usage of cfssljson:
  -bare
    	the response from CFSSL is not wrapped in the API standard response
  -f string
    	JSON input (default "-")
  -stdout
    	output the response instead of saving to a file

創建證書所需的配置文件

CA配置文件 (ca-config.json)

從模板文件中生成 ca-config.json 文件

## cfssl print-defaults config > ca-config.json
# cfssl print-defaults config
{
    "signing": {
        "default": {
            "expiry": "168h"
        },
        "profiles": {
            "www": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            }
        }
    }
}

修改默認json文件，適用特定場景的配置

cat > ca-config.json <<-'EOF'
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

上面配置了一個default默認的配置，和一個kubernetes profiles，profiles可以設置多個profile

字段說明：

default默認策略，指定了證書的默認有效期是一年(876000h)
kubernetes：表示該配置(profile)的用途是為kubernetes生成證書及相關的校驗工作
- signing：表示該證書可用於簽名其它證書；生成的 ca.pem 證書中 CA=TRUE
- server auth：表示可以該CA 對 server 提供的證書進行驗證
- client auth：表示可以用該 CA 對 client 提供的證書進行驗證
expiry：也表示過期時間，如果不寫以default中的為准

CA證書簽名請求 (ca-csr.json)

用於生成CA證書和私鑰(root 證書和私鑰)

從模板文件生成 CSR

## cfssl print-defaults csr > csr.json
# cfssl print-defaults csr
{
    "CN": "example.net",
    "hosts": [
        "example.net",
        "www.example.net"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}

將默認csr.json 文件修改成適合指定場景環境

cat > etcd-ca-csr.json <<-'EOF'
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

參數字段說明

CN: Common Name，瀏覽器使用該字段驗證網站是否合法，一般寫的是域名。非常重要。瀏覽器使用該字段驗證網站是否合法
key：生成證書的算法
hosts：表示哪些主機名(域名)或者IP可以使用此csr申請的證書，為空或者""表示所有的都可以使用(上面這個沒有hosts字段)
names：一些其它的屬性
- C: Country，國家
- ST: State，州或者是省份
- L: Locality Name，地區，城市
- O: Organization Name，組織名稱，公司名稱(在k8s中常用於指定Group，進行RBAC綁定)
- OU: Organization Unit Name，組織單位名稱，公司部門

客戶端證書請求文件（client-crs.json）

cat > etcd-csr.json <<-'EOF'
{
  "CN": "etcd",
  "hosts": [
      "127.0.0.1",
      "192.168.10.221",
      "192.168.10.222",
      "192.168.10.223",
      "192.168.10.224",
      "192.168.10.225",
      "192.168.10.226",
      "k8s01",
      "k8s02",
      "k8s03",
      "k8s04",
      "k8s05",
      "k8s06"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

創建證書文件

ca-key.pem（私鑰）和ca.pem（證書），還會生成ca.csr（證書簽名請求），用於交叉簽名或重新簽名。

創建簽名證書或私鑰

# 生成 CA 證書和私鑰
## 初始化
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca
## 使用現有私鑰, 重新生成
cfssl gencert -initca -ca-key etcd-ca-key.pem etcd-ca-csr.json | cfssljson -bare etcd-ca

# 生成私鑰證書
cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

gencert: 生成新的key(密鑰)和簽名證書

-initca：初始化一個新ca
-ca：指明ca的證書
-ca-key：指明ca的私鑰文件
-config：指明請求證書的json文件
-profile：與-config中的profile對應，是指根據config中的profile段來生成證書的相關信息

查看證書信息

查看cert(證書信息)

cfssl certinfo -cert ca.pem

查看CSR(證書簽名請求)信息

cfssl certinfo -csr etcd-ca.csr

etcd配置方式

環境變量方式

cat > /ups/app/etcd/cfg/etcd.conf.sample <<-'EOF'
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/data/etcd/data/"
ETCD_WAL_DIR="/data/etcd/wal/"
ETCD_MAX_WALS="5"
ETCD_LISTEN_PEER_URLS="https://192.168.10.221:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.221:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.221:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.221:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.10.221:2380,etcd02=https://192.168.10.222:2380,etcd03=https://192.168.10.223:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/ups/app/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/ups/app/etcd/pki/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/ups/app/etcd/pki/etcd-ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/ups/app/etcd/pki/etcd.pem"
ETCD_PEER_KEY_FILE="/ups/app/etcd/pki/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/ups/app/etcd/pki/etcd-ca.pem"
ETCD_PEER_AUTO_TLS="true"

#[Log]
ETCD_LOGGER="zap"
ETCD_LOG_OUTPUTS="stderr"
ETCD_LOG_LEVEL="error"
EOF

yml 配置文件

cat > etcd.conf.yml.sample <<-'EOF'
# This is the configuration file for the etcd server.

# Human-readable name for this member.
name: 'default'

# Path to the data directory.
data-dir:

# Path to the dedicated wal directory.
wal-dir:

# Number of committed transactions to trigger a snapshot to disk.
snapshot-count: 10000

# Time (in milliseconds) of a heartbeat interval.
heartbeat-interval: 100

# Time (in milliseconds) for an election to timeout.
election-timeout: 1000

# Raise alarms when backend size exceeds the given quota. 0 means use the
# default quota.
quota-backend-bytes: 0

# List of comma separated URLs to listen on for peer traffic.
listen-peer-urls: http://localhost:2380

# List of comma separated URLs to listen on for client traffic.
listen-client-urls: http://localhost:2379

# Maximum number of snapshot files to retain (0 is unlimited).
max-snapshots: 5

# Maximum number of wal files to retain (0 is unlimited).
max-wals: 5

# Comma-separated white list of origins for CORS (cross-origin resource sharing).
cors:

# List of this member's peer URLs to advertise to the rest of the cluster.
# The URLs needed to be a comma-separated list.
initial-advertise-peer-urls: http://localhost:2380

# List of this member's client URLs to advertise to the public.
# The URLs needed to be a comma-separated list.
advertise-client-urls: http://localhost:2379

# Discovery URL used to bootstrap the cluster.
discovery:

# Valid values include 'exit', 'proxy'
discovery-fallback: 'proxy'

# HTTP proxy to use for traffic to discovery service.
discovery-proxy:

# DNS domain used to bootstrap initial cluster.
discovery-srv:

# Initial cluster configuration for bootstrapping.
initial-cluster:

# Initial cluster token for the etcd cluster during bootstrap.
initial-cluster-token: 'etcd-cluster'

# Initial cluster state ('new' or 'existing').
initial-cluster-state: 'new'

# Reject reconfiguration requests that would cause quorum loss.
strict-reconfig-check: false

# Accept etcd V2 client requests
enable-v2: true

# Enable runtime profiling data via HTTP server
enable-pprof: true

# Valid values include 'on', 'readonly', 'off'
proxy: 'off'

# Time (in milliseconds) an endpoint will be held in a failed state.
proxy-failure-wait: 5000

# Time (in milliseconds) of the endpoints refresh interval.
proxy-refresh-interval: 30000

# Time (in milliseconds) for a dial to timeout.
proxy-dial-timeout: 1000

# Time (in milliseconds) for a write to timeout.
proxy-write-timeout: 5000

# Time (in milliseconds) for a read to timeout.
proxy-read-timeout: 0

client-transport-security:
  # Path to the client server TLS cert file.
  cert-file:

  # Path to the client server TLS key file.
  key-file:

  # Enable client cert authentication.
  client-cert-auth: false

  # Path to the client server TLS trusted CA cert file.
  trusted-ca-file:

  # Client TLS using generated certificates
  auto-tls: false

peer-transport-security:
  # Path to the peer server TLS cert file.
  cert-file:

  # Path to the peer server TLS key file.
  key-file:

  # Enable peer client cert authentication.
  client-cert-auth: false

  # Path to the peer server TLS trusted CA cert file.
  trusted-ca-file:

  # Peer TLS using generated certificates.
  auto-tls: false

# Enable debug-level logging for etcd.
debug: false

logger: zap

# Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd.
log-outputs: [stderr]

# Force to create a new one member cluster.
force-new-cluster: false

auto-compaction-mode: periodic
auto-compaction-retention: "1"
EOF

## 過濾空行或注釋行 
grep -Ev "^[ \t]*(#|$)" etcd.conf.yml.sample > etcd.conf.yml

系統內核相關

系統內核相關參數參考：https://docs.openshift.com/enterprise/3.2/admin_guide/overcommit.html

3.10.x 內核 kmem bugs 相關的討論和解決辦法：
https://github.com/kubernetes/kubernetes/issues/61937
https://support.mesosphere.com/s/article/Critical-Issue-KMEM-MSPH-2018-0006
https://pingcap.com/blog/try-to-fix-two-linux-kernel-bugs-while-testing-tidb-operator-in-k8s/

kubelete認證

關於 controller 權限和 use-service-account-credentials 參數：https://github.com/kubernetes/kubernetes/issues/48208

kubelet 認證和授權：https://kubernetes.io/docs/admin/kubelet-authentication-authorization/#kubelet-authorization

kubectl 命令行工具

kubectl 是 kubernetes 集群的命令行管理工具，它默認從 ~/.kube/config 文件讀取 kube-apiserver 地址、證書、用戶名等信息。需要 admin 證書權限對集群進行管理。

證書

需要證書


Default CN	Parent CA	O (in Subject)	kind	hosts (SAN)
kube-etcd	etcd-ca		server, client	`localhost`, `127.0.0.1`
kube-etcd-peer	etcd-ca		server, client	`,` , `localhost`, `127.0.0.1`
kube-etcd-healthcheck-client	etcd-ca		client
kube-apiserver-etcd-client	etcd-ca	system:masters	client
kube-apiserver	kubernetes-ca		server	`,` , ``, `[1]`
kube-apiserver-kubelet-client	kubernetes-ca	system:masters	client
front-proxy-client	kubernetes-front-proxy-ca		client

證書路徑

Default CN	recommended key path	recommended cert path	command	key argument	cert argument
etcd-ca	etcd/ca.key	etcd/ca.crt	kube-apiserver		--etcd-cafile
kube-apiserver-etcd-client	apiserver-etcd-client.key	apiserver-etcd-client.crt	kube-apiserver	--etcd-keyfile	--etcd-certfile
kubernetes-ca	ca.key	ca.crt	kube-apiserver		--client-ca-file
kubernetes-ca	ca.key	ca.crt	kube-controller-manager	--cluster-signing-key-file	--client-ca-file, --root-ca-file, --cluster-signing-cert-file
kube-apiserver	apiserver.key	apiserver.crt	kube-apiserver	--tls-private-key-file	--tls-cert-file
kube-apiserver-kubelet-client	apiserver-kubelet-client.key	apiserver-kubelet-client.crt	kube-apiserver	--kubelet-client-key	--kubelet-client-certificate
front-proxy-ca	front-proxy-ca.key	front-proxy-ca.crt	kube-apiserver		--requestheader-client-ca-file
front-proxy-ca	front-proxy-ca.key	front-proxy-ca.crt	kube-controller-manager		--requestheader-client-ca-file
front-proxy-client	front-proxy-client.key	front-proxy-client.crt	kube-apiserver	--proxy-client-key-file	--proxy-client-cert-file
etcd-ca	etcd/ca.key	etcd/ca.crt	etcd		--trusted-ca-file, --peer-trusted-ca-file
kube-etcd	etcd/server.key	etcd/server.crt	etcd	--key-file	--cert-file
kube-etcd-peer	etcd/peer.key	etcd/peer.crt	etcd	--peer-key-file	--peer-cert-file
etcd-ca		etcd/ca.crt	etcdctl		--cacert
kube-etcd-healthcheck-client	etcd/healthcheck-client.key	etcd/healthcheck-client.crt	etcdctl	--key	--cert

service account key pair

private key path	public key path	command	argument
sa.key		kube-controller-manager	--service-account-private-key-file
	sa.pub	kube-apiserver	--service-account-key-file

證書類型說明

證書名稱	配置文件	用途
ca.pem	ca-csr.json	ca根證書
kube-proxy.pem	ca-config.json kube-proxy-csr.json	kube-proxy使用的證書
admin.pem	admin-csr.json ca-config.json	kubectl 使用的證書
kubernetes.pem	ca-config.json kubernetes-csr.json	apiserver使用的證書

使用證書的組件如下

組件	證書
kube-apiserver	ca.pem、kubernetes-key.pem、kubernetes.pem
kube-controller-manager	ca-key.pem、ca.pem
kubelet	ca.pem
kube-proxy	ca.pem、kube-proxy-key.pem、kube-proxy.pem
kubectl	ca.pem、admin-key.pem、admin.pem

etcd證書：

peer.pem、peer-key.pem：etcd各節點相互通信的對等證書及私鑰（hosts指定所有etcd節點IP）
server.pem、server-key.pem：etcd各節點自己的服務器證書及私鑰（hosts指定當前etcd節點的IP）
client.pem、client-key.pem：命令行客戶端訪問etcd使用的證書私鑰（hosts可以不寫或者為空）
apiserver-etcd-client.pem、apiserver-etcd-client-key.pem：apiserver訪問etcd的證書及私鑰；
注：其中peer.pem和server.pem可以使用一個，因為都是服務端證書（hosts指定所有etcd節點IP）

client.pem和apiserver-etcd-client.pem可以使用一個，因為都是客戶端證書（hosts都為空或不寫）

k8s證書：

kube-apiserver.pem：kube-apiserver節點使用的證書（每個master生成一個，hosts為當前master的IP）
kubelet.pem：kube-apiserver訪問kubelet時的客戶端證書（每個master一個，hosts為當前master的IP）
aggregator-proxy.pem：kube-apiserver使用聚合時，客戶端訪問代理的證書（hosts為空）
admin.pem：kubectl客戶端的證書（hosts為空或者不寫）

kubernetes組件啟動參數說明

日志級別(-v)

--v=0 ： Generally useful for this to ALWAYS be visible to an operator.
--v=1 ： A reasonable default log level if you don’t want verbosity.
--v=2 ： Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level for most systems.
--v=3 ： Extended information about changes.
--v=4 ： Debug level verbosity.
--v=6 ： Display requested resources.
--v=7 ： Display HTTP request headers.
--v=8 ： Display HTTP request contents

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 二進制安裝部署k8s高可用集群V1.20 Kubernetes全棧架構師（二進制高可用安裝k8s集群部署篇）--學習筆記二進制部署k8s 1.18.14（高可用） k8s二進制部署(高可用) 使用二進制的方式部署 K8S-1.16 高可用集群 k8s高可用集群搭建（二進制方式安裝） 2、二進制安裝K8s 之部署ETCD集群 K8S學習筆記之二進制部署Kubernetes v1.13.4 高可用集群 Kubernetes全棧架構師（二進制高可用安裝k8s集群擴展篇）--學習筆記二進制搭建kubernetes多master集群【三、配置k8s master及高可用】