一、安装环境准备
1.机器列表
| 主机名 | IP | 操作系统 | 角色 | 安装软件 | VM配置 |
| k8s-nginx | 192.168.0.100 | CentOS 7 | nginx代理 | nginx |
CPU：1c MEM：1g |
| k8s-master1 | 192.168.0.101 | CentOS 7 | 管理节点 | docker |
CPU：2c MEM：2g |
| k8s-master2 | 192.168.0.102 | CentOS 7 | 管理节点 | docker |
CPU：2c MEM：2g |
| k8s-node1 | 192.168.0.111 | CentOS 7 | 工作节点 | docker kubelet kube-proxy flannel |
CPU：2c MEM：2g |
| k8s-node2 | 192.168.0.112 | CentOS 7 | 工作节点 | docker kubelet kube-proxy flannel |
CPU：2c MEM：2g |
| master | 192.168.0.123 | CentOS 7 | 主控节点 | 不需要 | CPU：1c MEM：1g |
2.版本信息
docker: 19.03.13
etcd: 3.4.14
kubernetes: 1.18.14
pod 网段：10.244.0.0/16
service 网段：10.96.0.0/12
kubernetes 内部地址：10.96.0.1
flannel: 0.13.0
coredns: v1.6.7
coredns 地址: 10.96.0.10
3.环境初始化
说明：以 maser 做主控机，对其他机器做远程操作。
3.1免密登录
在master主控机上执行以下两步操作。
[root@master ~]# ssh-keygen -t rsa [root@master ~]# for i in 101 102 111 112; do ssh-copy-id 192.168.0.$i; done
3.2在master主机添加hosts
[root@master ~]# vim /etc/hosts 192.168.0.101 k8s-master1 192.168.0.102 k8s-master2 192.168.0.111 k8s-node1 192.168.0.112 k8s-node2
注意：以下操作在四台k8s机器和两台ha机器上都要执行
3.2时间同步
将系统时间设置为当前时间，并写入CMOS，否则系统重启后修改时间会失效。
#设置系统时间 [root@master ~]# date -s '2021-01-10 12:09:00' Sun Jan 10 12:09:00 CST 2021 #系统时间强制写入CMOS [root@master ~]# clock -w #显示系统时间 [root@master ~]# date Sun Jan 10 12:09:41 CST 2021 #显示硬件时间 [root@master ~]# hwclock --show Sun 10 Jan 2021 12:09:49 PM CST -0.411765 seconds
3.3关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
3.4关闭selinux
setenforce 0 && sed -i 's/=enforcing/=disabled/g' /etc/selinux/config
3.5关闭swap
swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab
默认情况下，kubelet不允许所在的主机存在交换分区，后期规划的时候，可以考虑在系统安装的时候不创建交换分区，针对已经存在交换分区的可以设置忽略禁止使用swap的限制，不然无法启动kubelet。一般直接禁用swap就可以了，不需要执行此步骤。
vim /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false"
3.6安装docker并配置加速器
vim install_docker.sh #!/bin/bash yum install -y yum-utils device-mapper-persistent-data lvm2 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install docker-ce-19.03.13 -y systemctl enable docker systemctl start docker vim /etc/docker/daemon.json { "registry-mirrors": ["https://wyt8gbxw.mirror.aliyuncs.com"] } systemctl daemon-reload systemctl restart docker
二、安装nginx
为两台master节点提供高可用，可以选用云厂商的 slb，也可以用两台 nginx + keepalived 实现。
此处，为实验环境，用单台 nginx 做四层代理实现负载均衡。
# 安装 nginx [root@k8s-nginx ~]# yum install -y nginx # 创建子配置文件 [root@k8s-nginx ~]# cd /etc/nginx/conf.d/ [root@k8s-nginx conf.d]# vim lb.tcp stream { upstream master { hash $remote_addr consistent; server 192.168.0.101:6443 max_fails=3 fail_timeout=30; server 192.168.0.102:6443 max_fails=3 fail_timeout=30; } server { listen 6443; proxy_pass master; } } # 在主配置文件中引入该文件 [root@k8s-nginx ~]# cd /etc/nginx/ [root@k8s-nginx nginx]# vim nginx.conf ... include /etc/nginx/conf.d/*.tcp; ... # 加入开机自启，并启动 nginx [root@k8s-nginx nginx]# systemctl enable nginx Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service. [root@k8s-nginx nginx]# systemctl start nginx
三、集群搭建
说明：相关执行脚本和文件都放在/root/k8s目录下，提前在每个节点创建好该目录
1.创建相应目录
1.1管理节点用
[root@master mkdir]# vim mkdir_master.sh #!/bin/bash mkdir /opt/etcd/{bin,data,cfg,ssl} -p mkdir /opt/kubernetes/{bin,cfg,ssl,logs} -p mkdir /opt/kubernetes/logs/{kubelet,kube-proxy,kube-scheduler,kube-apiserver,kube-controller-manager} -p echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile echo 'export PATH=$PATH:/opt/etcd/bin' >> /etc/profile source /etc/profile
1.2工作节点用
[root@master mkdir]# vim mkdir_node.sh #!/bin/bash mkdir /opt/kubernetes/{bin,cfg,ssl,logs} -p mkdir /opt/kubernetes/logs/{kubelet,kube-proxy} -p echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile source /etc/profile
1.3发送脚本到相应远程k8s主机
[root@master mkdir]# ls mkdir_master.sh mkdir_node.sh [root@master mkdir]# cd .. [root@master k8s]# ls install_docker.sh mkdir [root@master k8s]# for i in 1 2; do scp -r mkdir/ k8s-master$i:/root/k8s/; done [root@master k8s]# for i in 1 2; do scp -r mkdir/ k8s-node$i:/root/k8s/; done
1.4在所有k8s主机上执行该脚本
[root@k8s-master1 mkdir]# bash mkdir_master.sh [root@k8s-master2 mkdir]# bash mkdir_master.sh [root@k8s-node1 mkdir]# bash mkdir_node.sh [root@k8s-node2 mkdir]# bash mkdir_node.sh
2.生成证书
2.1脚本内容如下
#!/bin/bash command_exists() { command -v "$@" > /dev/null 2>&1 } if command_exists cfssl; then echo "命令已存在" else # 下载生成证书命令 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 # 添加执行权限 chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 # 移动到 /usr/local/bin 目录下 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo fi #配置文件，默认签 10 年 cat > ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF #颁发者信息 CN - 一般名词，O - 组织 ， OU组织单位， cat > ca-csr.json <<EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing", "O": "k8s", "OU": "System" } ] } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare ca - #----------------------- #颁发给组织信息 #包含master节点、node节点、nginx节点、coredns等 cat > server-csr.json <<EOF { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.0.101", "192.168.0.102", "192.168.0.111", "192.168.0.112", "192.168.0.100", "10.96.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server #----------------------- #颁发给管理员管理集群 cat > admin-csr.json <<EOF { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin #----------------------- #颁发给proxy cat > kube-proxy-csr.json <<EOF { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy #----------------------- #查看证书有效期 for item in $(ls *.pem |grep -v key) ;do echo ======================$item===================;openssl x509 -in $item -text -noout| grep Not;done
2.2执行脚本
[root@master ssl]# bash certificate.sh ...... ======================admin.pem=================== Not Before: Jan 10 04:30:00 2021 GMT Not After : Jan 8 04:30:00 2031 GMT ======================ca.pem=================== Not Before: Jan 10 04:30:00 2021 GMT Not After : Jan 9 04:30:00 2026 GMT ======================kube-proxy.pem=================== Not Before: Jan 10 04:30:00 2021 GMT Not After : Jan 8 04:30:00 2031 GMT ======================server.pem=================== Not Before: Jan 10 04:30:00 2021 GMT Not After : Jan 8 04:30:00 2031 GMT
2.3将证书发送到相应节点
etcd集群的 [root@master ssl]# for i in 1 2; do scp ca.pem server.pem server-key.pem k8s-master$i:/opt/etcd/ssl; done k8s集群的 [root@master ssl]# for i in 1 2; do scp *.pem k8s-master$i:/opt/kubernetes/ssl; done [root@master ssl]# for i in 1 2; do scp *.pem k8s-node$i:/opt/kubernetes/ssl; done
3.安装etcd集群
3.1下载etcd二进制包，解压，并发送到各管理节点
[root@master src]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz [root@master src]# cd etcd-v3.4.14-linux-amd64/ [root@master etcd-v3.4.14-linux-amd64]# for i in 1 2; do scp etcd etcdctl k8s-master$i:/opt/etcd/bin; done
3.2启动etcd
启动脚本内容如下
[root@master k8s]# vim etcd.sh #!/bin/bash # example: bash etcd.sh etcd01 192.168.0.101 etcd01=https://192.168.0.101:2380,etcd02=https://192.168.0.102:2380 ETCD_NAME=$1 ETCD_IP=$2 ETCD_CLUSTER=$3 ETCD_VERSION=3.4.14 cat <<EOF >/opt/etcd/cfg/etcd.yml #etcd ${ETCD_VERSION} name: ${ETCD_NAME} data-dir: /opt/etcd/data listen-peer-urls: https://${ETCD_IP}:2380 listen-client-urls: https://${ETCD_IP}:2379,https://127.0.0.1:2379 advertise-client-urls: https://${ETCD_IP}:2379 initial-advertise-peer-urls: https://${ETCD_IP}:2380 initial-cluster: ${ETCD_CLUSTER} initial-cluster-token: etcd-cluster initial-cluster-state: new enable-v2: true client-transport-security: cert-file: /opt/etcd/ssl/server.pem key-file: /opt/etcd/ssl/server-key.pem client-cert-auth: false trusted-ca-file: /opt/etcd/ssl/ca.pem auto-tls: false peer-transport-security: cert-file: /opt/etcd/ssl/server.pem key-file: /opt/etcd/ssl/server-key.pem client-cert-auth: false trusted-ca-file: /opt/etcd/ssl/ca.pem auto-tls: false debug: false logger: zap log-outputs: [stderr] EOF cat <<EOF >/usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Server Documentation=https://github.com/etcd-io/etcd Conflicts=etcd.service After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify LimitNOFILE=65536 Restart=on-failure RestartSec=5s TimeoutStartSec=0 ExecStart=/opt/etcd/bin/etcd --config-file=/opt/etcd/cfg/etcd.yml [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable etcd systemctl restart etcd
发送启动脚本到各管理节点
[root@master k8s]# for i in 1 2; do scp etcd.sh k8s-master$i:/root/k8s; done
在各管理节点执行启动脚本
[root@k8s-master1 k8s]# bash etcd.sh etcd01 192.168.0.101 etcd01=https://192.168.0.101:2380,etcd02=https://192.168.0.102:2380 [root@k8s-master2 k8s]# bash etcd.sh etcd02 192.168.0.102 etcd01=https://192.168.0.101:2380,etcd02=https://192.168.0.102:2380 说明：在etcd01执行后会卡住，这时是在等待其他节点加入，此时接着启动etcd02即可
验证集群是否健康
[root@k8s-master1 k8s]# etcdctl --write-out="table" --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints=https://192.168.0.101:2379,https://192.168.0.102:2379 endpoint health +----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +----------------------------+--------+-------------+-------+ | https://192.168.0.101:2379 | true | 10.305523ms | | | https://192.168.0.102:2379 | true | 12.624803ms | | +----------------------------+--------+-------------+-------+
查看集群成员
[root@k8s-master1 k8s]# etcdctl --write-out="table" --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints=https://192.168.0.101:2379,https://192.168.0.102:2379 member list +------------------+---------+--------+----------------------------+----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+--------+----------------------------+----------------------------+------------+ | aeed280fbd4b48b | started | etcd01 | https://192.168.0.101:2380 | https://192.168.0.101:2379 | false | | 1e149427bfba9593 | started | etcd02 | https://192.168.0.102:2380 | https://192.168.0.102:2379 | false | +------------------+---------+--------+----------------------------+----------------------------+------------+
4.master管理节点部署
4.1下载kubernetes二进制包，解压，并将相应组件发送到各节点
[root@master src]# wget https://dl.k8s.io/v1.18.14/kubernetes-server-linux-amd64.tar.gz #管理节点 [root@master bin]# for i in 1 2; do scp kube-apiserver kube-scheduler kube-controller-manager kubectl k8s-master$i:/opt/kubernetes/bin/; done #工作节点 [root@master bin]# for i in 1 2; do scp kubelet kube-proxy k8s-node$i:/opt/kubernetes/bin/; done
4.2生成kubeconfig文件
创建token文件
[root@master kubeconfig]# head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 8e72984386f2ab78f0b769bb66d75b98 [root@master kubeconfig]# cat token.csv 8e72984386f2ab78f0b769bb66d75b98,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
脚本内容如下
[root@master kubeconfig]# vim kubeconfig.sh #!/bin/bash # TLS Bootstrapping Token BOOTSTRAP_TOKEN=8e72984386f2ab78f0b769bb66d75b98 APISERVER=$1 SSL_DIR=$2 export KUBE_APISERVER="https://${APISERVER}:6443" #---------------------- # 创建kubelet bootstrapping kubeconfig # 设置集群参数 kubectl config set-cluster kubernetes \ --certificate-authority=${SSL_DIR}/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=bootstrap.kubeconfig # 设置客户端认证参数 kubectl config set-credentials kubelet-bootstrap \ --token=${BOOTSTRAP_TOKEN} \ --kubeconfig=bootstrap.kubeconfig # 设置上下文参数 kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig # 设置默认上下文 kubectl config use-context default --kubeconfig=bootstrap.kubeconfig #---------------------- # 创建kube-proxy kubeconfig文件 kubectl config set-cluster kubernetes \ --certificate-authority=${SSL_DIR}/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials kube-proxy \ --client-certificate=${SSL_DIR}/kube-proxy.pem \ --client-key=${SSL_DIR}/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=kube-proxy \ --kubeconfig=kube-proxy.kubeconfig kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig #---------------------- # 创建 admin kubeconfig文件 kubectl config set-cluster kubernetes \ --certificate-authority=${SSL_DIR}/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=admin.kubeconfig kubectl config set-credentials admin \ --client-certificate=${SSL_DIR}/admin.pem \ --client-key=${SSL_DIR}/admin-key.pem \ --embed-certs=true \ --kubeconfig=admin.kubeconfig kubectl config set-context default \ --cluster=kubernetes \ --user=admin \ --kubeconfig=admin.kubeconfig kubectl config use-context default --kubeconfig=admin.kubeconfig
注意：设置kube-apiserver访问地址， 因为需要对kube-apiserver配置高可用集群， 这里设置apiserver为nginx节点IP， 端口为6443
发送token.csv文件和执行脚本到各管理节点
[root@master k8s]# for i in 1 2; do scp -r kubeconfig k8s-master$i:/root/k8s/; done
在各管理节点上执行脚本，192.168.0.100为nginx节点IP，/opt/kubernetes/ssl为证书路径
[root@k8s-master1 kubeconfig]# bash kubeconfig.sh 192.168.0.100 /opt/kubernetes/ssl Cluster "kubernetes" set. User "kubelet-bootstrap" set. Context "default" created. Switched to context "default". Cluster "kubernetes" set. User "kube-proxy" set. Context "default" created. Switched to context "default". Cluster "kubernetes" set. User "admin" set. Context "default" created. Switched to context "default". [root@k8s-master1 kubeconfig]# ls admin.kubeconfig bootstrap.kubeconfig kubeconfig.sh kube-proxy.kubeconfig token.csv [root@k8s-master1 kubeconfig]# cp token.csv *config /opt/kubernetes/cfg/ [root@k8s-master2 kubeconfig]# bash kubeconfig.sh 192.168.0.100 /opt/kubernetes/ssl [root@k8s-master2 kubeconfig]# ls admin.kubeconfig bootstrap.kubeconfig kubeconfig.sh kube-proxy.kubeconfig token.csv [root@k8s-master2 kubeconfig]# cp token.csv *config /opt/kubernetes/cfg/
发送配置文件到各工作节点
[root@k8s-master1 kubeconfig]# for i in 111 112; do scp token.csv *config 192.168.0.$i:/opt/kubernetes/cfg/; done
4.3启动各组件
4.3.1各组件启动脚本内容如下
kube-apiserver
[root@master master]# vim apiserver.sh #!/bin/bash MASTER_ADDRESS=$1 ETCD_SERVERS=$2 cat <<EOF >/opt/kubernetes/cfg/kube-apiserver KUBE_APISERVER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs/kube-apiserver \\ --etcd-servers=${ETCD_SERVERS} \\ --bind-address=0.0.0.0 \\ --secure-port=6443 \\ --advertise-address=${MASTER_ADDRESS} \\ --allow-privileged=true \\ --service-cluster-ip-range=10.96.0.0/12 \\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction \\ --authorization-mode=RBAC,Node \\ --kubelet-https=true \\ --enable-bootstrap-token-auth=true \\ --token-auth-file=/opt/kubernetes/cfg/token.csv \\ --service-node-port-range=30000-50000 \\ --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\ --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\ --tls-cert-file=/opt/kubernetes/ssl/server.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\ --client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --etcd-cafile=/opt/etcd/ssl/ca.pem \\ --etcd-certfile=/opt/etcd/ssl/server.pem \\ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \\ --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\ --requestheader-group-headers=X-Remote-Group \\ --requestheader-username-headers=X-Remote-User \\ --runtime-config=api/all=true \\ --audit-log-maxage=30 \\ --audit-log-maxbackup=3 \\ --audit-log-maxsize=100 \\ --audit-log-truncate-enabled=true \\ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log" EOF cat <<EOF >/usr/lib/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kube-apiserver systemctl restart kube-apiserve
kube-scheduler
[root@master master]# vim scheduler.sh #!/bin/bash MASTER_ADDRESS=$1 cat <<EOF >/opt/kubernetes/cfg/kube-scheduler KUBE_SCHEDULER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs/kube-scheduler \\ --master=${MASTER_ADDRESS}:8080 \\ --address=0.0.0.0 \\ --leader-elect" EOF cat <<EOF >/usr/lib/systemd/system/kube-scheduler.service [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kube-scheduler systemctl restart kube-scheduler
kube-controller-manager
[root@master master]# vim controller-manager.sh #!/bin/bash MASTER_ADDRESS=$1 cat <<EOF >/opt/kubernetes/cfg/kube-controller-manager KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs/kube-controller-manager \\ --master=${MASTER_ADDRESS}:8080 \\ --leader-elect=true \\ --bind-address=0.0.0.0 \\ --service-cluster-ip-range=10.96.0.0/12 \\ --cluster-name=kubernetes \\ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --experimental-cluster-signing-duration=87600h0m0s \\ --feature-gates=RotateKubeletServerCertificate=true \\ --feature-gates=RotateKubeletClientCertificate=true \\ --allocate-node-cidrs=true \\ --cluster-cidr=10.244.0.0/16 \\ --root-ca-file=/opt/kubernetes/ssl/ca.pem" EOF cat <<EOF >/usr/lib/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kube-controller-manager systemctl restart kube-controller-manager
4.3.2发送脚本到各管理节点
[root@master k8s]# for i in 1 2; do scp -r master k8s-master$i:/root/k8s/; done
4.3.3在各管理节点执行启动脚本，apiserver参数为节点IP和etcd集群地址
#管理节点1 cd /root/k8s/master [root@k8s-master1 master]# bash apiserver.sh 192.168.0.101 https://192.168.0.101:2379,https://192.168.0.102:2379 [root@k8s-master1 master]# bash scheduler.sh 127.0.0.1 [root@k8s-master1 master]# bash controller-manager.sh 127.0.0.1 #管理节点2 cd /root/k8s/master [root@k8s-master2 master]# bash apiserver.sh 192.168.0.102 https://192.168.0.101:2379,https://192.168.0.102:2379 [root@k8s-master2 master]# bash scheduler.sh 127.0.0.1 [root@k8s-master2 master]# bash controller-manager.sh 127.0.0.1
4.3.4查看各组件状态
#管理节点1 [root@k8s-master1 master]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} #管理节点2 [root@k8s-master2 master]# kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-1 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"}
输出以上结果表示各组件正常运行。
5.node工作节点部署
5.1将kubelet-bootstrap用户绑定到系统集群角色
节点 kubelet 启动时自动创建 CSR 请求，将kubelet-bootstrap用户绑定到系统集群角色 ,这个是为了颁发证书用的权限。
Master apiserver启用TLS认证后，Node节点kubelet组件想要加入集群，必须使用CA签发的有效证书才能与 apiserver通信，当Node节点很多时，签署证书是一件很繁琐的事情，因此有了TLS Bootstrapping机制，kubelet 会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。
在任意一台管理节点上执行以下命令
[root@k8s-master1 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
5.2启动各组件
5.2.1各组件启动脚本内容如下
kubelet
[root@master node]# vim kubelet.sh #!/bin/bash NODE_ADDRESS=$1 cat <<EOF >/opt/kubernetes/cfg/kubelet KUBELET_OPTS="--logtostderr=true \\ --v=2 \\ --log-dir=/opt/kubernetes/logs/kubelet \\ --hostname-override=${NODE_ADDRESS} \\ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\ --config=/opt/kubernetes/cfg/kubelet.config \\ --cert-dir=/opt/kubernetes/ssl \\ --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0" EOF cat <<EOF >/opt/kubernetes/cfg/kubelet.config kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0.0.0 port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - 10.96.0.10 clusterDomain: cluster.local. failSwapOn: false # 身份验证 authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem # 授权 authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s # Node 资源保留 evictionHard: imagefs.available: 15% memory.available: 300Mi nodefs.available: 10% nodefs.inodesFree: 5% evictionPressureTransitionPeriod: 5m0s # 镜像删除策略 imageGCHighThresholdPercent: 85 imageGCLowThresholdPercent: 80 imageMinimumGCAge: 2m0s # 旋转证书 rotateCertificates: true # 旋转kubelet client 证书 featureGates: RotateKubeletServerCertificate: true RotateKubeletClientCertificate: true maxOpenFiles: 1000000 maxPods: 110 EOF cat <<EOF >/usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet After=docker.service Requires=docker.service [Service] EnvironmentFile=/opt/kubernetes/cfg/kubelet ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS Restart=on-failure KillMode=process [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kubelet systemctl restart kubelet
kube-proxy
[root@master node]# vim kube-proxy.sh #!/bin/bash NODE_ADDRESS=$1 cat <<EOF >/opt/kubernetes/cfg/kube-proxy.conf KUBE_PROXY_OPTS="--logtostderr=false \\ --v=2 \\ --log-dir=/opt/kubernetes/logs/kube-proxy \\ --config=/opt/kubernetes/cfg/kube-proxy-config.yml" EOF cat <<EOF >/opt/kubernetes/cfg/kube-proxy-config.yml kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 address: 0.0.0.0 clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig hostnameOverride: ${NODE_ADDRESS} clusterCIDR: 10.244.0.0/16 mode: iptables EOF cat <<EOF >/usr/lib/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy.conf ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kube-proxy systemctl restart kube-proxy
5.2.2发送脚本到各工作节点
[root@master k8s]# for i in 1 2; do scp -r node k8s-node$i:/root/k8s/; done
5.2.3在各工作节点执行启动脚本
#工作节点1 cd /root/k8s/node [root@k8s-node1 node]# bash kubelet.sh 192.168.0.111 [root@k8s-node1 node]# bash kube-proxy.sh 192.168.0.111 #工作节点2 cd /root/k8s/node [root@k8s-node2 node]# bash kubelet.sh 192.168.0.112 [root@k8s-node2 node]# bash kube-proxy.sh 192.168.0.112
5.2.4在master节点手动审批
kubelet启动后还没加入到集群中，需要手动允许该节点才可以。
[root@k8s-master1 master]# kubectl get node No resources found in default namespace. [root@k8s-master1 master]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION csr-2xx8v 2m2s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending csr-6n8bw 4s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
未审批的时候，csr请求处于Pending状态，审批通过后即处于Approved,Issued状态
[root@k8s-master1 master]# kubectl certificate approve csr-2xx8v certificatesigningrequest.certificates.k8s.io/csr-2xx8v approved [root@k8s-master1 master]# kubectl certificate approve csr-6n8bw certificatesigningrequest.certificates.k8s.io/csr-6n8bw approved [root@k8s-master1 master]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION csr-2xx8v 8m7s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued csr-6n8bw 6m9s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
等待一会，即可查看node节点已处于Ready状态
[root@k8s-master1 master]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.0.111 Ready <none> 82s v1.18.14
192.168.0.112 Ready <none> 76s v1.18.14
6.安装flannel网络插件
6.1下载flannel二进制包，解压并发送到各节点
#下载 [root@master flannel]# wget https://github.com/coreos/flannel/releases/download/v0.13.0/flannel-v0.13.0-linux-amd64.tar.gz #管理节点 [root@master flannel]# for i in 1 2; do scp flanneld k8s-master$i:/opt/kubernetes/bin/; done #工作节点 [root@master flannel]# for i in 1 2; do scp flanneld k8s-node$i:/opt/kubernetes/bin/; done
6.2将 pod 网段信息写入 etcd 中
falnnel要用etcd存储自身一个子网信息，所以要保证能成功连接etcd，写入预定义子网段。
[root@k8s-master1 ~]# ETCDCTL_API=2 etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem --endpoints=https://192.168.0.101:2379,https://192.168.0.102:2379 set /coreos.com/network/config '{"Network": "10.244.0.0/16", "Backend": {"Type": "vxlan"}}' {"Network": "10.244.0.0/16", "Backend": {"Type": "vxlan"}}
6.3启动脚本内容如下
[root@master flannel]# vim flannel.sh #!/bin/bash ETCD_ENDPOINTS=$1 cat <<EOF >/opt/kubernetes/cfg/flanneld FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \ -etcd-cafile=/opt/kubernetes/ssl/ca.pem \ -etcd-certfile=/opt/kubernetes/ssl/server.pem \ -etcd-keyfile=/opt/kubernetes/ssl/server-key.pem" EOF cat <<EOF >/usr/lib/systemd/system/flanneld.service [Unit] Description=Flanneld overlay address etcd agent After=network-online.target network.target Before=docker.service [Service] Type=notify EnvironmentFile=/opt/kubernetes/cfg/flanneld ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable flanneld systemctl restart flanneld
6.4将脚本发送到各节点
#管理节点 [root@master flannel]# for i in 1 2; do scp flannel.sh k8s-master$i:/root/k8s/; done #工作节点 [root@master flannel]# for i in 1 2; do scp flannel.sh k8s-node$i:/root/k8s/; done
6.5在各节点执行脚本
[root@k8s-master1 k8s]# bash flannel.sh https://192.168.0.101:2379,https://192.168.0.102:2379 [root@k8s-master1 k8s]# ifconfig flannel.1 flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 10.244.30.0 netmask 255.255.255.255 broadcast 10.244.30.0 inet6 fe80::ec32:5bff:fee7:a854 prefixlen 64 scopeid 0x20<link> ether ee:32:5b:e7:a8:54 txqueuelen 0 (Ethernet) RX packets 2 bytes 168 (168.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2 bytes 168 (168.0 B) TX errors 0 dropped 20 overruns 0 carrier 0 collisions 0
如上，生成flannel.1虚拟网卡，且各节点flannel.1网卡能够互相ping通，则表示flannel安装成功。
7.安装coredns
注意：k8s 与 coredns 的版本对应关系
https://github.com/coredns/deployment/blob/master/kubernetes/CoreDNS-k8s_version.md
7.1yaml文件内容如下
注意修改 clusterIP 和镜像版本1.6.7
[root@master k8s]# vim coredns.yaml apiVersion: v1 kind: ServiceAccount metadata: name: coredns namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } --- apiVersion: apps/v1 kind: Deployment metadata: name: coredns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/name: "CoreDNS" spec: # replicas: not specified here: # 1. Default is 1. # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on. replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns spec: priorityClassName: system-cluster-critical serviceAccountName: coredns tolerations: - key: "CriticalAddonsOnly" operator: "Exists" nodeSelector: kubernetes.io/os: linux affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname containers: - name: coredns image: coredns/coredns:1.6.7 imagePullPolicy: IfNotPresent resources: limits: memory: 170Mi requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP dnsPolicy: Default volumes: - name: config-volume configMap: name: coredns items: - key: Corefile path: Corefile --- apiVersion: v1 kind: Service metadata: name: kube-dns namespace: kube-system annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" kubernetes.io/name: "CoreDNS" spec: selector: k8s-app: kube-dns clusterIP: 10.96.0.10 ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP - name: metrics port: 9153 protocol: TCP
7.2将yaml文件发送到各管理节点，并在任一管理节点上通过kubectl命令创建
[root@master k8s]# for i in 1 2; do scp coredns.yaml k8s-master$i:/root/k8s/; done [root@k8s-master1 k8s]# kubectl apply -f coredns.yaml serviceaccount/coredns created clusterrole.rbac.authorization.k8s.io/system:coredns created clusterrolebinding.rbac.authorization.k8s.io/system:coredns created configmap/coredns created deployment.apps/coredns created service/kube-dns created [root@k8s-master1 k8s]# kubectl get service -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 67m kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 2m7s
7.3验证dns是否正常运行
#创建一个 busybox 容器作为客户端 [root@k8s-master1 k8s]# kubectl create -f https://k8s.io/examples/admin/dns/busybox.yaml pod/busybox created #解析 kubernetes [root@k8s-master1 k8s]# kubectl exec -it busybox -- nslookup kubernetes Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
参考博客 https://www.cnblogs.com/liyongjian5179/p/13198143.html