第一章 背景说明
1. k8s调用机制
2.组件分布情况
master节点组件：apiserver、controller-manager、scheduler
node节点组件：kubelet、kube-proxy
其他组件：coredns、CNI网络（flannel或者Calico）
3.组件介绍
apiserver：负责整个k8s集群内部接口消息的接收及转发
controller-manager：负责维护集群的状态，比如故障检测、自动扩展、滚动新等
scheduler：负责资源的调度，按照预定的调度策略将 Pod 调度到相应的机器上
kubelet：负责维护容器的生命周期，同时也负责 Volume（CSI）和网络（CNI）的管理
kube-proxy：负责为 Service 提供 cluster 内部的服务发现和负载均衡
coredns：负责为整个集群提供 DNS 服务
CNI网络：负责整个集群的网络
第二章 部署
| 服务 |
要求 |
| 内核 |
内核版本在3.10+ |
| centos/redhat系统 |
使用7.5以上的系统 |
| 多master节点 |
需要准备一个vip |
1.环境准备
服务器要求：
- 建议最小硬件配置：2核CPU、2G内存、40G硬盘
- 服务器最好可以访问外网，会有从网上拉取镜像需求，如果服务器不能上网，需要提前下载对应镜像并导入节点
服务器整体规划：
| 角色 |
IP |
其他单装组件 |
| k8s-master1 |
192.168.31.61 |
docker，etcd，nginx，keepalived |
| k8s-master2 |
192.168.31.62 |
docker，etcd，nginx，keepalived |
| k8s-node1 |
192.168.31.63 |
docker，etcd |
| 负载均衡器对外IP |
192.168.31.88 (VIP) |
|
2.部署架构图
3.系统初始化
安装ansible工具
所有节点都安装
配置ssh免密登录
在master-1上操作
编辑ansible配置文件，在master-1主机上操作
测试是否互通
基本初始化操作
直接使用ansible剧本，对防火墙，主机名，swap等进行更改
| [root@k8s-master-1 ~]# cat basic.yaml - hosts: k8s-all tasks: - name: 关闭selinux shell: sed -ri 's/enforcing/disabled/' /etc/selinux/config && setenforce 0 ignore_errors: yes - name: 分发hosts文件 copy: src=/etc/hosts dest=/etc/hosts - name: 分发k8s.conf copy: src=k8s.conf dest=/etc/sysctl.d/k8s.conf # 注意此处src的k8s.conf需要准备好 - name: 参数生效 shell: sysctl --system - name: 关闭swap lineinfile: dest: /etc/selinux/config regexp: '^SELINUX' line: 'SELINUX=disabled' - name: 关闭防火墙 service: name=firewalld state=stopped enabled=no |
| [root@k8s-master-1 ~]# cat k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 |
ansible-playbook basic.yaml
4.部署nginx+keepalived
只在两台master节点上安装,在官网上下载nginx的rpm包进行安装
| Kubernetes作为容器集群系统，通过健康检查+重启策略实现了Pod故障自我修复能力，通过调度算法实现将Pod分布式部署，并保持预期副本数，根据Node失效状态自动在其他Node拉起Pod，实现了应用层的高可用性。 针对Kubernetes集群，高可用性还应包含以下两个层面的考虑：Etcd数据库的高可用性和Kubernetes Master组件的高可用性。 而kubeadm搭建的K8s集群，Etcd只起了一个，存在单点，所以我们这里会独立搭建一个Etcd集群。 Master节点扮演着总控中心的角色，通过不断与工作节点上的Kubelet和kube-proxy进行通信来维护整个集群的健康工作状态。如果Master节点故障，将无法使用kubectl工具或者API做任何集群管理。 Master节点主要有三个服务kube-apiserver、kube-controller-manager和kube-scheduler，其中kube-controller-manager和kube-scheduler组件自身通过选择机制已经实现了高可用，所以Master高可用主要针对kube-apiserver组件，而该组件是以HTTP API提供服务，因此对他高可用与Web服务器类似，增加负载均衡器对其负载均衡即可，并且可水平扩容。
|
kube-apiserver高可用架构图：
注：为了节省机器，这里与K8s master节点机器复用。也可以独立于k8s集群之外部署，只要nginx与apiserver能通信就行。
|
安装软件包
在两台master上执行
yum install epel-release -y ##nginx使用RPM包安装 Rpm包网址：http://nginx.org/packages/rhel/7/x86_64/RPMS/ wget http://nginx.org/packages/rhel/7/x86_64/RPMS/nginx-1.20.0-1.el7.ngx.x86_64.rpm |
修改nginx配置文件
两台master节点上一样
| [root@k8s-master-1 ~]# cat /etc/nginx/nginx.conf worker_processes auto; #user root; events { worker_connections 1024; use epoll; } error_log /var/log/nginx/error.log notice; pid /var/run/nginx.pid;
stream { upstream kube-apiservers { hash $remote_addr consistent; # 轮询方式 server 192.168.2.34:6443 weight=6 max_fails=1 fail_timeout=10s; #代理得url server 192.168.2.174:6443 weight=6 max_fails=1 fail_timeout=10s; #代理得url }
server { listen 8443; # 端口 proxy_connect_timeout 30s; proxy_timeout 60s; proxy_pass kube-apiservers; } } |
修改Keepalived配置
在master-1节点上执行，为主
yum install keepalived -y
| cat > /etc/keepalived/keepalived.conf << EOF global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_MASTER }
vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" }
vrrp_instance VI_1 { state MASTER interface eth0 # 修改为实际网卡名 virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 priority 100 # 优先级，备服务器设置 90 advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒 authentication { auth_type PASS auth_pass 1111 } # 虚拟IP virtual_ipaddress { 192.168.5.8 } track_script { check_nginx } } EOF |
- vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）
- virtual_ipaddress：虚拟IP（VIP）
准备上述配置文件中检查nginx运行状态的脚本：
|
cat > /etc/keepalived/check_nginx.sh << "EOF" nginx_num=`ps -ef|grep ^[n]ginx|wc -l` if [[ $nginx_num -eq 0 ]] then systemctl stop keepalived && exit 1 else exit 0 fi chmod +x /etc/keepalived/check_nginx.sh
|
| cat > /etc/keepalived/keepalived.conf << EOF global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_BACKUP }
vrrp_script check_nginx { script "/etc/keepalived/check_nginx.sh" }
vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.5.8 } track_script { check_nginx } } EOF |
注：keepalived根据脚本返回状态码（0为工作正常，非0不正常）判断是否故障转移。
启动nginx，keepalived
在两台master主机上都执行如下命令
systemctl daemon-reload
|
查看keepalived工作状态
Nginx+keepalived高可用测试
关闭主节点Nginx，测试VIP是否漂移到备节点服务器。
在Nginx Master执行 pkill nginx
在Nginx Backup，ip addr命令查看已成功绑定VIP。
第三章 部署ETCD集群
Etcd 是一个分布式键值存储系统，Kubernetes使用Etcd进行数据存储，kubeadm搭建默认情况下只启动一个Etcd Pod，存在单点故障，生产环境强烈不建议，所以我们这里使用3台服务器组建集群，可容忍1台机器故障，当然，你也可以使用5台组建集群，可容忍2台机器故障。
| 节点名称 |
IP |
| K8s-master-1 |
192.168.5.66 |
| K8s-master-2 |
192.168.5.117 |
| K8s-node-1 |
192.168.5.82 |
注：为了节省机器，这里与K8s节点机器复用。也可以独立于k8s集群之外部署，只要apiserver能连接到就行。
准备cfssl证书生成工具
cfssl是一个开源的证书管理工具，使用json文件生成证书，相比openssl更方便使用。
找任意一台服务器操作，这里用k8s-master-1节点。
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
生成ETCD证书
自签证书颁发机构（CA）
创建工作目录：
mkdir -p ~/etcd_tls |
自签CA：
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
cat > ca-csr.json << EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
生成证书：
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
会生成ca.pem和ca-key.pem文件
使用自签CA签发Etcd HTTPS证书
创建证书申请文件：
cat > server-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"192.168.5.66",
"192.168.5.117",
"192.168.5.82"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
注：上述文件hosts字段中IP为所有etcd节点的集群内部通信IP，一个都不能少！为了方便后期扩容可以多写几个预留的IP。
生成证书：
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
会生成server.pem和server-key.pem文件。
从GitHub下载etcd二进制文件
下载地址：
| https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz |
部署ETCD集群
以下在节点1上操作(k8s-master-1)，为简化操作，待会将节点1生成的所有文件拷贝到节点2(k8s-master-2)和节点3(k8s-node-1)。
创建工作目录并解压二进制包
mkdir –p /opt/etcd/{bin,cfg,ssl}
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
创建ETCD配置文件
cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.5.66:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.5.66:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.5.66:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.5.66:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.5.66:2380,etcd-2=https://192.168.5.117:2380,etcd-3=https://192.168.5.82:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
说明：
- ETCD_NAME：节点名称，集群中唯一
- ETCDDATADIR：数据目录
- ETCDLISTENPEER_URLS：集群通信监听地址
- ETCDLISTENCLIENT_URLS：客户端访问监听地址
- ETCDINITIALADVERTISEPEERURLS：集群通告地址
- ETCDADVERTISECLIENT_URLS：客户端通告地址
- ETCDINITIALCLUSTER：集群节点地址
- ETCDINITIALCLUSTER_TOKEN：集群Token
- ETCDINITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群
systemd管理etcd
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
拷贝证书
把前面的证书拷贝到配置文件中的路径：
cp ~/etcd_tls/ca*pem ~/etcd_tls/server*pem /opt/etcd/ssl/
将节点1所有生成的文件拷贝到节点2和节点3
scp -r /opt/etcd/ root@192.168.5.117:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.5.117:/usr/lib/systemd/system/
scp -r /opt/etcd/ root@192.168.5.82:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.5.82:/usr/lib/systemd/system/
然后在节点2和节点3分别修改etcd.conf配置文件中的节点名称和当前服务器IP：
vi /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1" # 修改此处，节点2改为etcd-2，节点3改为etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.31.71:2380" # 修改此处为当前服务器IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.31.71:2379" # 修改此处为当前服务器IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.31.71:2380" # 修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.31.71:2379" # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.31.71:2380,etcd-2=https://192.168.31.72:2380,etcd-3=https://192.168.31.73:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
最后启动etcd并设置开机启动
注：etcd节点要一起启动
ansible操作
ansible etcd -m service -a 'name=etcd state=started enabled=yes daemon-reload=yes'
注：etcd是在ansible里做的分组，这个组下面只有etcd这三台的主机ip
查看集群状态
| ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.5.66:2379,https://192.168.5.117:2379,https://192.168.5.82:2379" endpoint health --write-out=table |
如果输出上面信息，就说明集群部署成功。
如果有问题第一步先看日志：/var/log/message 或 journalctl -u etcd
查看etcd主备
| ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.5.66:2379,https://192.168.5.117:2379,https://192.168.5.82:2379" endpoint status --write-out=table |
第四章Docker安装
安装
所有节点全部安装
##更换yum源
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
##yum安装docker，所有节点都安装
yum -y install docker-ce-19.03.15
##启动并加入开机自启动
systemctl enable docker && systemctl start docker
注：安装的docker版本需要与k8s的版本做匹配，并注意k8s在1.20版本开始将要不支持docker，将于1.23版本后彻底不支持docker（即：1.23版本之前可以使用docker）
替代docker目录比较成熟的产品：
1、podman
2、containerd # 推荐，资料较多
配置
[root@mater01 roles]# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"], # 公网镜像加速器，内网不用配置
"insecure-registries":["123.60.214.87:18080"], # harbor仓库地址，无harbor不需要配置
"data-root": "/home/docker_data" # 修改docker存储目录，要查看最大存储的磁盘在哪个目录下
}
{
"registry-mirrors":["https://b9pmyelo.mirror.aliyuncs.com"],
"insecure-registries":["119.3.224.205:18080"],
"data-root":"/home/docker_data"
}
##注意json文件中不能有空格
##创建镜像目录
mkdir -p /home/docker_data
##重启
root@GQ-202-DWGL-K8SJQ02 roles]# systemctl restart docker # 重启
注：此文件需要单独创建，10.217.54.220:18080是预选规划的地址，如harbor仓库是https协议的就不要配置，只需要配置http协议的，修改这里任何内容，都需要重启docker
第五章 Harbor安装
镜像仓库尽量找一台单独机器部署
下载镜像
wget https://storage.googleapis.com/harbor-releases/release-1.8.0/harbor-offline-installer-v2.1.2.tgz
docker-compose
下载地址：curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/bin/docker-compose
注：执行这条命令会直接给放到/usr/bin下，只需再给加执行权限即可
浏览器下载：https://github.com/docker/compose/releases/download/1.18.0/docker-compose-Linux-x86_64
配置
[root@master01]# cp docker-compose-Linux-x86_64 /usr/bin/docker-compose
[root@master01]# chmod +x /usr/bin/docker-compose
[root@master01]# docker-compose –version
docker-compose version 1.18.0, build 8dd22a9
[root@master01]# tar xf harbor-offline-installer-v2.1.2.tgz
[root@master01]# cd harbor
[root@master01 harbor]# cp harbor.yml.tmpl harbor.yml
|
[root@master01 harbor]# vim harbor.yml 5 hostname: 10.217.54.220 10 port: 18080 11 12 # https related config 13 #https: 14 # https port for harbor, default is 443 15 #port: 443 16 # The path of cert and key files for nginx 17 #certificate: /your/certificate/path 18 #private_key: /your/private/key/path 27 harbor_admin_password: Harbor135 40 data_volume: /data/harbor
|
注：标红部分需要根据实际情况修改
hostname à 访问ip地址（可域名/IP）
port à 访问端口（根据实际情况修改）
13-18是配置https访问的端口及证书所在位置，如没有，则注释
harbor_admin_password à 登录界面默认密码
data_volume à 相关容器存储目录
安装
| [root@master01 harbor]# ./prepare # 初始化准备工作 [root@master01 harbor]# ./install.sh # 输出内容较多，省略
|
注：此步大致分为，
1、生成docker-compose.yml文件
2、导入镜像
3、启动
验证
注：此处会默认找docker-compose.yml文件，如果当前目录下没有，则需要指定docker-compose.yml文件（参数-f）
界面验证：
http://123.60.214.87:18080/
Docker登录验证
第六章 k8s部署
更换yum源
所有节点全部操作
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
注：上述yum源是7系统的，如果安装系统为8系统，则把el7替换为el8
安装kubeadm、kubectl、kubelet
由于版本更新频繁，这里指定版本号部署：所有节点都需要安装
ansible all -m shell -a 'yum install -y kubelet-1.20.0 kubeadm-1.20.0 kubectl-1.20.0'
ansible all -m service -a 'name=kubelet enabled=yes'
ansible all -m shell -a "mkdir -p /home/kubelet_data && echo 'KUBELET_EXTRA_ARGS=--root-dir=/home/kubelet_data' > /etc/sysconfig/kubelet"
初始化k8s-master-1节点
| cat > kubeadm-config.yaml << EOF apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: 9037x2.tcaqnpaqkra9vsbw ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.5.66 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: k8s-master-1 #注意修改主机名 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: certSANs: #包含所有Master/LB/VIP IP，一个都不能少！为了方便后期扩容可以多写几个预留的IP。 - k8s-master-1 - k8s-master-2 - 192.168.5.66 - 192.168.5.117 - 192.168.5.82 - 192.168.5.8 - 127.0.0.1 extraArgs: authorization-mode: Node,RBAC timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: 192.168.5.8:80 #负载均衡虚拟IP（VIP）和端口 controllerManager: {} dns: type: CoreDNS etcd: external: #使用外部etcd endpoints: - https://192.168.5.66:2379 #etcd集群3个节点 - https://192.168.5.117:2379 - https://192.168.5.82:2379 caFile: /opt/etcd/ssl/ca.pem #连接etcd所需证书 certFile: /opt/etcd/ssl/server.pem keyFile: /opt/etcd/ssl/server-key.pem imageRepository: registry.aliyuncs.com/google_containers #由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址 kind: ClusterConfiguration kubernetesVersion: v1.19.0 #K8s版本，与上面安装的一致 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 #Pod网络，与下面部署的CNI网络组件yaml中保持一致 serviceSubnet: 10.96.0.0/12 #集群内部虚拟网络，Pod统一访问入口 scheduler: {} EOF |
执行初始化命令
kubeadm init --config=kubeadm-config.yaml
| [root@k8s-master-1 ~]# kubeadm init --config=kubeadm-config.yaml [init] Using Kubernetes version: v1.20.0 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master-1 k8s-master-2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.96.0.1 192.168.5.66 192.168.5.8 192.168.5.117 192.168.5.82 127.0.0.1] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] External etcd mode: Skipping etcd/ca certificate authority generation [certs] External etcd mode: Skipping etcd/server certificate generation [certs] External etcd mode: Skipping etcd/peer certificate generation [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "kubelet.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 11.559587 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)" [mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: 9037x2.tcaqnpaqkra9vsbw [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities and service account keys on each node and then running the following as root:
kubeadm join 192.168.5.8:8443 --token 9037x2.tcaqnpaqkra9vsbw \ --discovery-token-ca-cert-hash sha256:39c7b865f08c6a0f410e5ae5adbd45a3d1e49822739acf4ab6156d05ae27d0c8 \ --control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.5.8:8443 --token 9037x2.tcaqnpaqkra9vsbw \ --discovery-token-ca-cert-hash sha256:39c7b865f08c6a0f410e5ae5adbd45a3d1e49822739acf4ab6156d05ae27d0c8 [root@k8s-master-1 ~]# |
kubeadm init执行后，做了哪些事
[init]：指定版本进行初始化操作
[preflight] ：初始化前的检查和下载所需要的Docker镜像文件
[kubelet-start] ：生成kubelet的配置文件，没有这个文件kubelet无法启动，所以初始化之前的kubelet实际上启动失败。
[certificates]：生成Kubernetes使用的证书，存放在/etc/kubernetes/pki目录中。
[kubeconfig] ：生成 KubeConfig 文件，存放在/etc/kubernetes目录中，组件之间通信需要使用对应文件。
[control-plane]：使用/etc/kubernetes/manifest目录下的YAML文件，安装Master 组件。
[etcd]：使用/etc/kubernetes/manifest/etcd.yaml安装Etcd服务。
[wait-control-plane]：等待control-plan部署的Master组件启动。
[apiclient]：检查Master组件服务状态。
[uploadconfig]：更新配置
[kubelet]：使用configMap配置kubelet。
[patchnode]：更新CNI信息到Node上，通过注释的方式记录。
[mark-control-plane]：为当前节点打标签，打了角色Master，和不可调度标签，这样默认就不会使用Master节点来运行Pod。
[bootstrap-token]：生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
[addons]：安装附加组件CoreDNS和kube-proxy
初始化后操作
图中圈出部分说明：
第一部分：在本地创建操作权限认证目录，将admin的认证复制到默认目录
第二部分：master节点加入集群命令
第三部分：node几点加入集群命令
拷贝kubectl使用的连接k8s认证文件到默认路径：
假设配错参数可以重新初始化操作：
kubeadm reset -f kubeadm-config.yaml
添加其它master节点
将Master1节点生成的证书拷贝到Master2：
scp -r /etc/kubernetes/pki/ 192.168.5.117:/etc/kubernetes/
复制加入master join命令在master2执行
| kubeadm join 192.168.5.8:8443 --token 9037x2.tcaqnpaqkra9vsbw \ --discovery-token-ca-cert-hash sha256:39c7b865f08c6a0f410e5ae5adbd45a3d1e49822739acf4ab6156d05ae27d0c8 \ --control-plane |
添加node节点
复制前面init初始化中加入node的命令 在其它node节点上执行
| kubeadm join 192.168.5.8:8443 --token 9037x2.tcaqnpaqkra9vsbw \ --discovery-token-ca-cert-hash sha256:39c7b865f08c6a0f410e5ae5adbd45a3d1e49822739acf4ab6156d05ae27d0c8 |
注：加入集群命令的token有效期为24小时，过期后重新获取命令如下
kubeadm token create --print-join-command
注：这个token可以理解为验证码，在一段时间内有效，过期后，需要重新生成，但不影响以加入集群的节点
验证
注：如STATUS出现NotReady状态的是因为还没有部署CNI网络
访问负载均衡测试
找K8s集群中任意一个节点，使用curl查看K8s版本测试，使用VIP访问
| [root@k8s-master-2 ~]# curl -k https://192.168.5.8:8443/version { "major": "1", "minor": "20", "gitVersion": "v1.20.0", "gitCommit": "af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", "gitTreeState": "clean", "buildDate": "2020-12-08T17:51:19Z", "goVersion": "go1.15.5", "compiler": "gc", "platform": "linux/amd64" |
可以正确获取到K8s版本信息，说明负载均衡器搭建正常。
该请求数据流程：curl -> vip(nginx) -> apiserver
通过查看Nginx日志也可以看到转发apiserver IP
查看整个pod状态，看到coredns还在创建中
解决方法：
需要等flannel网络打通后，coredns会自动创建好
部署CNI网络
Calico是一个纯三层的数据中心网络方案，是目前Kubernetes主流的网络方案，这里采用flannel部署
kubectl apply -f calico.yaml
kubectl get pods -n kube-system
验证：
如遇到如上问题，修改kube-controller-manager.yaml和kube-scheduler.yaml文件，在两台master都操作
| [root@k8s-master-2 ~]# sed -ri 's/- --port=0/#&/' /etc/kubernetes/manifests/kube-controller-manager.yaml [root@k8s-master-2 ~]# sed -ri 's/- --port=0/#&/' /etc/kubernetes/manifests/kube-scheduler.yaml [root@k8s-master-2 ~]# systemctl restart kubelet.service |
注：全部master节点都需要修改
再次验证coredns：
k8s从镜像仓库中拉取镜像
[root@k8s-master-1 ~]# kubectl create secret docker-registry docker-harbor-registry --docker-server=123.60.214.87:18080 --docker-username=admin --docker-p
assword='Harbor135'
编辑yaml文件
| [root@k8s-master-1 ~]# cat nginx_test.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx2.1 spec: replicas: 1 selector: matchLabels: app: nginx2.1 template: metadata: labels: app: nginx2.1 spec: imagePullSecrets: - name: docker-harbor-registry containers: - name: nginx2.1 image: 123.60.214.87:18080/test/nginx2.1 command: [ "/bin/bash", "-ce", "tail -f /dev/null" ] |
第七章 部署Dashboard
Dashboard是官方提供的一个UI，可用于基本管理K8s资源。
| [root@k8s-master-1 ~]# cat dashboard.yaml # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License.
apiVersion: v1 kind: Namespace metadata: name: kubernetes-dashboard
---
apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard
---
kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: ports: - port: 443 targetPort: 8443 selector: k8s-app: kubernetes-dashboard
---
apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-certs namespace: kubernetes-dashboard type: Opaque
---
apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-csrf namespace: kubernetes-dashboard type: Opaque data: csrf: ""
---
apiVersion: v1 kind: Secret metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-key-holder namespace: kubernetes-dashboard type: Opaque
---
kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 31443 selector: k8s-app: kubernetes-dashboard
---
kind: ConfigMap apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard-settings namespace: kubernetes-dashboard
---
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard rules: # Allow Dashboard to get, update and delete Dashboard exclusive secrets. - apiGroups: [""] resources: ["secrets"] resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"] verbs: ["get", "update", "delete"] # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map. - apiGroups: [""] resources: ["configmaps"] resourceNames: ["kubernetes-dashboard-settings"] verbs: ["get", "update"] # Allow Dashboard to get metrics. - apiGroups: [""] resources: ["services"] resourceNames: ["heapster", "dashboard-metrics-scraper"] verbs: ["proxy"] - apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"] verbs: ["get"]
---
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard rules: # Allow Metrics Scraper to get metrics from the Metrics server - apiGroups: ["metrics.k8s.io"] resources: ["pods", "nodes"] verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubernetes-dashboard roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kubernetes-dashboard subjects: - kind: ServiceAccount name: kubernetes-dashboard namespace: kubernetes-dashboard
---
kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: kubernetes-dashboard template: metadata: labels: k8s-app: kubernetes-dashboard spec: containers: - name: kubernetes-dashboard image: kubernetesui/dashboard:v2.0.4 imagePullPolicy: Always ports: - containerPort: 8443 protocol: TCP env: - name: ACCEPT_LANGUAGE value: zh args: - --auto-generate-certificates - --namespace=kubernetes-dashboard # Uncomment the following line to manually specify Kubernetes API server Host # If not specified, Dashboard will attempt to auto discover the API server and connect # to it. Uncomment only if the default does not work. # - --apiserver-host=http://my-address:port volumeMounts: - name: kubernetes-dashboard-certs mountPath: /certs # Create on-disk volume to store exec logs - mountPath: /tmp name: tmp-volume livenessProbe: httpGet: scheme: HTTPS path: / port: 8443 initialDelaySeconds: 30 timeoutSeconds: 30 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 volumes: - name: kubernetes-dashboard-certs secret: secretName: kubernetes-dashboard-certs - name: tmp-volume emptyDir: {} serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule
---
kind: Service apiVersion: v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: ports: - port: 8000 targetPort: 8000 selector: k8s-app: dashboard-metrics-scraper
---
kind: Deployment apiVersion: apps/v1 metadata: labels: k8s-app: dashboard-metrics-scraper name: dashboard-metrics-scraper namespace: kubernetes-dashboard spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: k8s-app: dashboard-metrics-scraper template: metadata: labels: k8s-app: dashboard-metrics-scraper annotations: seccomp.security.alpha.kubernetes.io/pod: 'runtime/default' spec: containers: - name: dashboard-metrics-scraper image: kubernetesui/metrics-scraper:v1.0.4 ports: - containerPort: 8000 protocol: TCP livenessProbe: httpGet: scheme: HTTP path: / port: 8000 initialDelaySeconds: 30 timeoutSeconds: 30 volumeMounts: - mountPath: /tmp name: tmp-volume securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 2001 serviceAccountName: kubernetes-dashboard nodeSelector: "kubernetes.io/os": linux # Comment the following tolerations if Dashboard must not be deployed on master tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule volumes: - name: tmp-volume emptyDir: {} |
kubectl apply -f kubernetes-dashboard.yaml
# 查看部署
kubectl get pods -n kubernetes-dashboard
创建service account并绑定，默认cluster-admin为管理员集群角色
| [root@k8s-master-1 ~]# kubectl create serviceaccount dashboard-admin -n kube-system serviceaccount/dashboard-admin created [root@k8s-master-1 ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created [root@k8s-master-1 ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}') Name: dashboard-admin-token-hcvbq Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name: dashboard-admin kubernetes.io/service-account.uid: 1f70af4b-588f-4fb8-b784-10bef10db7cf
Type: kubernetes.io/service-account-token
Data ==== ca.crt: 1066 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6Im5MMDlZeWZ2ZVdDM0x0RHk5YmdWOWJjeFpPUFVIUTZaOVJfM1VaUXpYcG8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9 uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4taGN2YnEiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3Vud C5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMWY3MGFmNGItNTg4Zi00ZmI4LWI3ODQtMTBiZWYxMGRiN2NmIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW5 0Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.i3xgiLVefiC02cn9UQTN4ZJNdoXJ0lgBuPmXm2YK7DwzokX4HVNSPMllh-drJ5NhazeQ1sVKEjQry9aACLGkM3Le26i6hPIiykeVfGjmiDO2pKbJ6uEe9u3SI5q5JuLIGKrsAF9aDAk0x0Eq637P TRU9kJk_sTiU6gNWw9-ax5zLvX-dlzr3nSAPbudkhuou3eb0K8q9hZR9kY9QM8YZhK7ribCW6BGrT_4XeH4efdKLtrbFhQ7iyl2xgL8gjF_f5EoyzXc5k2YyfImMpuzeuJE0a_KgOxy9-Z66eGzrchRnqJ9vZbTo42i4ywW4OIrCbI-ezYVobBfxkkJ1Uh 0Q4Q |
验证登录：使用生成的token登录
登录后提示 匿名用户没有权限，测试环境的话用下面命令解决
kubectl create clusterrolebinding test:anonymous --clusterrole=cluster-admin --user=system:anonymous
第八章 监控组件 - Heapster 部署
1、下载官方提供的 Heapster 组件部署的 yaml 文件
# 新建文件夹，用于存放 Heapster 部署所需的 yaml 文件
mkdir heapster
cd heapster
# 获取相关 yaml 文件
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
2、修改 yaml 中 image 的值
k8s.gcr.io 全部修改为 registry.cn-hangzhou.aliyuncs.com/google_containers
将所有yaml文件中出现k8s.gcr.io的替换
3、部署 Heapster
kubectl create -f heapster/
4、几分钟后，刷新 Dashboard 面板
