一、環境准備
1、三台CentOS6.7虛擬機
master：192.168.0.54 注意：主節點最好是2顆cpu，否則在k8s控制平面初始化的時候會報錯；
node1:192.168.0.68
node2:192.168.0.56
2、三台主機時間要一致
3、關閉防火牆
4、禁用swap分區
swapoff -a 臨時禁用
vim /etc/fstab 注釋掉swap的行，永久禁止
5、為三台主機添加hosts文件內容，使其能互相通過主機名訪問；
192.168.0.54 k8smaster
192.168.0.68 k8snode1
192.168.0.56 kusnode2
6、開啟ip_forword轉發
臨時生效： echo "1" > /proc/sys/net/ipv4/ip_forward
永久生效：編輯/etc/rc.d/rc.local,將echo "1" > /proc/sys/net/ipv4/ip_forward加入該文件中；
二、軟件安裝部分
1、安裝docker-ce和kubernetes的yum源，本次實驗用的是華為雲的源；
地址：https://mirrors.huaweicloud.com/
安裝方式華為雲都有說明，在這不做描述；
2、安裝軟件，master和node都需要安裝
yum install kubelet kubeadm kubectl docker-ce -y
因為master上的每個組件都是通過pod的方式來運行的，因此master上也需要部署kubelet和docker；
kubelet和docker不運行為pod，運行為系統守護進程；
三、初始化k8s控制平面
支持兩種初始化方式：指定配置文件和用命令行；本次實驗用命令行的方式初始化；初始化官方文檔地址：https://kubernetes.io/zh/docs/reference/setup-tools/kubeadm/kubeadm-init/
1、初始化
kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.14.2 --apiserver-advertise-address 192.168.0.54 --apiserver-bind-port 6443 --pod-network-cidr 10.244.0.0/16 --image-repository 選擇用於拉取控制平面鏡像的容器倉庫。默認值："k8s.gcr.io"，國內環境訪問不到，所以指向國內倉庫； --kubernetes-version 為控制平面選擇一個特定的 Kubernetes 版本，(用kubeadm version命令查看版本號) --apiserver-advertise-address API 服務器所公布的其正在監聽的 IP 地址； --apiserver-bind-port API服務器綁定的端口； --pod-network-cidr 指明 pod 網絡可以使用的 IP 地址段；這里設置10.244.0.0/16，是因為這是flannel網絡插件的默認地址段，等會要用到flannel插件，為了不引起不必要的麻煩，先這么指定；
2、遇到的警告/報錯信息
1)、文件驅動報錯 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ 原因：默認的文件驅動是cgroupfs，而docker使用的是systemd，因此要將文件驅動修改成systemd； 解決方法：(三台設備都添加一下) 1、vim /etc/docker/daemon.json
{ "exec-opts": ["native.cgroupdriver=systemd"] } 2、重啟docker服務查看docker狀態 [root@k8smaster ~]# docker info |grep Cgroup Cgroup Driver: systemd 2)、CPU數量少報錯 error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2 解決方法：添加CPU數量或者核心數(沒有測試過增加cpu核心數能否報錯，有待測試) 3)、內核參數報錯 error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1 解決方法：開啟bridge-nf-call-iptables，將0改成1； 1、vim /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 2、重載配置文件 sysctl -p /etc/sysctl.d/k8s.conf sysctl -a |grep bridge #查看更改結果
3、解決報錯信息后初始化完成的狀態
[init] Using Kubernetes version: v1.14.2 #自檢部分 [preflight] Running pre-flight checks [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 18.09 [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' #啟動kubectl [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service #生成自簽名的CA證書來為集群中的每個組件建立身份標識； [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8smaster localhost] and IPs [192.168.0.54 127.0.0.1 ::1] [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8smaster localhost] and IPs [192.168.0.54 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8smaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.54] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "sa" key and public key #將 kubeconfig 文件寫入 /etc/kubernetes/ 目錄以便 kubelet、控制器管理器和調度器用來連接到 API 服務器，它們每一個都有自己的身份標識，同時生成一個名為 admin.conf 的獨立的 kubeconfig 文件，用於管理操作。 [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file #為 API 服務器、控制器管理器和調度器生成靜態 Pod 的清單文件。 [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 22.005211 seconds [upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --experimental-upload-certs [mark-control-plane] Marking the node k8smaster as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node k8smaster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] #生成令牌，將來其他節點可以使用該令牌向控制平面注冊自己； [bootstrap-token] Using token: d9kx53.g4t2ia169zyh9byg [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: #這幾條命令在master主機上，原則上是要用普通用戶去執行，測試環境就用root用戶執行； mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: #下面這條命令是要在從節點上執行的，將從節點接入集群中，要記錄好，不能丟了，后面要用到； kubeadm join 192.168.0.54:6443 --token d9kx53.g4t2ia169zyh9byg \ --discovery-token-ca-cert-hash sha256:d8beb243d699f2cb7e5198419887441440d22722ab1cd144121a7f810cc4177a
此時，在master上執行docker image ls可以看到k8s控制平面所用的鏡像；
[root@k8smaster ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/coreos/flannel v0.14.0-rc1 0a1a2818ce59 3 weeks ago 67.9MB registry.aliyuncs.com/google_containers/kube-proxy v1.14.2 5c24210246bb 24 months ago 82.1MB registry.aliyuncs.com/google_containers/kube-apiserver v1.14.2 5eeff402b659 24 months ago 210MB registry.aliyuncs.com/google_containers/kube-controller-manager v1.14.2 8be94bdae139 24 months ago 158MB registry.aliyuncs.com/google_containers/kube-scheduler v1.14.2 ee18f350636d 24 months ago 81.6MB registry.aliyuncs.com/google_containers/coredns 1.3.1 eb516548c180 2 years ago 40.3MB registry.aliyuncs.com/google_containers/etcd 3.3.10 2c4adeb21b4f 2 years ago 258MB registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 3 years ago 742kB
1、初始化失敗時遇到的錯誤 [kubelet-check] Initial timeout of 40s passed. error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition 解決方法： swapoff -a && kubeadm reset && systemctl daemon-reload && systemctl restart kubelet && iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X 2、執行kubectl命令時出現的錯誤，例如執行(kubectl get pods) Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes") 解決方法： cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 以上問題均是原來集群出問題后，重新部署新集群的時候出現的，均是原來的配置沒有清除干凈導致的；
四、部署flannel網絡插件
部署flannel需要用到kube-flannel.yml文件，可以去github上下載，也有執行命令；地址：https://github.com/flannel-io/flannel
[root@k8smaster ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created [root@k8smaster ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-8686dcc4fd-22qbc 1/1 Running 0 4h52m coredns-8686dcc4fd-flvfx 1/1 Running 0 4h52m etcd-k8smaster 1/1 Running 0 4h51m kube-apiserver-k8smaster 1/1 Running 0 4h51m kube-controller-manager-k8smaster 1/1 Running 0 4h51m kube-flannel-ds-dbfmf 1/1 Running 0 4h36m kube-flannel-ds-gd2gw 1/1 Running 0 4h44m kube-flannel-ds-zsrjj 1/1 Running 0 4h36m kube-proxy-cr7r4 1/1 Running 0 4h36m kube-proxy-mnm49 1/1 Running 0 4h52m kube-proxy-r9g4b 1/1 Running 0 4h36m kube-scheduler-k8smaster 1/1 Running 0 4h51m
五、添加node節點，每台需要加入節點的node都需要執行
[root@k8snode1 ~]# kubeadm join 192.168.0.54:6443 --token d9kx53.g4t2ia169zyh9byg --discovery-token-ca-cert-hash sha256:d8beb243d699f2cb7e5198419887441440d22722ab1cd144121a7f810cc4177a
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 18.09
[WARNING Hostname]: hostname "k8snode2" could not be reached
[WARNING Hostname]: hostname "k8snode2": lookup k8snode2 on 114.114.114.114:53: no such host
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
在主節點上查看狀態： [root@k8smaster ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8smaster Ready master 5h7m v1.14.2 192.168.0.54 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://20.10.6 k8snode1 Ready <none> 4h51m v1.14.2 192.168.0.68 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://20.10.6 k8snode2 Ready <none> 4h50m v1.14.2 192.168.0.56 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://20.10.6
添加node節點時遇到的報錯信息及解決辦法
[root@k8snode1 ~]# kubeadm join 192.168.0.54:6443 --token d9kx53.g4t2ia169zyh9byg --discovery-token-ca-cert-hash sha256:d8beb243d699f2cb7e5198419887441440d22722ab1cd144121a7f810cc4177a [preflight] Running pre-flight checks [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 18.09 [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists [ERROR FileAvailable--etc-kubernetes-bootstrap-kubelet.conf]: /etc/kubernetes/bootstrap-kubelet.conf already exists [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1 [ERROR Swap]: running with swap on is not supported. Please disable swap [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` ##提示文件已經存在(already exists),需要將/etc/kubernetes/下的所有文件刪除再初始化 ##############
需要開啟ipv4的轉發功能 [root@k8snode1 ~]# kubeadm join 192.168.0.54:6443 --token d9kx53.g4t2ia169zyh9byg --discovery-token-ca-cert-hash sha256:d8beb243d699f2cb7e5198419887441440d22722ab1cd144121a7f810cc4177a [preflight] Running pre-flight checks [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 18.09 [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1 [ERROR Swap]: running with swap on is not supported. Please disable swap [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` [root@k8snode1 ~]# [root@k8snode1 ~]# cat /proc/sys/net/ipv4/ip_forward 0 [root@k8snode1 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward [root@k8snode1 ~]# cat /proc/sys/net/ipv4/ip_forward 1
################
需要關閉swap [root@k8snode1 ~]# kubeadm join 192.168.0.54:6443 --token d9kx53.g4t2ia169zyh9byg --discovery-token-ca-cert-hash sha256:d8beb243d699f2cb7e5198419887441440d22722ab1cd144121a7f810cc4177a [preflight] Running pre-flight checks [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 18.09 [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR Swap]: running with swap on is not supported. Please disable swap [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` [root@k8snode1 ~]# free -h total used free shared buff/cache available Mem: 1.8G 201M 821M 9.7M 796M 1.4G Swap: 2.0G 0B 2.0G
token過期的處理辦法
[root@k8snode1 ~]# kubeadm join 192.168.0.54:6443 --token d9kx53.g4t2ia169zyh9byg --discovery-token-ca-cert-hash sha256:d8beb243d699f2cb7e5198419887441440d22722ab1cd144121a7f810cc4177a
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 18.09
error execution phase preflight: couldn't validate the identity of the API Server: abort connecting to API servers after timeout of 5m0s
##此錯誤表示token過期；
[root@k8smaster ~]# kubeadm token create --print-join-command
kubeadm join 192.168.0.54:6443 --token 1axrit.s0u8ar8v0d218t0r --discovery-token-ca-cert-hash sha256:d8beb243d699f2cb7e5198419887441440d22722ab1cd144121a7f810cc4177a
#用新生成的命令去擴容node節點；
[root@k8smaster ~]# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
1axrit.s0u8ar8v0d218t0r 23h 2021-07-01T14:31:46+08:00 authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
... ...
六、以命令行的方式在集群中跑個容器測試下
1、先執行docker search nginx，選擇一個demo版本的nginx nginxdemos/hello NGINX webserver that serves a simple page co… 68 [OK] 2、在集群中運行該實例 [root@k8smaster ~]# kubectl create deployment nginx --image="nginxdemos/hello" deployment.apps/nginx created [root@k8smaster ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-dcf8cc94c-5snlw 1/1 Running 0 11m 10.244.1.2 k8snode1 <none> <none> 3、測試 curl -vo /dev/null "10.244.1.2"
七、擴展多個實例
kubectl scale deployment nginx --replicas=3 [root@k8smaster ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-dcf8cc94c-5snlw 1/1 Running 0 4h45m 10.244.1.2 k8snode1 <none> <none> nginx-dcf8cc94c-mncrp 1/1 Running 0 123m 10.244.2.2 k8snode2 <none> <none> nginx-dcf8cc94c-wc7wv 1/1 Running 0 123m 10.244.1.3 k8snode1 <none> <none>
八、創建一個service
由於在擴展多個實例的時候，指定了3個實例，那么就會存在一個問題，當我刪掉一個實例的時候，系統會自動創建一個實例，這時候就會分配一個新的ip地址，導致訪問舊ip報錯，需要連帶的替換新的ip地址，為了避免這個問題，創建一個server，類似於負載均衡的作用，在訪問的時候訪問server的地址，無論后端實例怎么變化也不會影響訪問；(kubectl delete pods 資源名稱)；
[root@k8smaster ~]# kubectl create service clusterip nginx --tcp=80:80 service/nginx created clusterip 指定類型 nginx 實例名稱，要跟之前定義的deployment名稱保持一致； [root@k8smaster ~]# kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h26m nginx ClusterIP 10.108.249.197 <none> 80/TCP 129m 測試： [root@k8smaster ~]# curl -I "10.108.249.197" HTTP/1.1 200 OK Server: nginx/1.13.8 Date: Wed, 12 May 2021 07:55:24 GMT Content-Type: text/html Connection: keep-alive Expires: Wed, 12 May 2021 07:55:23 GMT Cache-Control: no-cache