Kubernetes之（四）kubeadm部署集群
kubeadm是Kubernetes項目自帶的集群構建工具，它負責執行構建一個最小化的可用集群以及將其啟動等的必要基本步驟，簡單來講，kubeadm是Kubernetes集群全生命周期的管理工具，可用於實現集群的部署、升級/降級及拆除。
kubeadm集成了kubeadminit和kubeadmjoin等工具程序，其中kubeadminit用於集群的快速初始化，初始化，其核心功能是部署Master節點的各個組件，而kubeadmjoin則用於將節點快速加入到指定集群中，它們是創建Kubernetes集群最佳實踐的“快速路徑”。另外，kubeadmtoken可於集群構建后管理用於加入集群時使用的認證令牌（token），而kubeadmreset命令的功能則是刪除集群構建過程中生成的文件以重置回初始狀態。kubeadm還支持管理初始引導認證令牌（BootstrapToken），完成待加入的新節點首次聯系APIServer時的身份認證（基於共享密鑰）。另外，它們還支持管理集群版本的升級和降級操作。
使用kubeadm部署集群有以下幾大優勢：
- 簡單易用，kubeadm可完成集群的部署、升級和拆除操作，對新手用戶非常友好。
- 使用領域廣，支持將集群部署於裸機、VMware、AWS、Azure、GCE及更多環境的主機上，並且部署過程基本一致。
- 富有彈性：1.11版本及其以后版本的kubeadm支持階段式部署，管理員可分為多個步驟獨立操作。
- 生產環境可用，kubeadm遵循以最佳實踐的方式部署Kubernetes集群，它強制啟用RBAC，設定Master各組件間以及API Server與kubelet之間進行認證及安全通信，並鎖定了kubelet API等。
1、部署前准備
實驗環境准備
| 主機 | IP地址 | 系統版本 | Pod網段 | Services網段 | 虛擬機內存 |
|---|---|---|---|---|---|
| master | 10.0.10 | Centos 7.5 1804 | 10.244.0.0/16 | 10.96.0.0/12 | 2G |
| node01 | 10.0.11 | Centos 7.5 1804 | 10.244.0.0/16 | 10.96.0.0/12 | 2G |
| node02 | 10.0.12 | Centos 7.5 1804 | 10.244.0.0/16 | 10.96.0.0/12 | 2G |
kubernetes官方要求內存最低2G起.個人建議4G起好一些
所有機器運行
#關閉防火牆和selinux
systemctl stop firewalld &&systemctl disable firewalld &&setenforce 0
#關閉swap分區
[root@master ~]# swapoff -a
#配置syctl.conf核內核參數
[root@master ~]# cat youhua.sh
#!/bin/sh
echo "* soft nofile 190000" >> /etc/security/limits.conf
echo "* hard nofile 200000" >> /etc/security/limits.conf
echo "* soft nproc 252144" >> /etc/security/limits.conf
echo "* hadr nproc 262144" >> /etc/security/limits.conf
tee /etc/sysctl.conf <<-'EOF'
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.tcp_tw_recycle = 0
net.ipv4.ip_local_port_range = 10000 61000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.ip_forward = 1
net.core.netdev_max_backlog = 2000
net.ipv4.tcp_mem = 131072 262144 524288
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_low_latency = 0
net.core.rmem_default = 256960
net.core.rmem_max = 513920
net.core.wmem_default = 256960
net.core.wmem_max = 513920
net.core.somaxconn = 2048
net.core.optmem_max = 81920
net.ipv4.tcp_mem = 131072 262144 524288
net.ipv4.tcp_rmem = 8760 256960 4088000
net.ipv4.tcp_wmem = 8760 256960 4088000
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_syn_retries = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
EOF
echo "options nf_conntrack hashsize=819200" >> /etc/modprobe.d/mlx4.conf
modprobe br_netfilter
sysctl -p
[root@master ~]# sh youhua.sh
准備工作
**配置host解析**
[root@master ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.10 master
10.0.0.11 node01
10.0.0.12 node02
#生成公鑰並發送至node01 node02節點
[root@master ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:2sVd/Xh8vuuUOkoy+PP1VHQvsf08JuzLCWBGyfOHQPA root@master
The key's randomart image is:
+---[RSA 2048]----+
| ... |
| + . . |
| E o +|
| ..+... B+|
| S+oo..+ O|
| o+.. o ==|
| ...o o + *+|
| ..+ =.O o|
| .oo.*+=.|
+----[SHA256]-----+
[root@master ~]# ssh-copy-id node01
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'node01 (10.0.0.11)' can't be established.
ECDSA key fingerprint is SHA256:0WjkfswyQXRv+zeS03AF9xLANd4uZtFo0YcY7kGiagA.
ECDSA key fingerprint is MD5:32:e0:54:7e:8c:a0:1c:59:17:7b:00:3a:71:89:e1:a4.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@node01's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'node01'"
and check to make sure that only the key(s) you wanted were added.
[root@master ~]# ssh-copy-id node02
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'node02 (10.0.0.12)' can't be established.
ECDSA key fingerprint is SHA256:0WjkfswyQXRv+zeS03AF9xLANd4uZtFo0YcY7kGiagA.
ECDSA key fingerprint is MD5:32:e0:54:7e:8c:a0:1c:59:17:7b:00:3a:71:89:e1:a4.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@node02's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'node02'"
and check to make sure that only the key(s) you wanted were added.
配置源
#配置docker-ce的阿里源
[root@master ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
--2019-03-27 14:17:07-- https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
正在解析主機 mirrors.aliyun.com (mirrors.aliyun.com)... 183.60.159.225, 183.60.159.230, 183.60.159.232, ...
正在連接 mirrors.aliyun.com (mirrors.aliyun.com)|183.60.159.225|:443... 已連接。
已發出 HTTP 請求，正在等待回應... 200 OK
長度：2640 (2.6K) [application/octet-stream]
正在保存至: “/etc/yum.repos.d/docker-ce.repo”
100%[==============================================>] 2,640 --.-K/s 用時 0s
2019-03-27 14:17:12 (261 MB/s) - 已保存 “/etc/yum.repos.d/docker-ce.repo” [2640/2640])
#配置Kubernetes的阿里源
[root@master ~]# cat /etc/yum.repos.d/k8s.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
enabled=1
#拷貝給node節點
[root@master ~]# cd /etc/yum.repos.d/
[root@master yum.repos.d]# scp ./docker-ce.repo ./k8s.repo node01:/etc/yum.repos.d/
docker-ce.repo 100% 2640 3.4MB/s 00:00
k8s.repo 100% 202 311.7KB/s 00:00
[root@master yum.repos.d]# scp ./docker-ce.repo ./k8s.repo node02:/etc/yum.repos.d/
docker-ce.repo 100% 2640 2.8MB/s 00:00
k8s.repo 100% 202 252.1KB/s 00:00
#重建yum緩存
[root@master yum.repos.d]# yum makecache fast
已加載插件：fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
base | 3.6 kB 00:00:00
docker-ce-stable | 3.5 kB 00:00:00
extras | 3.4 kB 00:00:00
kubernetes | 1.4 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/2): kubernetes/primary | 47 kB 00:00:05
(2/2): updates/7/x86_64/primary_db | 3.4 MB 00:00:08
kubernetes 336/336
元數據緩存已建立
安裝docker-ce和kubetnetes所需軟件
目前 kubernetes官方支持的docker-ce最高版本是18.06，親測18.09也可以使用，初始化會提示warnning，但是不會報錯，運行也無異常；這里kubelet，kubeadm，kubectl使用1.13.1，docker使用18.06。
| 節點 | 部署軟件及版本 |
|---|---|
| master | docker-ce 18.06、kubelet 1.13.1、kubeadm 1.13.1、kubectl 1.13.1 |
| node01 | docker-ce 18.06、kubelet 1.13.1、kubeadm 1.13.1 |
| node02 | docker-ce 18.06、kubelet 1.13.1、kubeadm 1.13.1 |
#master
[root@master ~]# yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1 docker-ce-18.06.3.ce-3.el7
[root@master ~]# systemctl start docker kubelet &&systemctl enable docker kubelet
#node01 node02
[root@node01 ~]# yum install -y kubelet-1.13.1 kubeadm-1.13.1 docker-ce-18.06.3.ce-3.el7
[root@node01 ~]# systemctl start docker kubelet &&systemctl enable docker kubelet
#到此，准備工作完成
2、集群初始化
所有機器都需要初始化容器執行引擎（如docker或frakti等） 和kubelet。這是因為kubeadm依賴kubelet來啟動Master組件，比如kube-apiserver、kube-managercontroller、kube-scheduler、kube-proxy等。
master初始化創建集群
[root@master ~]# kubeadm init \
--kubernetes-version=v1.13.1 \
--pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12 \
--image-repository registry.aliyuncs.com/google_containers \
--apiserver-advertise-address=0.0.0.0 \
--ignore-preflight-errors=Swap
[init] Using Kubernetes version: v1.13.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.0.10]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [10.0.0.10 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [10.0.0.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.543677 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master" as an annotation
[mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: jffthg.hb83xpoxjcm5vh54
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 10.0.0.10:6443 --token jffthg.hb83xpoxjcm5vh54 --discovery-token-ca-cert-hash sha256:37c446787d8ff4383a50dc5855afb8e307f4d1fff5e5cdbb9235b8af6e49e28f
上面命令的選項決定了集群環境的眾多特性設定，這些設定對於伺候在集群中部署運行應用程序很重要。
--kubernetes-version：正在使用的kubetnetes程序組建的版本號，需要與kubelet版本相同
--pod-network-cidr: Pod的地址范圍，其值為CIDR格式的網絡地址；使用flannel網絡插件時，其默認地址為10.244.0.0/16.
--service-cidr: Service網絡地址范圍，，其值為CIDR格式的網絡地址；其默認地址為10.96.0.0/16.
--apiserver-advertise-address： APIserver通告給其他組件的IP地址，默認為0.0.0.0，表示節點上所有可用的地址
--ignore-preflight-errors=Swap: 忽略哪些運行時的錯誤，此時表示忽略swap未關閉導致的錯誤
--image-repository registry.aliyuncs.com/google_containers 表示選擇拉取初始化所需鏡像的倉庫，kubernetes的倉庫在國外，國內拉取可能失敗，1.13版本支持該選項選擇倉庫，此倉庫為阿里雲的倉庫
設定kubectl的配置文件
kubectl是執行kubernetes集群管理的核心工具，默認情況kubectl會從當前用戶主目錄的隱藏目錄.kube下名為config的配置文件讀取配置，包括要接入的集群，以及用於集群認證的證書或令牌等。初始化集群時，kubeadm會自動生成一個用於此功能的配置文件/et/kubernetes/admin.conf，將它復制到$HOME/.kube/config，這里master節點root用戶為例，生產用應該用普通用戶身份進行。
[root@master ~]# mkdir -p $HOME/.kube
[root@master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
到此，集群的master節點已經基本配置完成，可以使用API server來驗證各組件是否正常,必要時可使用kubeadm reset命令重置之后重新初始化集群。
componentstatus 縮寫為cs
[root@master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
node加入集群
[root@node01 ~]# kubeadm join 10.0.0.10:6443 --token jffthg.hb83xpoxjcm5vh54 --discovery-token-ca-cert-hash sha256:37c446787d8ff4383a50dc5855afb8e307f4d1fff5e5cdbb9235b8af6e49e28f
[preflight] Running pre-flight checks
[WARNING Hostname]: hostname "node01" could not be reached
[WARNING Hostname]: hostname "node01": lookup node01 on 114.114.114.114:53: no such host
[discovery] Trying to connect to API Server "10.0.0.10:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.0.10:6443"
[discovery] Requesting info from "https://10.0.0.10:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.0.0.10:6443"
[discovery] Successfully established connection with API Server "10.0.0.10:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node01" as an annotation
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
master上查詢集群狀態
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady master 10m v1.13.1
node01 NotReady <none> 114s v1.13.1
node02 NotReady <none> 107s v1.13.1
此時狀態為Notready，因為此時沒有網絡CNI，需要加載falnnel或者calico網絡插件，此處以flannel插件為例
[root@master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created
#稍等幾分鍾
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 39m v1.13.1
node01 Ready <none> 30m v1.13.1
node02 Ready <none> 30m v1.13.1
Kubernetes集群以及部署的插件提供了多種不同的服務 如此前部署過的API Server、kube-dns等。API客戶端訪問集群時需要事先知道API Server的通告地址，管理員可使用“kubectl cluster-info”命令了解到這些信息：
[root@master ~]# kubectl cluster-info
Kubernetes master is running at https://10.0.0.10:6443
KubeDNS is running at https://10.0.0.10:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Kubernetes集群server和client版本信息查詢，可使用kubectl version命令查看:
[root@master ~]# kubectl version --short=true
Client Version: v1.13.1
Server Version: v1.13.1
從集群中移除節點：
運行過程中，若有節點需要從正常運行得集群中移除，則可使用以下步驟：
1） 在master節點使用如下命令排干 當前節點上得Pod資源並移除Node節點：
~]# kubectl drain node01 --delete-local-data --force --ignire-daemonsets
~]# kubectl delete node node01
2)而后在需要刪除得Node上執行如下命令重置系統狀態即可完成移除操作：
~]# kubeadm reset
根據官方說明tonken的默認有效時間為24h，由於時間差，導致這里的token失效，可以使用kubeadm token list查看token，發現之前初始化的tonken已經失效了
[root@master ~]# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
jffthg.hb83xpoxjcm5vh54 7h 2019-03-28T16:03:51+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
那么此處需要重新生成token，生成的方法如下：
[root@master ~]# kubeadm token create
如果沒有值--discovery-token-ca-cert-hash，可以通過在master節點上運行以下命令鏈來獲取：
[root@master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
openssl dgst -sha256 -hex | sed 's/^.* //'
此時，再運行kube join命令將新增node加入到集群當中，此處的--discovery-token-ca-cert-hash依舊可以使用初始化時的證書
kubeadm join 192.168.56.11:6443 --token 1*** --discovery-token-ca-cert-hash
到此kubernetes集群部署完成
參考資料
馬永亮. Kubernetes進階實戰 (雲計算與虛擬化技術叢書)
https://www.cnblogs.com/linuxk/