kubeadm 部署kubernetes1.14

節點信息：

主機名	IP	角色
k8s-master	10.10.0.10	master節點
k8s-node01	10.10.0.11	集群worke節點
k8s-node02	10.10.0.12	集群worke節點

 一、初始化系統環境

系統初始化環境配置，需要在三台服務器都執行，以下命令請看對應的命令參數，在不同服務器執行。

1）主機名

分別在三台機器執行對應命令，設置主機名

[root@centos01 ~]# hostnamectl set-hostname k8s-master
[root@centos02 ~]# hostnamectl set-hostname k8s-node01
[root@centos03 ~]# hostnamectl set-hostname k8s-node02

2）關閉防火牆、selinux

[root@centos01 ~]# systemctl disable firewalld
[root@centos01 ~]# systemctl stop firewalld
[root@centos01 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

3）設置內核

[root@centos01 ~]# vim /etc/sysctl.d/k8s.conf
## 添加如下內容
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
## 執行命令生效
[root@centos01 ~]# modprobe br_netfilter
[root@centos01 ~]# sysctl -p /etc/sysctl.d/k8s.conf

4）免密鑰

在master節點進行對2台node節點進行免密鑰登陸

[root@k8s-master ~]# ssh-keygen
[root@k8s-master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.10.0.11
[root@k8s-master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.10.0.12

 二、部署kubernetes集群

1）配置yum源

yum源三台機器都需要配置，這里我們以master主機為例，node節點也按照此yum配置即可

[root@k8s-master ~]# yum -y install wget
[root@k8s-master ~]# cd /etc/yum.repos.d
## 配置docker-ce源 [root@k8s-master yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
## 配置kubernetes源
[root@k8s-master yum.repos.d]# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1   
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
enabled=1 ## 下載校驗文件
[root@k8s-master ~]# wget https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
[root@k8s-master ~]# wget https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
## 導入校驗文件
[root@k8s-master ~]# rpm --import rpm-package-key.gpg
[root@k8s-master ~]# rpm --import yum-key.gpg
[root@k8s-master ~]# yum clean all  && yum makecache fast

2）安裝docker、kubelet、kubeadm等工具

[root@k8s-master ~]# yum  install kubelet-1.14.1 kubeadm-1.14.1 kubectl-1.14.1 docker-ce -y
## node節點無需配置kubctl組件
[root@k8s-node01 ~]# yum  install kubelet-1.14.1 kubeadm-1.14.1 docker-ce -y
[root@k8s-node02 ~]# yum  install kubelet-1.14.1 kubeadm-1.14.1 docker-ce -y

3）配置docker kubelet

2台node節點也需要修改對應配置

## 設置kubelet啟動時忽略swap報錯
[root@k8s-master ~]# vim /etc/sysconfig/kubelet 
 KUBELET_EXTRA_ARGS="--fail-swap-on=false"
## 設置開機自啟動
[root@k8s-master ~]# systemctl daemon-reload 
[root@k8s-master ~]# systemctl enable docker && systemctl restart docker
[root@k8s-master ~]# systemctl enable kubelet && systemctl restart kubelet

4）初始化master節點

kubernetes從1.13版本開始，可以指定鏡像倉庫進行集群初始化操作，所以我們直接指定阿里雲鏡像倉庫進行集群初始化，這樣無需再關心國內網絡環境是否可以下載到對應的官方原始pod鏡像

[root@k8s-master ~]# swapoff -a
[root@k8s-master ~]# kubeadm init --apiserver-advertise-address=10.10.0.10 \
 --image-repository registry.aliyuncs.com/google_containers \
 --kubernetes-version v1.14.1 \
 --pod-network-cidr=10.244.0.0/16

參數解釋：

• apiserver-advertise-address：apiserver地址及master節點地址
• image-repository：鏡像倉庫地址
• pod-network-cidr：pod網絡

以上初始化，需要等待一段時間，因為需要下載對應組件鏡像，master節點初始化完成后，可看到以下提示信息：

........
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.0.10:6443 --token 5ti5kd.o32bm9lofv6zej94 \
    --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f

注意：

    token是node節點加入時需要用到的信息，需要記錄下來，tocken值24小時后失效，若果超過24小時你再進行node節點加入集群，需要重新生成tocken。

5）配置集群環境變量

用戶想要使用kubectl操作集群，則需要配置kubectl環境變量，這些命令也是上面kubeadm init后輸出的內容

[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# chown $(id -u):$(id -g) $HOME/.kube/config
## 查看集群信息
[root@k8s-master ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}
## 查看node就緒狀態
### 由於還沒有安裝網絡插件，以及node節點未加入集群，所以只顯示一個master節點信息
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   NotReady   master   7m33s   v1.14.1

 三、部署網絡插件

集群的運行依賴於網絡，k8s本身並不支持網絡，需要額外部署對應的網絡插件，才可實現集群的個組件網絡通信。我們這里采用flannel作為集群網絡插件。

項目地址：https://github.com/coreos/flannel

## 在線部署
[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
## 或者把清單配置文件下載本地再執行清單文件應用
[root@k8s-master ~]# mkdir /opt/k8s/flannel
[root@k8s-master ~]# cd /opt/k8s/flannel
[root@k8s-master ~]# wget  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[root@k8s-master ~]# kubectl apply -f kube-flannel.yml

四、node節點加入集群

1）加入集群

node節點加入集群操作基本一致，這里以其中一台為例。kubeadm join命令為master節點初始化成功后顯示的命令，上文已經提到過。

[root@k8s-node01 ~]# swapoff -a
[root@k8s-node01 ~]# kubeadm join 10.10.0.10:6443 --token 5ti5kd.o32bm9lofv6zej94 \
     --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f

           [preflight] Running pre-flight checks 
          
 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ 
          
 [WARNING Hostname]: hostname "k8s-node01" could not be reached 
          
 [WARNING Hostname]: hostname "k8s-node01": lookup k8s-node01 on 114.114.114.114:53: no such host 
          
[preflight] Reading configuration from the cluster... 
          
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' 
          
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace 
          
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 
          
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 
          
[kubelet-start] Activating the kubelet service 
          
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 
         

 
          This node has joined the cluster: 
          
* Certificate signing request was sent to apiserver and a response was received. 
          
* The Kubelet was informed of the new secure connection details. 
         

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

注意：

　　node節點初始化加入集群，會看到對應的初始化成功信息，初始化時由於需要下載對應pod鏡像，需要等待一段時間，node才會成功加入。

2）查看節點就緒狀態

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
k8s-master   Ready    master   139m   v1.14.1
k8s-node01   Ready    <none>   2m8s   v1.14.1
k8s-node02   Ready    <none>   64s    v1.14.1

3）集群刪除

以上就是kubeadm部署集群過程，過程其實並不復雜，如果在集群部署過程中，出現問題，想要刪除重新初始化集群，使用以下命令：

kubeadm reset
ifconfig cni0 down && ip link delete cni0
ifconfig flannel.1 down && ip link delete flannel.1
rm -rf /var/lib/cni/

特別說明：

　　1）關於token失效

　　上面說過，master集群初始化后，token24小時后就會失效，如果到了token失效時間，node再加入集群，需要重新生產token：

## 查看token狀態
### TTL值 就是token生於時間
[root@k8s-master ~]# kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
5ti5kd.o32bm9lofv6zej94   21h       2019-05-22T11:16:31+08:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
## 重新生產token
[root@k8s-master ~]# kubeadm token create
　　hb0mhv.ckb79uumxh06br8e
## 獲取--discovery-token-ca-cert-hash值
[root@k8s-master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'
cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f
## 加入集群命令
[root@k8s-master ~]# kubeadm join 10.10.0.10:6443 --token hb0mhv.ckb79uumxh06br8e \
    --discovery-token-ca-cert-hash sha256:cd778ad01bdbc656eaff7d3b1273691f0070ebbadd2f1b8a3189a6dc1e88f39f