kubeadm部署K8S

一、K8S的搭建

0、准備

網絡地址：
節點網絡：10.201.106.0/24
Service網絡：10.96.0.0/12
Pod網絡（flannel默認網段）：10.244.0.0/16

節點功能：
master1（10.201.106.131）：master節點，etcd
master2（10.201.106.132）：node1節點
master3（10.201.106.133）：node2節點

預配置：
1、NTP時間同步
2、基於主機名通信：/etc/hosts
3、關閉firewalld和iptables.service
4、所有節點關閉swap，打開內生橋接功能，需要重啟：
[root@master1 ~]# vim  /usr/lib/sysctl.d/00-system.conf

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1


備注：通過微軟鏡像源，快速拉取鏡像，小工具azk8spull安裝
git clone https://github.com/xuxinkun/littleTools
cd littleTools
chmod +x install.sh
./install.sh
source /etc/profile

0.1 設置阿里雲鏡像

設置docker鏡像源：
地址：https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

master1設置docker鏡像源：
[root@master1 yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo


設置k8s鏡像源：
master1設置K8S鏡像
[root@master1 yum.repos.d]# vim kubernetes.repo

[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
enabeld=1

0.2 將master上的docker和k8s鏡像源文件拷貝到兩個node節點

[root@master1 yum.repos.d]# scp docker-ce.repo kubernetes.repo master2:/etc/yum.repos.d/

[root@master1 yum.repos.d]# scp docker-ce.repo kubernetes.repo master3:/etc/yum.repos.d/

1、安裝K8S

1.1 master節點安裝docker和k8s的rpm包

默認安裝最新版本：
[root@master1 ~]# yum install docker-ce kubeadm kubectl kubelet

如需指定版本：
[root@master1 ~]# yum install docker-ce-18.06.3.ce-3.el7 kubeadm-1.13.6-0 kubectl-1.13.6-0 kubelet-1.13.6-0

設置服務自啟動：
[root@master1 ~]# systemctl enable docker
[root@master1 ~]# systemctl enable kubelet

1.2 master節點docker設置

設置服務啟動腳本（設置鏡像源）
[root@master1 ~]# vim /usr/lib/systemd/system/docker.service

# for containers run by docker
Environment="HTTPS_PROXY=http://www.ik8s.io:10080"
Environment="NO_PROXY=127.0.0.0/8,10.201.106.0/24"

啟動docker服務：
[root@master1 ~]# systemctl daemon-reload
[root@master1 ~]# systemctl start docker

查看設置
[root@master1 ~]# docker info | grep Proxy
HTTPS Proxy: http://www.ik8s.io:10080
No Proxy: 127.0.0.0/8,10.201.106.0/24

如代理無法下載，使用微軟鏡像下載：
kubeadm config images list
azk8spull k8s.gcr.io*

1.3 master節點初始化

查看kubelet生成文件：
[root@master1 ~]# rpm -ql kubelet

查看kudeadm幫助：
[root@master1 ~]# kubeadm init --help

忽略swap報錯設置（若已關閉swap可不做）：
[root@master1 ~]# vim /etc/sysconfig/kubelet 

KUBELET_EXTRA_ARGS="--fail-swap-on=false"

重要：開始初始化master節點：
[root@master1 ~]# kubeadm init --kubernetes-version=v1.14.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap


（記錄下來）得到輸出，用於node節點加入主節點認證：
kubeadm join 10.201.106.131:6443 --token 27ojcb.sc3hc3e43rru0zu5 \
--discovery-token-ca-cert-hash sha256:9b68cf142c04ba3508af6fe7fb89fe268ddc751b37ff435669f39e7906e76e

重要：拷貝配置文件並更改權限（該配置文件包含認證證書信息）：
[root@master1 ~]# mkdir -p $HOME/.kube
[root@master1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config

查看信息：
查看組件狀態信息
[root@master1 ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
[root@master1 ~]# kubectl get componentstatus

查看集群節點信息：
[root@master1 ~]# kubectl get nodes
NAME          STATUS     ROLES    AGE   VERSION
master1.com   NotReady   master   82m   v1.14.2

1.4 設置網絡插件

下載flannel（基於清單下載部署，默認下載最新版本）：
地址：https://github.com/coreos/flannel

部署flannel
[root@master1 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

等待初始化完成，可以看到pod節點：
[root@master1 ~]# kubectl get nodes
NAME          STATUS   ROLES    AGE    VERSION
master1.com   Ready    master   119m   v1.14.2
查看kube-system名稱空間：
[root@master1 ~]# kubectl get pods -n kube-system
NAME                                  READY   STATUS    RESTARTS   AGE
coredns-fb8b8dccf-8ms4d               1/1     Running   0          120m
coredns-fb8b8dccf-wb97j               1/1     Running   0          120m
etcd-master1.com                      1/1     Running   0          119m
kube-apiserver-master1.com            1/1     Running   0          119m
kube-controller-manager-master1.com   1/1     Running   0          119m
kube-flannel-ds-amd64-z7vjw           1/1     Running   0          17m
kube-proxy-xkhhn                      1/1     Running   0          120m
kube-scheduler-master1.com            1/1     Running   0          119m

查看集群中名稱空間狀態：
[root@master1 ~]# kubectl get ns
NAME              STATUS   AGE
default           Active   122m
kube-node-lease   Active   122m
kube-public       Active   122m
kube-system       Active   122m

1.5 初始化Node節點

從master1復制docker服務文件覆蓋master2,3節點：
[root@master1 ~]# scp /usr/lib/systemd/system/docker.service master2:/usr/lib/systemd/system/docker.service
[root@master1 ~]# scp /usr/lib/systemd/system/docker.service master3:/usr/lib/systemd/system/docker.service

從master1復制kubelet配置文件：
[root@master1 ~]# scp /etc/sysconfig/kubelet master2:/etc/sysconfig/
[root@master1 ~]# scp /etc/sysconfig/kubelet master3:/etc/sysconfig/

node1配置：
安裝docker和kubelet：
[root@master2 ~]# yum install docker-ce kubelet kubeadm
[root@master2 ~]# systemctl enable docker
[root@master2 ~]# systemctl enable kubelet

node1啟動docker服務：
[root@master2 ~]# systemctl daemon-reload
[root@master2 ~]# systemctl restart docker


node1加入k8s集群（如果網絡無法下載鏡像，通過微軟鏡像（azk8spull）導入kube-proxy，flannel，pause三個鏡像）
[root@master2 ~]# kubeadm join 10.201.106.131:6443 --token 27ojcb.sc3hc3e43rru0zu5 \
    --discovery-token-ca-cert-hash sha256:9b68cf142c04ba3508af6fe7fb89fe268ddc751b37ff435669f39e7906e76ea1


等待node1節點下載好kube-proxy，flannel，pause三個鏡像並啟動容器后在master節點查看狀態：
[root@master1 ~]# kubectl get node
NAME          STATUS   ROLES    AGE     VERSION
master1.com   Ready    master   4h24m   v1.14.2
master2.com   Ready    <none>   9m27s   v1.14.2


查看pods：
[root@master1 ~]# kubectl get pods -n kube-system -o wide
NAME                                  READY   STATUS    RESTARTS   AGE     IP               NODE          NOMINATED NODE   READINESS GATES
coredns-fb8b8dccf-8ms4d               1/1     Running   3          4h25m   10.244.0.6       master1.com   <none>           <none>
coredns-fb8b8dccf-wb97j               1/1     Running   2          4h25m   10.244.0.5       master1.com   <none>           <none>
etcd-master1.com                      1/1     Running   1          4h25m   10.201.106.131   master1.com   <none>           <none>
kube-apiserver-master1.com            1/1     Running   1          4h25m   10.201.106.131   master1.com   <none>           <none>
kube-controller-manager-master1.com   1/1     Running   1          4h25m   10.201.106.131   master1.com   <none>           <none>
kube-flannel-ds-amd64-qr4p6           1/1     Running   0          11m     10.201.106.132   master2.com   <none>           <none>
kube-flannel-ds-amd64-z7vjw           1/1     Running   2          163m    10.201.106.131   master1.com   <none>           <none>
kube-proxy-f86s9                      1/1     Running   0          11m     10.201.106.132   master2.com   <none>           <none>
kube-proxy-xkhhn                      1/1     Running   2          4h25m   10.201.106.131   master1.com   <none>           <none>
kube-scheduler-master1.com            1/1     Running   1          4h25m   10.201.106.131   master1.com   <none>           <none>


node2加入步驟跟node1一樣：
[root@master3 ~]# kubeadm join 10.201.106.131:6443 --token 27ojcb.sc3hc3e43rru0zu5     --discovery-token-ca-cert-hash sha256:9b68cf142c04ba3508af6fe7fb89fe268ddc751b37ff435669f39e7906e76ea1 --ignore-preflight-errors=Swap

2、K8S應用基礎入門

2.1 查看

查看節點詳細描述信息：
[root@master1 ~]# kubectl describe node master1.com

查看主節點和工作節點版本：
[root@master1 ~]# kubectl version

查看整個集群信息：
[root@master1 ~]# kubectl cluster-info

2.2 部署應用

2.2.1 創建一個nginx（Pod）

[root@master1 ~]# kubectl run --help

嘗試創建一個nginx：
[root@master1 ~]# kubectl run nginx-deploy --image=nginx:1.14-alpine --port=80 --replicas=1 --dry-run=true

真正創建一個nginx：
[root@master1 ~]# kubectl run nginx-deploy --image=nginx:1.14-alpine --port=80 --replicas=1

查看當前系統上已經創建的deployment：
[root@master1 ~]# kubectl get deployment
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deploy   1/1     1            1           23m


查看Pod：
[root@master1 ~]# kubectl  get pods
NAME                           READY   STATUS    RESTARTS   AGE
nginx-deploy-55d8d67cf-r2rwn   1/1     Running   0          23m

查看屬於哪個節點：
[root@master1 ~]# kubectl  get pods -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
nginx-deploy-55d8d67cf-r2rwn   1/1     Running   0          24m   10.244.1.5   master2.com   <none>           <none>

其它報錯處理

問題1：代理無法拉取鏡像

代理如果無法使用，手動拉鏡像:
https://blog.csdn.net/sjyu_ustc/article/details/79990858
https://blog.51cto.com/liuzhengwei521/2301497
https://www.cnblogs.com/zxy860320/p/9996109.html
https://blog.csdn.net/Mr_rsq/article/details/84943480
https://blog.51cto.com/purplegrape/2315451

拉鏡像：
docker pull mirrorgooglecontainers/kube-apiserver:v1.13.6
docker pull mirrorgooglecontainers/kube-controller-manager:v1.13.6
docker pull mirrorgooglecontainers/kube-scheduler:v1.13.6
docker pull mirrorgooglecontainers/kube-proxy:v1.13.6
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd:3.2.24
docker pull coredns/coredns:1.2.6
docker pull quay.io/coreos/flannel:v0.10.0-amd64

打標：
docker tag mirrorgooglecontainers/kube-apiserver:v1.13.6 k8s.gcr.io/kube-apiserver:v1.13.6
docker tag mirrorgooglecontainers/kube-controller-manager:v1.13.6 k8s.gcr.io/kube-controller-manager:v1.13.6
docker tag mirrorgooglecontainers/kube-scheduler:v1.13.6 k8s.gcr.io/kube-scheduler:v1.13.6
docker tag mirrorgooglecontainers/etcd:3.2.24 k8s.gcr.io/etcd:3.2.24                                                                                                    docker tag coredns/coredns:1.2.6 k8s.gcr.io/coredns:1.2.6
docker tag anjia0532/pause:3.1 k8s.gcr.io/pause:3.1
docker tag mirrorgooglecontainers/kube-proxy:v1.13.6 k8s.gcr.io/kube-proxy:v1.13.6

拉取：
kubeadm config images list |sed -e 's/^/docker pull /g' -e 's#k8s.gcr.io#mirrorgooglecontainers#g' | sh -x

docker pull coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

打標：
docker images |grep mirrorgooglecontainers |awk '{print "docker tag",$1":"$2,$1":"$2}' |sed -e 's#mirrorgooglecontainers#k8s.gcr.io#2' |sh -x

docker tag coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

刪除舊鏡像：
docker images |grep mirrorgooglecontainers |awk '{print "docker rmi -f", $1":"$2}' |sh -x

問題2：docker存儲報錯處理鏈接：

https://cloud.tencent.com/info/bfe4fa029038643970af08b6f13624c8.html

問題3：cgroup不一致報錯（如果無法解決，建議使用centos7.4以上系統，最新系統更好）：

https://blog.csdn.net/Andriy_dangli/article/details/85062983

[root@master2 ~]# vi /etc/docker/daemon.json 
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

[root@master2 ~]# vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --cgroup-driver=cgroupfs"

通過阿里雲鏡像下載K8S鏡像

https://blog.csdn.net/zsd498537806/article/details/85157560