k8s集群搭建

三台Linux系統搭建K8S集群

參考博客

服務器配置如下：

hostname	Server IP	centos version	docker version	k8s  version
master	192.168.72.131	7.7.1908	19.03.8	1.18.2
node2	192.168.72.132	7.7.1908	19.03.8	1.18.2
node3	192.168.72.133	7.7.1908	19.03.8	1.18.2

基本配置

1、host配置：

#分別在服務器上修改hostname，使用hostnamectl命令，或者直接修改 /etc/hostname 文件 hostnamectl --static set-hostname master hostnamectl --static set-hostname node2 hostnamectl --static set-hostname node3 #每台機器都執行 cat >> /etc/hosts << EOF 192.168.72.131 master 192.168.72.132 node2 192.168.72.133 node3 EOF

2、關閉防火牆：

systemctl stop firewalld & systemctl disable firewalld

3、關閉swap

#臨時關閉
swapoff -a    

#永久關閉,重啟后生效
vi /etc/fstab
/dev/mapper/centos-swap swap ...     #注釋以下代碼

4、關閉selinux

#獲取狀態 getenforce 
#暫時關閉 setenforce 0 
#永久關閉 需重啟 vim /etc/sysconfig/selinux SELINUX=disabled   #修改以下參數，設置為disable

5、修改網絡配置

# 所有機器上都要進行 cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system

6、統一時間【如果需要】

#統一時區，為上海時區 ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime bash -c "echo 'Asia/Shanghai' > /etc/timezone" #統一使用阿里服務器進行時間更新 yum install -y ntpdate #安裝ntpdate工具 ntpdate ntp1.aliyun.com #更新時間

安裝docker

1、刪除原有的docker組件

yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-selinux \ docker-engine-selinux \ docker-engine

2、配置系統docker源

yum install -y yum-utils device-mapper-persistent-data lvm2 
# 注意：此處更換了阿里的源，適用國內用戶 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

3、查看docker安裝列表，選擇並安裝

yum list docker-ce --showduplicates | sort -r # 此處直接安裝最新版本的docker-ce yum install -y docker-ce # 注：如果要安裝指定的版本可以參考下邊的命令 yum install -y docker-ce-3:19.03.8-3.el7.x86_64 

4、啟動docker

systemctl enable docker && systemctl start docker

5、更換鏡像倉庫源 阿里雲docker倉庫

# 進入阿里雲帳號，依次進入：控制台 --> 容器鏡像服務(可以搜索到) --> 鏡像中心 --> 鏡像加速器;
  鏡像加速器中獲取到加速器地址: "https://xxxxxxx.mirror.aliyuncs.com"

如下圖所示：https://cr.console.aliyun.com/cn-hangzhou/instances/mirrors

# linux下默認文件為/etc/docker/daemon.json，添加下列倉庫

cat > /etc/docker/daemon.json <<EOF { "registry-mirrors": ["https://79e563fi.mirror.aliyuncs.com"], "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ] } EOF

# 或者

tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://79e563fi.mirror.aliyuncs.com"]
}
EOF

# 重啟docker使其生效
systemctl daemon-reload  && systemctl restart docker

安裝K8S組件：

1、更新K8S源(所有節點)

# 訪問此地址
https://developer.aliyun.com/mirror/ https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.3e221b11QNoepV

# 寫入加速文件 cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF 
# 安裝組件 yum install -y kubelet kubeadm kubectl

 # 啟動kubelet 服務
 systemctl enable kubelet && systemctl start kubelet(maste節點未初始化完成之前，啟動kubelet會出現異常)

 參考：kubelet啟動異常   啟動失敗 

以上步驟每個node都需要執行

 

配置master服務器

配置k8s初始化文件

k8s config 命令參考

1、master節點下生成默認配置文件

kubeadm config print init-defaults > init-kubeadm.conf

2、修改init-kubeadm.conf 主要參數

# localAPIEndpointc，advertiseAddress為master-ip ，port默認不修改 localAPIEndpoint: advertiseAddress: 192.168.56.101 #此處為master的IP bindPort: 6443 #imageRepository: k8s.gcr.io #更換k8s鏡像倉庫 imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers # kubernetesVersion: v1.18.0  #修改為版本 v1.18.2 kubernetesVersion: v1.18.2 # 配置子網絡 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.246.0.0/16    #添加pod子網絡,使用的是flannel網絡

3、拉取下載k8s組件

# 查看安裝時需要的鏡像文件列表
kubeadm config images list --config init-kubeadm.conf 
# 更換k8s鏡像倉庫之前，即 imageRepository: k8s.gcr.io k8s.gcr.io/kube-apiserver:v1.18.2 k8s.gcr.io/kube-controller-manager:v1.18.2 k8s.gcr.io/kube-scheduler:v1.18.2 k8s.gcr.io/kube-proxy:v1.18.2 k8s.gcr.io/pause:3.2 k8s.gcr.io/etcd:3.4.3-0 k8s.gcr.io/coredns:1.6.7 # 更換k8s鏡像倉庫之后，即 imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.2 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.2 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.2 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.2 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0 registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7

 # 根據配置文件進行鏡像下載，這里使用鏡像倉庫 --> imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

 kubeadm config images pull --config init-kubeadm.conf

 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.2
 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.2
 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.2
 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.2
 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7

4、初始化k8s

# init # 如果kubeadm init過了，此時需要加個參數來忽略到這些：--ignore-preflight-errors=all kubeadm init --config init-kubeadm.conf

[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7
[root@master ~]# kubeadm init --config init-kubeadm.conf
W0506 23:11:46.807147   11111 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.72.131]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [192.168.72.131 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [192.168.72.131 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0506 23:11:53.514098   11111 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0506 23:11:53.515754   11111 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.505498 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.72.131:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:4082a7afce070910097c7926919f404cd25eb3be614598139704fd0367149aee 

 # 啟動后可以根據提示執行下列命令，並記錄john token

Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/
 Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.72.131:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:4082a7afce070910097c7926919f404cd25eb3be614598139704fd0367149aee 

# 此處要記錄下 join語句，如果join token忘記，則需要執行下邊命令重新生成 kubeadm token create --print-join-command

# k8s 其他節點加入集群(此步驟先不要執行)
kubeadm join 192.168.72.131:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:4082a7afce070910097c7926919f404cd25eb3be614598139704fd0367149aee 

#啟動kubelet
systemctl enable kubelet && systemctl start kubelet

 5、查看master節點初始化情況

# 查看啟動狀態 # kubectl get nodes NAME STATUS ROLES AGE VERSION master NotReady master 16m v1.18.2 //NotReady狀態是因為缺少flannel或者Calico這樣的網絡組件 
# 查看組件狀態
# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0               Healthy   {"health":"true"} 
# 查看具體初始化情況 # kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system   coredns-546565776c-76xsx         0/1     Pending   0 16m //需要安裝網絡插件 kube-system   coredns-546565776c-gkbgv         0/1     Pending   0 16m kube-system   etcd-master                      1/1     Running   0 16m kube-system   kube-apiserver-master            1/1     Running   0 16m kube-system   kube-controller-manager-master   1/1     Running   0 16m kube-system   kube-proxy-9889w                 1/1     Running   0 16m kube-system   kube-scheduler-master            1/1     Running   0          16m

 6、安裝網絡插件flannel

獲取flannel網絡組件：https://github.com/coreos/flannel

1、 在線獲取部署清單，並基於此清單下載鏡像，啟動並部署flannel 

2、flannel默認的網段是10.244.0.0/16，如果想修改為自己特定的網段，可以先下載yaml文件，修改網段
# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# vim kube-flannel.yml

  net-conf.json: |
    {
      "Network": "10.246.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

# kubectl apply -f kube-flannel.yml

 

在線部署flannel 

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
 podsecuritypolicy.policy/psp.flannel.unprivileged created
 clusterrole.rbac.authorization.k8s.io/flannel created
 clusterrolebinding.rbac.authorization.k8s.io/flannel created
 serviceaccount/flannel created
 configmap/kube-flannel-cfg created
 daemonset.apps/kube-flannel-ds-amd64 created
 daemonset.apps/kube-flannel-ds-arm64 created
 daemonset.apps/kube-flannel-ds-arm created
 daemonset.apps/kube-flannel-ds-ppc64le created
 daemonset.apps/kube-flannel-ds-s390x created

 # 正在下載flannel鏡像

 

 # flannel鏡像下載完成並加載后

 

配置node

node節點 初始化內容

# 拷貝 master 機器上 $HOME/.kube/config 到各node節點上 scp $HOME/.kube/config root@node2:~/ scp $HOME/.kube/config root@node3:~/ # 分別在node2和node3上執行下邊命令 # 不然執行kubectl 會報錯 mkdir -p $HOME/.kube mv $HOME/config $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config

node節點注冊master

# 直接使用指令加入
kubeadm join 192.168.72.131:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:4082a7afce070910097c7926919f404cd25eb3be614598139704fd0367149aee 

# token加入語句忘記了可以在master上使用下邊命令進行生成 
kubeadm token create --print-join-command

 

配置etcdctl命令

etcdctl 二進制包網站    使用etcdctl

下載文件，並解壓到 /usr/local/etcd 目錄，然后創建軟連接即可
mkdir -pv /usr/local/etcd/ && tar xzvf etcd-v3.4.9-linux-amd64.tar.gz -C /usr/local/etcd --strip-components=1
/usr/local/etcd/etcdctl version && echo " " && /usr/local/etcd/etcd --version
ln -s /usr/local/etcd/etcd /usr/bin/etcd && ln -s /usr/local/etcd/etcdctl /usr/bin/etcdctl

 

后續操作：

1、更新鏡像，參考

kubectl set image deployment nginx-deployment  nginx=nginx:1.14.4   //更新鏡像

kubectl rollout status deployment nginx-deployment   //查看鏡像更新過程

2、deployment

 

podSubnet: 10.246.0.0/16