kubernetes 高可用部署方法筆記

 環境：win10， VM16， centos 7. 9，k8s 1.22.2

 硬件要求：2GB或更多RAM，2個CPU或更多CPU，硬盤30GB或更多

一、規划

主機名	IP	網關	DNS
master1	192.168.17.101	192.168.17.2	192.168.17.0
master2	192.168.17.102	192.168.17.2	192.168.17.0
master3	192.168.17.103	192.168.17.2	192.168.17.0
node1	192.168.17.104	192.168.17.2	192.168.17.0
node2	192.168.17.105	192.168.17.2	192.168.17.0

 

VIP虛擬地址

192.168.17.10

 

二、主機初始化

1、創建五台虛擬機，按照圖示安裝主機（圖中數據可能不一致，具體參數按規划內容設置）

查看網絡路由

 

 2、選擇啟動centos 安裝

語言選擇簡體中文

 

 軟件選擇：基礎設施服務器

 網絡和主機名：打開以太網開關

 修改主機名，與我們規划虛擬機名一致，方便管理

 

 配置網絡，添加地址與子掩碼、網關、DNS服務器，文章開頭有標注

 

 

 

 設置root密碼

 

三、環境初始化（五台虛擬機相同操作）

查看系統發行版本

 cat /etc/redhat-release

1、設置主機域名解析

#主機名解析
vim /etc/hosts
#添加內容

192.168.17.101 master1
192.168.17.102 master2 192.168.17.103 master3 192.168.17.104 node1 192.168.17.105 node2

2、啟動時間同步

#啟動chrony服務
systemctl start chronyd
#開機自啟
systemctl enable chronyd
#date

3、禁用iptable和firewalld服務

systemctl stop firewalld && systemctl disable firewalld
systemctl stop iptables && systemctl disable iptables

 

4、禁用selinux,將SELINUX改為disabled

vim /etc/selinux/config

5、禁用swap分區,注釋掉/dev/mapper/centos-swap swap 

vim /etc/fstab
#/dev/mapper/centos-swap swap 

 

 

6、修改內核參數，新建文件

vim /etc/sysctl.d/kubernetes.conf

#添加內容

net.bridge.bridge-nf-call-ip6tables  = 1
net.bridge.bridge-nf-call-iptables  = 1 
net.ipv4.ip_forward =1

#重新加載配置
sysctl -p
#加載網橋過濾模塊
modprobe br_netfilter
#查看網橋過濾模塊是否加載成功
lsmod | grep br_netfilter

 

7、配置ipvs功能

yum install ipset ipvsadm -y

cat <<EOF > /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

 

 #為腳本添加執行權限
chmod +x /etc/sysconfig/modules/ipvs.modules
#執行腳本
/bin/bash /etc/sysconfig/modules/ipvs.modules
#查看模塊是否加載成功
lsmod |grep -e ip_vs -e nf_conntrack_ipv4

 

8、重啟服務器

reboot

 四、kubernetes高可用集群搭建

1、docker 安裝

切換鏡像

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

#查看當前鏡像源中支持的docker版本
yum list docker-ce --showduplicates
#安裝最新版本docker-ce
yum install  docker-ce

#添加配置文件
mkdir /etc/docker
cat <<EOF > /etc/docker/daemon.json
{
   "registry-mirrors": ["https://eeu0s9el.mirror.aliyuncs.com"],
   "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

 

#啟動docker，設置開機自啟
systemctl restart docker
systemctl enable docker

#檢查版本
docker version

 

 2、kubernetes 鏡像准備

①添加阿里雲的YUM軟件源

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg 
　　　　 https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

②安裝kubeadm、kubelet和kubectl

yum install -y kubelet kubeadm kubectl

# 為了實現Docker使用的cgroup drvier和kubelet使用的cgroup drver一致，建議修改"/etc/sysconfig/kubelet"文件的內容：
vim /etc/sysconfig/kubelet
# 修改為以下內容
KUBELET_CGROUP_ARGS="--cgroup-driver=systemd"
KUBE_PROXY_MODE="ipvs"

# 設置為開機啟動即可
systemctl enable kubelet

③查看kubeadm建議軟件版本、拉取鏡像

kubeadm config images list

images=( 
     kube-apiserver:v1.22.2
     kube-controller-manager:v1.22.2
     kube-scheduler:v1.22.2
     kube-proxy:v1.22.2
     pause:3.5
     etcd:3.5.0-0
)

for imageName in ${images[@]} ; do
   docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
   docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done

docker pull coredns/coredns:1.8.4 docker images docker tag coredns/coredns:1.8.4 k8s.gcr.io/coredns/coredns:v1.8.4 docker rmi coredns/coredns:1.8.4 docker images

3、所有master節點安裝keepalived haproxy

①安裝 keepalived

yum install -y keepalived

不同master節點創建不同配置。

 master1 配置：

cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived

global_defs {        #全局定義部分
   router_id master #Keepalived服務器的路由標識，自定義
}
vrrp_script check-haproxy { #定義vrrp腳本
    #檢查所在節點的 haproxy 進程是否正常。如果異常則將權重減少3
    script "killall -0 haproxy" #要執行的命令
    interval 5                  #腳本調用時間
    weight -3                   #根據權重值調整vrrp實例的優先級，默認為0
    fall 10                     #需要失敗多少次，vrrp才進行角色狀態切換(從正常切換為fault)
    rise 2                      #需要成功多少次，vrrp才進行角色狀態切換(從fault切換為正常)
}
vrrp_instance VI_1 {
    state MASTER      #指定Master或者BACKUP，必須大寫
    interface ens33   #監聽的實際網口,VIP所在接口
    dont_track_primary
    virtual_router_id 10  #VIP末段
    priority 100          #權重值，數字越大優先級越高，誰的優先級高誰就是master，該項取值范圍是1-255（在此范圍之外會被識別成默認值100）
    advert_int 3          #發送組播包的間隔時間，默認為1秒
    track_script {
        check-haproxy
    }
    virtual_ipaddress {
        192.168.17.10/24  #vip地址
    }
}
EOF

 master2 配置：

cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
   router_id master
}
vrrp_script check-haproxy {
    script "killall -0 haproxy"
    interval 5
    weight -3
    fall 10
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 10
    dont_track_primary
    priority 99
    advert_int 3
    track_script {
        check-haproxy
    }
    virtual_ipaddress {
        192.168.17.10/24
    }
}
EOF

 master3 配置：

cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
   router_id master
}
vrrp_script check-haproxy {
    script "killall -0 haproxy"
    interval 5
    weight -3
    fall 10
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 10
    dont_track_primary
    priority 98
    advert_int 3
    track_script {
        check-haproxy
    }
    virtual_ipaddress {
        192.168.17.10/24
    }
}
EOF

所有master節點執行開機自啟設置

systemctl start keepalived && systemctl enable keepalived

查看是否生效

 

 

 看到此vip后，嘗試停掉master1的keepalived服務，查看vip是否能漂移到master2，並且重新啟動master1的keepalived服務，查看vip是否能正常漂移回來，能漂移回來，證明配置沒有問題

② 安裝 haproxy

yum install -y haproxy

創建配置文件：

cat > /etc/haproxy/haproxy.cfg <<EOF
global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /var/run/haproxy-admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    nbproc 1

defaults
    log     global
    timeout connect 5000
    timeout client  10m
    timeout server  10m

listen  admin_stats
    bind 0.0.0.0:10080
    mode http
    log 127.0.0.1 local0 err
    stats refresh 30s
    stats uri /status
    stats realm welcome login\ Haproxy
    stats auth admin:123456
    stats hide-version
    stats admin if TRUE

listen kube-master
    bind 0.0.0.0:8443
    mode tcp
    option tcplog
    balance source
    server 192.168.17.101 192.168.17.101:6443 check inter 2000 fall 2 rise 2 weight 1
    server 192.168.17.102 192.168.17.102:6443 check inter 2000 fall 2 rise 2 weight 1
    server 192.168.17.103 192.168.17.103:6443 check inter 2000 fall 2 rise 2 weight 1
EOF

#設置開機自啟
systemctl enable haproxy && systemctl restart haproxy

#查看是否啟動完成
netstat -lntup|grep haproxy

4、初始化集群

①創建配置文件（在具有vip的master上操作，這里為master1）

cat > kubeadm-config.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.22.2
apiServer:
  certSANs:
  - 192.168.17.101  #master1
  - 192.168.17.102  #master2
  - 192.168.17.103  #master3
  - 192.168.17.10   #VIP
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
#vip地址
controlPlaneEndpoint: 192.168.17.10:6443
#已預拉鏡像，初始化可不指定
#imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12

EOF

②初始化集群（在具有vip的master上操作，這里為master1）

kubeadm init --config=kubeadm-config.yaml --upload-certs

# 在master1節點執行，這里的命令是kubeadm init命令執行后，提示出來的
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

③部署CNI網絡插件flannel（在具有vip的master上操作，這里為master1）

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

若訪問拒絕可使用使用以下載修改過鏡像地址的kube-flannel.yml

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.14.0
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.14.0
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

④ master2，master3執行第一條join 命令， node1 node2執行第二條join命令加入集群

 ⑤ 查看系統pod 和節點狀態

# 查看部署CNI網絡插件運行狀態
kubectl get pods -n kube-system
# 在狀態都為Running時，查看節點狀態
kubectl get nodes

五、測試

#創建測試命名空間
kubectl create ns dev
#部署nginx
kubectl create deployment  nginx --image=nginx:1.17.1 -n dev
#暴露端口
kubectl expose deployment nginx -n dev --port=80 --type=NodePort
#查看服務狀態
kubectl get pods,svc -n dev

 瀏覽器訪問VIP地址與31246端口，能訪問到則部署成功

 

 

 

六、所遇問題解決

1、如圖所示，如果coredns一直處於Pending狀態，查看網絡插件是否安裝完成flannel

2、當node節點超過24小時再加入，可再master節點使用以下命令生成

kubeadm token create --print-join-command

 3、環境還原

#如果環境出現問題，盡最大努力還原通過 'kubeadm init' 或者 'kubeadm join' 操作對主機所做的更改
kubeadm reset
rm -rf $HOME/.kube

4、虛擬機所有節點掛起，宿主機關機，再次開機vip丟失，master1節點重啟即可