008.Kubernetes二進制master節點部署高可用

本文轉載自查看原文 2019-11-16 10:57 442 Kubernetes Nginx/ Kubernetes高可用/ Kubernetes

一 kube-apiserver高可用

1.1 Keepalived實現VIP

Keepalived可以提供kube-apiserver VIP，配合Nginx實現kube-apiserver的高可用。

1.2 Nginx實現反向代理

基於 nginx 代理的 kube-apiserver 高可用方案。

控制節點的 kube-controller-manager、kube-scheduler 是多實例部署，所以只要有一個實例正常，就可以保證高可用；

集群內的 Pod 使用 K8S 服務域名 kubernetes 訪問 kube-apiserver， kube-dns 會自動解析出多個 kube-apiserver 節點的 VIP，所以也是高可用的；

在每個節點起一個 nginx 進程，后端對接多個 apiserver 實例，nginx 對它們做健康檢查和負載均衡；

kubelet、kube-proxy、controller-manager、scheduler 通過本地的 nginx（監聽 172.24.8.100）訪問 kube-apiserver，從而實現 kube-apiserver 的高可用；

基於 nginx 4 層透明代理功能實現 K8S 節點( master 節點和 worker 節點)高可用訪問 kube-apiserver 。

二 Kubernetes高可用部署

2.1 Keepalived安裝

  1 [root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
  2   do
  3     echo ">>> ${master_ip}"
  4     ssh ${master_ip} "mkdir -p /opt/k8s/kube-keepalived/"
  5     ssh ${master_ip} "mkdir -p /etc/keepalived/"
  6   done						#創建keepalived目錄
  7 [root@master01 ~]# cd /opt/k8s/work
  8 [root@master01 work]# wget http://down.linuxsb.com:8888/software/keepalived-2.0.20.tar.gz
  9 [root@master01 work]# tar -zxvf keepalived-2.0.20.tar.gz
 10 [root@master01 work]# cd keepalived-2.0.20/ && ./configure --sysconf=/etc --prefix=/opt/k8s/kube-keepalived/ && make && make install

提示：本步驟操作僅需要在master01節點操作。

2.2 分發Keepalived二進制文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp -rp /opt/k8s/kube-keepalived/ root@${master_ip}:/opt/k8s/
  7     scp -rp /usr/lib/systemd/system/keepalived.service  root@${master_ip}:/usr/lib/systemd/system/
  8     ssh ${master_ip} "systemctl daemon-reload && systemctl enable keepalived"
  9   done						#分發Keepalived二進制文件

提示：本步驟操作僅需要在master01節點操作。

2.3 Nginx安裝

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# wget http://nginx.org/download/nginx-1.19.0.tar.gz
  3 [root@master01 work]# tar -xzvf nginx-1.19.0.tar.gz
  4 [root@master01 work]# cd /opt/k8s/work/nginx-1.19.0/
  5 [root@master01 nginx-1.19.0]# mkdir nginx-prefix
  6 [root@master01 nginx-1.19.0]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
  7 [root@master01 nginx-1.19.0]# make && make install

解釋：

--with-stream：開啟 4 層透明轉發(TCP Proxy)功能；

--without-xxx：關閉所有其他功能，這樣生成的動態鏈接二進制程序依賴最小。

  1 [root@master01 nginx-1.19.0]# ./nginx-prefix/sbin/nginx -v

提示：本步驟操作僅需要在master01節點操作。

2.4 驗證編譯后的Nginx

  1 [root@master01 ~]# cd /opt/k8s/work/nginx-1.19.0/
  2 [root@master01 nginx-1.19.0]# ./nginx-prefix/sbin/nginx -v
  3 nginx version: nginx/1.19.0
  4 [root@master01 nginx-1.19.0]# ldd ./nginx-prefix/sbin/nginx	#查看 nginx 動態鏈接的庫
  5         linux-vdso.so.1 =>  (0x00007ffe7f596000)
  6         libdl.so.2 => /lib64/libdl.so.2 (0x00007f1df0fb8000)
  7         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1df0d9c000)
  8         libc.so.6 => /lib64/libc.so.6 (0x00007f1df09ce000)
  9         /lib64/ld-linux-x86-64.so.2 (0x00007f1df11bc000)

提示：由於只開啟了 4 層透明轉發功能，所以除了依賴 libc 等操作系統核心 lib 庫外，沒有對其它 lib 的依賴(如 libz、libssl 等)，以便達到精簡編譯的目的。

2.5 分發Nginx二進制文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
  7     scp /opt/k8s/work/nginx-1.19.0/nginx-prefix/sbin/nginx root@${master_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
  8     ssh root@${master_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
  9   done						#分發Nginx二進制文件

提示：本步驟操作僅需要在master01節點操作。

2.6 配置Nginx system

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# cat > kube-nginx.service <<EOF
  4 [Unit]
  5 Description=kube-apiserver nginx proxy
  6 After=network.target
  7 After=network-online.target
  8 Wants=network-online.target
  9 
 10 [Service]
 11 Type=forking
 12 ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
 13 ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
 14 ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
 15 PrivateTmp=true
 16 Restart=always
 17 RestartSec=5
 18 StartLimitInterval=0
 19 LimitNOFILE=65536
 20 
 21 [Install]
 22 WantedBy=multi-user.target
 23 EOF

提示：本步驟操作僅需要在master01節點操作。

2.7 分發Nginx systemd

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp kube-nginx.service  root@${master_ip}:/etc/systemd/system/
  7     ssh ${master_ip} "systemctl daemon-reload && systemctl enable kube-nginx.service"
  8   done

提示：本步驟操作僅需要在master01節點操作。

2.8 創建配置文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# wget http://down.linuxsb.com:8888/binngkek8s.sh	#拉取自動部署腳本
  4 [root@master01 work]# vi binngkek8s.sh  #其他部分保持默認

  1 #!/bin/sh
  2 #****************************************************************#
  3 # ScriptName: ngkek8s.sh
  4 # Author: xhy
  5 # Create Date: 2020-05-13 16:32
  6 # Modify Author: xhy
  7 # Modify Date: 2020-05-30 13:24
  8 # Version: v2
  9 #***************************************************************#
 10 
 11 #######################################
 12 # set variables below to create the config files, all files will create at ./config directory
 13 #######################################
 14 
 15 # master keepalived virtual ip address
 16 export K8SHA_VIP=172.24.8.100
 17 
 18 # master01 ip address
 19 export K8SHA_IP1=172.24.8.71
 20 
 21 # master02 ip address
 22 export K8SHA_IP2=172.24.8.72
 23 
 24 # master03 ip address
 25 export K8SHA_IP3=172.24.8.73
 26 
 27 # master01 hostname
 28 export K8SHA_HOST1=master01
 29 
 30 # master02 hostname
 31 export K8SHA_HOST2=master02
 32 
 33 # master03 hostname
 34 export K8SHA_HOST3=master03
 35 
 36 # master01 network interface name
 37 export K8SHA_NETINF1=eth0
 38 
 39 # master02 network interface name
 40 export K8SHA_NETINF2=eth0
 41 
 42 # master03 network interface name
 43 export K8SHA_NETINF3=eth0
 44 
 45 # keepalived auth_pass config
 46 export K8SHA_KEEPALIVED_AUTH=412f7dc3bfed32194d1600c483e10ad1d
 47 
 48 # kubernetes CIDR pod subnet
 49 export K8SHA_PODCIDR=10.10.0.0
 50 
 51 # kubernetes CIDR svc subnet
 52 export K8SHA_SVCCIDR=10.20.0.0

  1 [root@master01 work]# chmod u+x *.sh
  2 [root@master01 work]# ./binngkek8s.sh

解釋：如上僅需Master01節點操作。執行binngkek8s.sh腳本后，會自動生成以下配置文件：

keepalived：keepalived配置文件，位於各個master節點的/etc/keepalived目錄
nginx-lb：nginx-lb負載均衡配置文件，位於各個master節點的/opt/k8s/kube-nginx/conf/kube-nginx.conf目錄

三啟動高可用

3.1 確認配置

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     echo ">>>> check check sh"
  7     ssh root@${master_ip} "ls -l /etc/keepalived/check_apiserver.sh"
  8     echo ">>> check Keepalived config"
  9     ssh root@${master_ip} "cat /etc/keepalived/keepalived.conf"
 10     echo ">>> check Nginx config"
 11     ssh root@${master_ip} "cat /opt/k8s/kube-nginx/conf/kube-nginx.conf"
 12   done			#檢查高可用相關配置

提示：本步驟操作僅需要在master01節點操作。

3.2 啟動服務

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "systemctl restart keepalived.service && systemctl enable keepalived.service"
  7     ssh root@${master_ip} "systemctl restart kube-nginx.service && systemctl enable kube-nginx.service"
  8     ssh root@${master_ip} "systemctl status keepalived.service | grep Active"
  9     ssh root@${master_ip} "systemctl status kube-nginx.service | grep Active"
 10     ssh root@${master_ip} "netstat -tlunp | grep 16443"
 11   done

提示：本步驟操作僅需要在master01節點操作。

3.3 確認驗證

  1 3.3	確認驗證
  2 [root@master01 ~]# cd /opt/k8s/work
  3 [root@master01 ~]# cd /opt/k8s/work
  4 [root@master01 work]# source /root/environment.sh
  5 [root@master01 work]# for all_ip in ${ALL_IPS[@]}
  6   do
  7     echo ">>> ${all_ip}"
  8     ssh root@${all_ip} "ping -c1 172.24.8.100"
  9   done							#等待20s左右執行檢查

提示：本步驟操作僅需要在master01節點操作。

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 二進制部署kubernetes高可用 005.Kubernetes二進制master節點部署ETCD 009.Kubernetes二進制master節點部署kubectl 二進制搭建kubernetes多master集群【三、配置k8s master及高可用】 Kubernetes二進制（單/多master）部署 Kubernetes二進制部署單節點 Kubernetes 二進制部署（一）單節點部署（Master 與 Node 同一機器） 010.Kubernetes二進制master節點部署kube-apiserver 部署一套完整的Kubernetes高可用集群（二進制，v1.18版） k8s1.13.0二進制部署-master節點（三）