打造kubernetes 高可用集群(nginx+keepalived)

一、添加master

部署高可用k8s架構

1.拷貝/opt/kubernetes目錄到新的master上(注意如果新機上部署了etcd要排除掉)

scp -r /opt/kubernetes/ root@192.168.24.12:/opt/

2.拷貝主件服務

scp /usr/lib/systemd/system/{kube-apiserver.service,kube-controller-manager.service,kube-scheduler.service} root@192.168.24.12:/usr/lib/systemd/system/

3.修改配置文件kube-apiserver的ip為新master ip

4.啟動服務

systemctl start kube-apiserver

systemctl start kube-controller-manager

systemctl start kube-scheduler

二、在LB上安裝nginx（master,backup）且配置一樣。

1、yum安裝

Install the prerequisites:

sudo yum install yum-utils

To set up the yum repository, create the file named /etc/yum.repos.d/nginx.repo with the following contents:

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
​
[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key

By default, the repository for stable nginx packages is used. If you would like to use mainline nginx packages, run the following command:

sudo yum-config-manager --enable nginx-mainline

To install nginx, run the following command:

sudo yum install nginx

2、二進制安裝

一、安裝依賴包
yum install -y gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel
依賴包說明：
1、編譯依賴 gcc 環境，所以需要：gcc gcc-c++;
2、PCRE(Perl Compatible Regular Expressions) 是一個Perl庫，包括 perl 兼容的正則表達式庫。nginx 的 http 模塊使用 pcre 來解析正則表達式，所以需要在 linux 上安裝 pcre 庫，pcre-devel 是使用 pcre 開發的一個二次開發庫，所以需要：pcre pcre-devel ;
3、zlib 庫提供了很多種壓縮和解壓縮的方式， nginx 使用 zlib 對 http 包的內容進行 gzip ，所以需要在 Centos 上安裝 zlib 庫，所以需要：zlib zlib-devel ；
4、OpenSSL 是一個強大的安全套接字層密碼庫，囊括主要的密碼算法、常用的密鑰和證書封裝管理功能及 SSL 協議，並提供豐富的應用程序供測試或其它目的使用。nginx 不僅支持 http 協議，還支持 https（即在ssl協議上傳輸http），所以需要在 Centos 安裝 OpenSSL 庫，所以需要：openssl openssl-devel ；
二、從官網下載安裝包
wget https://nginx.org/download/nginx-1.16.0.tar.gz
三、解壓並安裝
tar zxvf nginx-1.16.0.tar.gz
cd nginx-1.16.0
./configure --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_realip_module --with-http_flv_module --with-http_mp4_module --with-http_gzip_static_module --with-stream --with-stream_ssl_module
make && make install

配置nginx文件：

stream {
   log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log  /var/log/nginx/k8s-access.log  main;
upstream k8s-apiserver {
    server 192.168.0.211:6443;
    server 192.168.0.214:6443;
}
server {
    listen 0.0.0.0:6443;
    proxy_passs k8s-apiserver;
}
}

 三、安裝keepalived（master.backup）

yum安裝keepalived

yum install keepalived -y

修改master配置文件：

! Configuration File for keepalived 
​
global_defs { 
​
# 接收郵件地址 
​
   notification_email { 
     [e-mail] 
     [e-mail] 
     [e-mail] 
   } 
​
# 郵件發送地址 
​
   notification_email_from [e-mail]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER 
} 
​
vrrp_script check_nginx {
    script "/usr/local/nginx/sbin/check_nginx.sh"
}
​
vrrp_instance VI_1 { 
    state MASTER 
    interface ens32     #主機網卡
    virtual_router_id 51 # VRRP 路由 ID實例，每個實例是唯一的 
    priority 100    # 優先級，備服務器設置 90 
    advert_int 1    # 指定VRRP 心跳包通告間隔時間，默認1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress { 
        192.168.7.43/24 
    } 
    track_script {
        check_nginx
    } 
}

修改backup配置文件：

! Configuration File for keepalived 
​
global_defs { 
​
# 接收郵件地址 
​
   notification_email { 
     [e-mail] 
     [e-mail] 
     [e-mail] 
   } 
​
# 郵件發送地址 
​
   notification_email_from [e-mail]  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER 
} 
​
vrrp_script check_nginx {
    script "/usr/local/nginx/sbin/check_nginx.sh"
}
​
vrrp_instance VI_1 { 
    state BACKUP 
    interface ens32     #主機網卡
    virtual_router_id 51 # VRRP 路由 ID實例，每個實例是唯一的 
    priority 90    # 優先級，備服務器設置 90 
    advert_int 1    # 指定VRRP 心跳包通告間隔時間，默認1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress { 
        192.168.7.43/24 
    } 
    track_script {
        check_nginx
    } 
}

添加檢查腳本：

/usr/local/nginx/sbin/check_nginx.sh
​
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
​
if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi

啟動keepalived:

systemctl start keepalived

 四、修改node配置文件

除了flannel，其他都要改成keepalived的vip地址，然后重啟。

systemctl restart kubelet
systemctl restart kube-proxy