生產環境k8s集群方案主要由以下四部分組成：
1、F5 ，rancher ui及業務服務訪問入口，所有域名訪問解析到此F5
2、nginx，k8s集群業務服務訪問入口、HA負載
3、rancher ui，k8s可視化管理平台
4、k8s集群
整體架構圖：
准備：
1、所有主機節點開通外網訪問權限
2、申請https://k8s-pro.jieyue.com rancher ui 域名解析
3、申請xxx.k8spro.jieyue.com 業務服務 域名解析
4、k8s相關命令文件地址：http://172.16.101.32/stgconfig/soft
一、安裝環境
1、硬件環境
|
節點 |
ip |
OS |
角色 |
配置 |
|---|---|---|---|---|
| nginx-115 | 172.19.101.115 | centos7.6 | nginx | 2C,4G,100G |
| nginx-116 | 172.19.101.116 | centos7.6 | nginx | 2C,4G,100G |
| rancher-117 | 172.19.101.117 | centos7.6 | rancher server | 4C,8G,100G |
| rancher-134 | 172.19.101.134 | centos7.6 | rancher server | 4C,8G,100G |
| k8s-135 | 172.19.101.135 | centos7.6 | controlplane,etcd,worker | 4C,16G,300G |
| k8s-136 | 172.19.101.136 | centos7.6 | controlplane,etcd,worker | 4C,16G,300G |
| k8s-137 | 172.19.101.137 | centos7.6 | controlplane,etcd,worker | 4C,16G,300G |
| f5-138 | 172.19.101.138 | f5 | ||
| f5-140 | 172.19.101.140 | f5 |
2、軟件環境
os：centos7.6
rancher：2.3.2
docker：17.03.2-ce ， overlay2存儲，extfs 文件系統
二、安裝前的准備
1、主機環境配置
1.1、 關閉防火牆
sudo systemctl stop firewalld.service && sudo systemctl disable firewalld.service
1.2、關閉setlinx
sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
1.3、關閉swap
注釋或刪除swap交換分區：vi /etc/fstab
#/dev/mapper/centos-swap swap swap defaults 0 0
1.4、配置主機時間、時區、系統語言
sudo ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
sudo echo 'LANG="en_US.UTF-8"' >> /etc/profile;source /etc/profile
1.5、修改最大進程數 、 用戶進程打開文件數
echo '1000000' > /proc/sys/kernel/pid_max
vim /etc/sysctl.conf
kernel.pid_max=1000000
sysctl -p 保存生效
--------
vi /etc/security/limits.conf
* soft nproc 102400
* hard nofile 65535
jyapp hard nofile 102400
jyapp soft nofile 102400
jyapp soft nproc 102400
root soft nproc unlimited
root soft nofile 102400
root hard nofile 102400
1.6、Kernel性能調優
vi /etc/sysctl.conf
追加
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.neigh.default.gc_thresh1=4096
net.ipv4.neigh.default.gc_thresh2=6144
net.ipv4.neigh.default.gc_thresh3=8192
2、安裝docker-ce並配置
# 添加用戶(可選)
sudo adduser jyapp
# 為新用戶設置密碼
sudo passwd jyapp
# 為新用戶添加sudo權限
sudo echo 'jyapp ALL=(ALL) ALL' >> /etc/sudoers
jyapp加用docker用戶組
sudo usermod -a -G docker jyapp
vi /etc/docker/daemon.json
{
"bip":"172.25.100.1/24"
,"insecure-registries":["172.16.101.43:5000","172.16.101.49","10.50.182.85","docker.jieyue.com"]
,"registry-mirrors": ["http://172.16.101.49","http://10.50.182.85","http://docker.jieyue.com","https://088aunhx.mirror.aliyuncs.com"]
,"log-driver": "json-file"
,"log-opts": {
"max-size": "100m",
"max-file": "3"
}
,"max-concurrent-downloads": 10
,"max-concurrent-uploads": 10
,"storage-driver": "overlay2"
,"storage-opts": ["overlay2.override_kernel_check=true"]
}
systemctl daemon-reload
systemctl restart docker
3、下載命令行工具
附件中下載
kubectl，rke，helm 到 172.19.101.135 /usr/local/bin 下
命令工具下載地址：http://172.16.101.32/stgconfig/soft/raw/master/k8s-kub.tar.gz
cd /usr/local/bin
賦可執行權限
sudo chmod +x *
修改profile
sudo vi /etc/profile
追加：
export PATH="/usr/local/bin:$PATH"
保存退出，使之生效
sudo source /etc/profile
4、免密登錄
在172.19.101.135用jyapp用戶執行：
ssh-keygen -t rsa
ssh-copy-id -p 10022 -i ~/.ssh/id_rsa.pub jyapp@172.19.101.135
ssh-copy-id -p 10022 -i ~/.ssh/id_rsa.pub jyapp@172.19.101.136
ssh-copy-id -p 10022 -i ~/.ssh/id_rsa.pub jyapp@172.19.101.137
三、rke安裝k8s
1、配置rke 創建rancher-cluster.yml文件
nodes:
- address: 172.19.101.135
user: jyapp
role: [controlplane,worker,etcd]
port: 10022
- address: 172.19.101.136
user: jyapp
role: [controlplane,worker,etcd]
port: 10022
- address: 172.19.101.137
user: jyapp
role: [controlplane,worker,etcd]
port: 10022
services:
etcd:
backup_config:
enabled: true
interval_hours: 6
retention: 12
extra_args:
quota-backend-bytes: '4294967296'
kubelet:
cluster_domain: cluster.local
cluster_dns_server: 10.43.0.10
fail_swap_on: true
extra_args:
max-pods: 50
kube-api-burst: '100'
kube-api-qps: '100'
#enforce-node-allocatable: 'pods'
#system-reserved: 'cpu=1,memory=4000Mi'
#kube-reserved: 'cpu=1,memory=4000Mi'
kube-controller:
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
extra_args:
node-cidr-mask-size: '24'
kube-api:
service_cluster_ip_range: 10.43.0.0/16
service_node_port_range: 30000-32767
pod_security_policy: false
always_pull_images: false
2、運行RKE命令
rke up --config rancher-cluster.yml
完成后，它應顯示：Finished building Kubernetes cluster successfully。
在本目錄會生成config文件：kube_config_rancher-cluster.yml
如若操作失敗，重新安裝需要清理數據：
su - root
rm -rf /var/lib/rancher/etcd/*
rm -rf /etc/kubernetes/*
su - jyapp
rke remove --config ./rancher-cluster.yml
3、測試集群
mkdir ~/.kube
cp kube_config_rancher-cluster.yml ~/.kube/config
kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
kubectl get node
NAME STATUS ROLES AGE VERSION
172.19.101.135 Ready controlplane,etcd,worker 20d v1.13.5
172.19.101.136 Ready controlplane,etcd,worker 20d v1.13.5
172.19.101.137 Ready controlplane,etcd,worker 20d v1.13.5
四、安裝 rancher server
1、分別在172.19.101.117、172.19.101.134上執行
docker run -d --restart=unless-stopped \ -p 80:80 -p 443:443 \ -v /var/lib/rancher/:/var/lib/rancher/ \-v /root/var/log/auditlog:/var/log/auditlog \ -e CATTLE_SYSTEM_CATALOG=bundled \ -e AUDIT_LEVEL=3 \ rancher/rancher:v2.3.2訪問：https://172.19.101.117
注：ui初始登錄時需指訪問地址為 k8s-pro.jieyue.com
2、通過rancher ui導入 rke k8s集群
五、 配置nginx
1、sudo vi /etc/nginx/nginx.conf
user nginx;
worker_processes 4;
error_log /var/log/nginx/error.log warn;
worker_rlimit_nofile 40000;
events {
worker_connections 8192;
}
http {
upstream rancher_web{
server 172.19.101.135:80;
server 172.19.101.136:80;
server 172.19.101.137:80;
}
server {
listen 80 default_server;
server_name _;
access_log logs/busiweb.access.log main;
client_max_body_size 300m;
client_body_buffer_size 128k;
proxy_connect_timeout 600;
proxy_read_timeout 600;
proxy_send_timeout 600;
proxy_buffer_size 64k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://rancher_web;
}
}
}
2、 或可以容器化nginx代替以上安裝
下載附件nginx.tar.gz 並解壓
略作修改：按上面nginx.conf 修改
執行 sh startNginx-80.sh 即可