系列文章說明
本系列文章,可以基本算是 老男孩2019年王碩的K8S周末班課程 筆記,根據視頻來看本筆記最好,否則有些地方會看不明白
需要視頻可以聯系我
目錄
- 系列文章說明
- 1 部署架構
- 2 部署准備
- 3 部署master節點-etcd服務
- 4 部署mater節點 kube-apiserver服務
- 5 部署4層反代去代理apiserver
- 6 部署node節點
- 7 驗證kubernetes集群
1 部署架構
1.1 架構圖
架構說明:
- etcd至少3台組成一個高可用集群
- 兩台proxy組成高可用代理對外提供VIP
- 兩台機器共同承擔master和node節點功能
- 運維主機非K8S套件,但為K8S服務
1.2 安裝方式選擇
- Minikube 預覽使用,僅供學習
- 二進制安裝(生產首選,新手推薦)
- kubeadmin安裝
簡單,用k8s跑k8s自己,熟手推薦
新手不推薦的原因是容易知其然不知其所以然
出問題后找不到解決辦法
2 部署准備
2.1 准備工作
准備5台2C/2g/50g虛擬機,網絡10.4.7.0/24
預裝centos7.4,做完基礎優化
安裝部署bind9,部署自建DNS系統
准備自簽證書環境
安裝部署docker和harbor倉庫
機器列表
| 主機名 | IP地址 | 用途 |
|---|---|---|
| hdss7-11 | 10.4.7.11 | proxy1 |
| hdss7-12 | 10.4.7.12 | proxy2 |
| hdss7-21 | 10.4.7.21 | master1 |
| hdss7-22 | 10.4.7.22 | master2 |
| hdss7-200 | 10.4.7.200 | 運維主機 |
基本部署軟件
[root@hdss7-11 ~]# hostname
hdss7-11
[root@hdss7-11 ~]# getenforce
Disabled
[root@hdss7-11 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=none
NAME=eth0
DEVICE=eth0
ONBOOT=yes
IPADDR=10.4.7.11
NETMASK=255.255.255.0
GATEWAY=10.4.7.254
DNS1=10.4.7.254
[root@hdss7-11 ~]# yum install wget net-tools telnet tree nmap sysstat lrzsz dos2unix -y
2.2 部署DNS服務bind9
2.2.1 安裝配置DNS服務
在7.11上部署bind的DNS服務
yum install bind bind-utils -y
修改並校驗配置文件
[root@hdss7-11 ~]# vim /etc/named.conf
listen-on port 53 { 10.4.7.11; };
allow-query { any; };
forwarders { 10.4.7.254; }; #上一層DNS地址(網關或公網DNS)
recursion yes;
dnssec-enable no;
dnssec-validation no
[root@hdss7-11 ~]# named-checkconf
2.2.2 增加自定義域和對於配置
在域配置中增加自定義域
cat >>/etc/named.rfc1912.zones <<'EOF'
# 添加自定義主機域
zone "host.com" IN {
type master;
file "host.com.zone";
allow-update { 10.4.7.11; };
};
# 添加自定義業務域
zone "zq.com" IN {
type master;
file "zq.com.zone";
allow-update { 10.4.7.11; };
};
EOF
host.com和zq.com都是我們自定義的域名,一般用host.com做為主機域
zq.com為業務域,業務不同可以配置多個
為自定義域host.com創建配置文件
cat >/var/named/host.com.zone <<'EOF'
$ORIGIN host.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.host.com. dnsadmin.host.com. (
2020041601 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.host.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
HDSS7-11 A 10.4.7.11
HDSS7-12 A 10.4.7.12
HDSS7-21 A 10.4.7.21
HDSS7-22 A 10.4.7.22
HDSS7-200 A 10.4.7.200
EOF
為自定義域zq.com創建配置文件
cat >/var/named/zq.com.zone <<'EOF'
$ORIGIN zq.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.zq.com. dnsadmin.zq.com. (
2020041601 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.zq.com.
$TTL 60 ; 1 minute
dns A 10.4.7.11
EOF
host.com域用於主機之間通信,所以要先增加上所有主機
zq.com域用於后面的業務解析用,因此不需要先添加主機
2.2.3 啟動並驗證DNS服務
再次檢查配置並啟動dns服務
[root@hdss7-11 ~]# named-checkconf
[root@hdss7-11 ~]# systemctl start named
[root@hdss7-11 ~]# ss -lntup|grep 53
udp UNCONN 0 0 10.4.7.11:53
udp UNCONN 0 0 :::53
tcp LISTEN 0 10 10.4.7.11:53
tcp LISTEN 0 128 127.0.0.1:953
tcp LISTEN 0 10 :::53
tcp LISTEN 0 128 ::1:953
# 驗證結果
[root@hdss7-11 ~]# dig -t A hdss7-11.host.com @10.4.7.11 +short
10.4.7.11
[root@hdss7-11 ~]# dig -t A hdss7-21.host.com @10.4.7.11 +short
10.4.7.21
2.2.4 所有主機修改網絡配置
5台K8S主機都需要按如下方式修改網絡配置
# 修改dns並添加搜索域
sed -i 's#^DNS.*#DNS1=10.4.7.11#g' /etc/sysconfig/network-scripts/ifcfg-eth0
echo "search=host.com" >>/etc/sysconfig/network-scripts/ifcfg-eth0
systemctl restart network
# 檢查DNS配置
~]# cat /etc/resolv.conf
# Generated by NetworkManager
search host.com
nameserver 10.4.7.11
~]# dig -t A hdss7-21.host.com +short
10.4.7.21
# 一定記得檢查dns配置文件中是否有search信息
windows宿主機也要改
wmnet8網卡更改DNS：10.4.7.11
# ping通才行,否則檢查
ping hdss7-200.host.com
2.3 自簽發證書環境准備
操作在7.200這個運維機上完成
2.3.1 下載安裝cfssl
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/bin/cfssl-json
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/bin/cfssl-certinfo
chmod +x /usr/bin/cfssl*
2.3.2 生成ca證書文件
mkdir /opt/certs
cat >/opt/certs/ca-csr.json <<EOF
{
"CN": "zqcd",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "chengdu",
"L": "chengdu",
"O": "zq",
"OU": "ops"
}
],
"ca": {
"expiry": "175200h"
}
}
EOF
CN: Common Name，瀏覽器使用該字段驗證網站是否合法，一般寫的是域名。非常重要。瀏覽器使用該字段驗證網站是否合法
C: Country， 國家
ST: State，州，省
L: Locality，地區，城市
O: Organization Name，組織名稱，公司名稱
OU: Organization Unit Name，組織單位名稱，公司部門
2.3.3 生成ca證書
cd /opt/certs
cfssl gencert -initca ca-csr.json | cfssl-json -bare ca
[root@hdss7-200 certs]# ll
total 16
-rw-r--r-- 1 root root 989 Apr 16 20:53 cacsr
-rw-r--r-- 1 root root 324 Apr 16 20:52 ca-csr.json
-rw------- 1 root root 1679 Apr 16 20:53 ca-key.pem
-rw-r--r-- 1 root root 1330 Apr 16 20:53 ca.pem
2.4 docker環境准備
2.4.1 安裝並配置docker
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
mkdir /etc/docker/
cat >/etc/docker/daemon.json <<EOF
{
"graph": "/data/docker",
"storage-driver": "overlay2",
"insecure-registries": ["registry.access.redhat.com","quay.io","harbor.zq.com"],
"registry-mirrors": ["https://q2gr04ke.mirror.aliyuncs.com"],
"bip": "172.7.21.1/24",
"exec-opts": ["native.cgroupdriver=systemd"],
"live-restore": true
}
EOF
注意:bip要根據宿主機ip變化
hdss7-21.host.com bip 172.7.21.1/24
hdss7-22.host.com bip 172.7.22.1/24
hdss7-200.host.com bip 172.7.200.1/24
2.4.2 啟動docker
mkdir -p /data/docker
systemctl start docker
systemctl enable docker
docker --version
2.5 部署harbor私有倉庫
下載地址:https://github.com/goharbor/harbor/releases/download/v1.8.5/harbor-offline-installer-v1.8.5.tgz
2.5.1 下載並解壓
tar xf harbor-offline-installer-v1.8.5.tgz -C /opt/
cd /opt/
mv harbor/ harbor-v1.8.5
ln -s /opt/harbor-v1.8.5/ /opt/harbor
2.5.2 編輯配置文件
[root@hdss7-200 opt]# vi /opt/harbor/harbor.yml
# 以下是修改項,手動在配置文件中更改
hostname: harbor.zq.com
http:
port: 180
harbor_admin_password:Harbor12345
data_volume: /data/harbor
log:
level: info
rotate_count: 50
rotate_size:200M
location: /data/harbor/logs
[root@hdss7-200 opt]# mkdir -p /data/harbor/logs
2.5.3 使用docker-compose啟動harbor
[root@hdss7-200 opt]cd /opt/harbor/
yum install docker-compose -y
sh /opt/harbor/install.sh
docker-compose ps
docker ps -a
2.5.4 使用dns解析harbor
在7.11DNS服務上操作
[root@hdss7-11 ~]# vi /var/named/zq.com.zone
2020032002 ; serial #每次修改DNS解析后,都要滾動此ID
harbor A 10.4.7.200
[root@hdss7-11 ~]# systemctl restart named
[root@hdss7-11 ~]# dig -t A harbor.zq.com +short
10.4.7.200
2.5.5 使用nginx反向代理harbor
回到7.200運維機上操作
[root@hdss7-200 harbor]# yum install nginx -y
[root@hdss7-200 harbor]# vi /etc/nginx/conf.d/harbor.zq.com.conf
server {
listen 80;
server_name harbor.zq.com;
client_max_body_size 1000m;
location / {
proxy_pass http://127.0.0.1:180;
}
}
[root@hdss7-200 harbor]# nginx -t
[root@hdss7-200 harbor]# systemctl start nginx
[root@hdss7-200 harbor]# systemctl enable nginx
瀏覽器輸入：harbor.zq.com
用戶名：admin 密碼：Harbor12345
新建項目：public 訪問級別：公開
2.5.6 提前准備pauser/nginx基礎鏡像
pauser鏡像是k8s啟動pod時,預先用來創建相關資源(如名稱空間)的
nginx鏡像是k8s部署好以后,我們測試pod創建所用的
docker login harbor.zq.com -uadmin -pHarbor12345
docker pull kubernetes/pause
docker pull nginx:1.17.9
docker tag kubernetes/pause:latest harbor.zq.com/public/pause:latest
docker tag nginx:1.17.9 harbor.zq.com/public/nginx:v1.17.9
docker push harbor.zq.com/public/pause:latest
docker push harbor.zq.com/public/nginx:v1.17.9
2.6 准備nginx文件服務
創建一個nginx虛擬主機,用來提供文件訪問訪問,主要依賴nginx的autoindex屬性
2.6.1 創建文件訪問
在7.200上
# 創建配置
cat >/etc/nginx/conf.d/k8s-yaml.zq.com.conf <<EOF
server {
listen 80;
server_name k8s-yaml.zq.com;
location / {
autoindex on;
default_type text/plain;
root /data/k8s-yaml;
}
}
EOF
# 啟動nginx
mkdir -p /data/k8s-yaml/coredns
nginx -t
nginx -s reload
2.6.2 添加域名解析
在7.11的bind9域名服務器上,增加DNS記錄
vi /var/named/zq.com.zone
# 在最后添加一條解析記錄
k8s-yaml A 10.4.7.200
# 同時滾動serial為
@ IN SOA dns.zq.com. dnsadmin.zq.com. (
2019061803 ; serial
重啟服務並驗證:
systemctl restart named
[root@hdss7-11 ~]# dig -t A k8s-yaml.zq.com +short
10.4.7.200
3 部署master節點-etcd服務
3.1 部署etcd集群
分別在12/21/22 上安裝ectd服務,11節點作為備選節點
3.1.1 創建生成CA證書的JSON配置文件
在7.200上操作
一個配置里面包含了server端,clinet端和雙向(peer)通信所需要的配置,后面創建證書的時候會傳入不同的參數調用不同的配置
cat >/opt/certs/ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
證書時間統一為10年,不怕過期
證書類型
client certificate：客戶端使用，用於服務端認證客戶端,例如etcdctl、etcd proxy、fleetctl、docker客戶端
server certificate：服務端使用，客戶端以此驗證服務端身份,例如docker服務端、kube-apiserver
peer certificate：雙向證書，用於etcd集群成員間通信
3.1.3.創建生成自簽發請求(csr)的json配置文件
注意:
需要將所有可能用來部署etcd的機器,都加入到hosts列表中
否則后期重新加入不在列表中的機器,需要更換所有etcd服務的證書
cat >/opt/certs/etcd-peer-csr.json <<EOF
{
"CN": "k8s-etcd",
"hosts": [
"10.4.7.11",
"10.4.7.12",
"10.4.7.21",
"10.4.7.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "zq",
"OU": "ops"
}
]
}
EOF
3.1.4.生成etcd證書文件
cd /opt/certs/
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
-config=ca-config.json -profile=peer \
etcd-peer-csr.json |cfssl-json -bare etcd-peer
[root@hdss7-200 certs]# ll
total 36
-rw-r--r-- 1 root root 837 Apr 19 15:35 ca-config.json
-rw-r--r-- 1 root root 989 Apr 16 20:53 ca.csr
-rw-r--r-- 1 root root 324 Apr 16 20:52 ca-csr.json
-rw------- 1 root root 1679 Apr 16 20:53 ca-key.pem
-rw-r--r-- 1 root root 1330 Apr 16 20:53 ca.pem
-rw-r--r-- 1 root root 1062 Apr 19 15:35 etcd-peer.csr
-rw-r--r-- 1 root root 363 Apr 19 15:35 etcd-peer-csr.json
-rw------- 1 root root 1679 Apr 19 15:35 etcd-peer-key.pem
-rw-r--r-- 1 root root 1419 Apr 19 15:35 etcd-peer.pem
3.2 安裝啟動etcd集群
以7.12做為演示,另外2台機器大同小異,不相同的配置都會特別說明
3.2.1 創建etcd用戶和安裝軟件
etcd地址:https://github.com/etcd-io/etcd/tags
建議使用3.1版本,更高版本有問題
useradd -s /sbin/nologin -M etcd
wget https://github.com/etcd-io/etcd/archive/v3.1.20.tar.gz
tar xf etcd-v3.1.20-linux-amd64.tar.gz -C /opt/
cd /opt/
mv etcd-v3.1.20-linux-amd64/ etcd-v3.1.20
ln -s /opt/etcd-v3.1.20/ /opt/etcd
3.2.2 創建目錄，拷貝證書文件
創建證書目錄、數據目錄、日志目錄
mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
chown -R etcd.etcd /opt/etcd-v3.1.20/
chown -R etcd.etcd /data/etcd/
chown -R etcd.etcd /data/logs/etcd-server/
拷貝生成的證書文件
cd /opt/etcd/certs
scp hdss7-200:/opt/certs/ca.pem .
scp hdss7-200:/opt/certs/etcd-peer.pem .
scp hdss7-200:/opt/certs/etcd-peer-key.pem .
chown -R etcd.etcd /opt/etcd/certs
也可以先創建一個NFS,直接從NFS中拷貝
3.2.3 創建etcd服務啟動腳本
參數說明: https://blog.csdn.net/kmhysoft/article/details/71106995
cat >/opt/etcd/etcd-server-startup.sh <<'EOF'
#!/bin/sh
./etcd \
--name etcd-server-7-12 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://10.4.7.12:2380 \
--listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://10.4.7.12:2380 \
--advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
EOF
[root@hdss7-12 ~]# chmod +x /opt/etcd/etcd-server-startup.sh
注意:以上啟動腳本,有幾個配置項在每個服務器都有所不同
--name #節點名字
--listen-peer-urls #監聽其他節點所用的地址
--listen-client-urls #監聽etcd客戶端的地址
--initial-advertise-peer-urls #與其他節點交互信息的地址
--advertise-client-urls #與etcd客戶端交互信息的地址
3.2.4 使用supervisor啟動etcd
安裝supervisor軟件
yum install supervisor -y
systemctl start supervisord
systemctl enable supervisord
創建supervisor管理etcd的配置文件
配置說明參考: https://www.jianshu.com/p/53b5737534e8
cat >/etc/supervisord.d/etcd-server.ini <<EOF
[program:etcd-server] ; 顯示的程序名,類型my.cnf,可以有多個
command=sh /opt/etcd/etcd-server-startup.sh
numprocs=1 ; 啟動進程數 (def 1)
directory=/opt/etcd ; 啟動命令前切換的目錄 (def no cwd)
autostart=true ; 是否自啟 (default: true)
autorestart=true ; 是否自動重啟 (default: true)
startsecs=30 ; 服務運行多久判斷為成功(def. 1)
startretries=3 ; 啟動重試次數 (default 3)
exitcodes=0,2 ; 退出狀態碼 (default 0,2)
stopsignal=QUIT ; 退出信號 (default TERM)
stopwaitsecs=10 ; 退出延遲時間 (default 10)
user=etcd ; 運行用戶
redirect_stderr=true ; 是否重定向錯誤輸出到標准輸出(def false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log
stdout_logfile_maxbytes=64MB ; 日志文件大小 (default 50MB)
stdout_logfile_backups=4 ; 日志文件滾動個數 (default 10)
stdout_capture_maxbytes=1MB ; 設定capture管道的大小(default 0)
;子進程還有子進程,需要添加這個參數,避免產生孤兒進程
killasgroup=true
stopasgroup=true
EOF
啟動etcd服務並檢查
supervisorctl update
supervisorctl status
netstat -lntup|grep etcd
3.2.5 部署啟動集群其他機器
略
3.2.6 檢查集群狀態
[root@hdss7-12 certs]# /opt/etcd/etcdctl cluster-health
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
[root@hdss7-12 certs]# /opt/etcd/etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.22:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.21:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true
4 部署mater節點 kube-apiserver服務
下載頁面: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.15.md
下載地址:
https://dl.k8s.io/v1.15.5/kubernetes-server-linux-amd64.tar.gz
https://dl.k8s.io/v1.15.5/kubernetes-client-linux-amd64.tar.gz
https://dl.k8s.io/v1.15.5/kubernetes-node-linux-amd64.tar.gz
4.1 簽發client端證書
證書簽發都在7.200上操作
此證書的用途是apiserver和etcd之間通信所用
4.1.1 創建生成證書csr的json配置文件
cat >/opt/certs/client-csr.json <<EOF
{
"CN": "k8s-node",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "zq",
"OU": "ops"
}
]
}
EOF
4.1.2 生成client證書文件
cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-profile=client \
client-csr.json |cfssl-json -bare client
[root@hdss7-200 certs]# ll|grep client
-rw-r--r-- 1 root root 993 Apr 20 21:30 client.csr
-rw-r--r-- 1 root root 280 Apr 20 21:30 client-csr.json
-rw------- 1 root root 1675 Apr 20 21:30 client-key.pem
-rw-r--r-- 1 root root 1359 Apr 20 21:30 client.pem
4.2 簽發kube-apiserver證書
此證書的用途是apiserver對外提供的服務的證書
4.2.1 創建生成證書csr的json配置文件
此配置中的hosts包含所有可能會部署apiserver的列表
其中10.4.7.10是反向代理的vip地址
cat >/opt/certs/apiserver-csr.json <<EOF
{
"CN": "k8s-apiserver",
"hosts": [
"127.0.0.1",
"192.168.0.1",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "zq",
"OU": "ops"
}
]
}
EOF
4.2.2 生成kube-apiserver證書文件
cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-profile=server \
apiserver-csr.json |cfssl-json -bare apiserver
[root@hdss7-200 certs]# ll|grep apiserver
-rw-r--r-- 1 root root 1249 Apr 20 21:31 apiserver.csr
-rw-r--r-- 1 root root 566 Apr 20 21:31 apiserver-csr.json
-rw------- 1 root root 1675 Apr 20 21:31 apiserver-key.pem
-rw-r--r-- 1 root root 1590 Apr 20 21:31 apiserver.pem
4.3 下載安裝kube-apiserver
以7.21為例
# 上傳並解壓縮
tar xf kubernetes-server-linux-amd64-v1.15.2.tar.gz -C /opt
cd /opt
mv kubernetes/ kubernetes-v1.15.2
ln -s /opt/kubernetes-v1.15.2/ /opt/kubernetes
# 清理源碼包和docker鏡像
cd /opt/kubernetes
rm -rf kubernetes-src.tar.gz
cd server/bin
rm -f *.tar
rm -f *_tag
# 創建命令軟連接到系統環境變量下
ln -s /opt/kubernetes/server/bin/kubectl /usr/bin/kubectl
4.4 部署apiserver服務
4.4.1 拷貝證書文件
拷貝證書文件到/opt/kubernetes/server/bin/cert目錄下
# 創建目錄
mkdir -p /opt/kubernetes/server/bin/cert
cd /opt/kubernetes/server/bin/cert
# 拷貝三套證書
scp hdss7-200:/opt/certs/ca.pem .
scp hdss7-200:/opt/certs/ca-key.pem .
scp hdss7-200:/opt/certs/client.pem .
scp hdss7-200:/opt/certs/client-key.pem .
scp hdss7-200:/opt/certs/apiserver.pem .
scp hdss7-200:/opt/certs/apiserver-key.pem .
4.4.2 創建audit配置
audit日志審計規則配置是k8s要求必須要有得配置,可以不理解,直接用
mkdir /opt/kubernetes/server/conf
cat >/opt/kubernetes/server/conf/audit.yaml <<'EOF'
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
- level: RequestResponse
resources:
- group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
# Log "pods/log", "pods/status" at Metadata level
- level: Metadata
resources:
- group: ""
resources: ["pods/log", "pods/status"]
# Don't log requests to a configmap called "controller-leader"
- level: None
resources:
- group: ""
resources: ["configmaps"]
resourceNames: ["controller-leader"]
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
# Don't log authenticated requests to certain non-resource URL paths.
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
# Log the request body of configmap changes in kube-system.
- level: Request
resources:
- group: "" # core API group
resources: ["configmaps"]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
namespaces: ["kube-system"]
# Log configmap and secret changes in all other namespaces at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]
# Log all other resources in core and extensions at the Request level.
- level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
# A catch-all rule to log all other requests at the Metadata level.
- level: Metadata
# Long-running requests like watches that fall under this rule will not
# generate an audit event in RequestReceived.
omitStages:
- "RequestReceived"
EOF
4.4.3 創建apiserver啟動腳本
cat >/opt/kubernetes/server/bin/kube-apiserver.sh <<'EOF'
#!/bin/bash
./kube-apiserver \
--apiserver-count 2 \
--audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \
--audit-policy-file ../conf/audit.yaml \
--authorization-mode RBAC \
--client-ca-file ./cert/ca.pem \
--requestheader-client-ca-file ./cert/ca.pem \
--enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
--etcd-cafile ./cert/ca.pem \
--etcd-certfile ./cert/client.pem \
--etcd-keyfile ./cert/client-key.pem \
--etcd-servers https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
--service-account-key-file ./cert/ca-key.pem \
--service-cluster-ip-range 192.168.0.0/16 \
--service-node-port-range 3000-29999 \
--target-ram-mb=1024 \
--kubelet-client-certificate ./cert/client.pem \
--kubelet-client-key ./cert/client-key.pem \
--log-dir /data/logs/kubernetes/kube-apiserver \
--tls-cert-file ./cert/apiserver.pem \
--tls-private-key-file ./cert/apiserver-key.pem \
--v 2
EOF
# 授權
chmod +x /opt/kubernetes/server/bin/kube-apiserver.sh
4.4.4 創建supervisor啟動apiserver的配置
安裝supervisor軟件
yum install supervisor -y
systemctl start supervisord
systemctl enable supervisord
cat >/etc/supervisord.d/kube-apiserver.ini <<EOF
[program:kube-apiserver] ; 顯示的程序名,類似my.cnf,可以有多個
command=sh /opt/kubernetes/server/bin/kube-apiserver.sh
numprocs=1 ; 啟動進程數 (def 1)
directory=/opt/kubernetes/server/bin
autostart=true ; 是否自啟 (default: true)
autorestart=true ; 是否自動重啟 (default: true)
startsecs=30 ; 服務運行多久判斷為成功(def. 1)
startretries=3 ; 啟動重試次數 (default 3)
exitcodes=0,2 ; 退出狀態碼 (default 0,2)
stopsignal=QUIT ; 退出信號 (default TERM)
stopwaitsecs=10 ; 退出延遲時間 (default 10)
user=root ; 運行用戶
redirect_stderr=true ; 重定向錯誤輸出到標准輸出(def false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log
stdout_logfile_maxbytes=64MB ; 日志文件大小 (default 50MB)
stdout_logfile_backups=4 ; 日志文件滾動個數 (default 10)
stdout_capture_maxbytes=1MB ; 設定capture管道的大小(default 0)
;子進程還有子進程,需要添加這個參數,避免產生孤兒進程
killasgroup=true
stopasgroup=true
EOF
4.4.5 啟動apiserver服務並檢查
mkdir -p /data/logs/kubernetes/kube-apiserver
supervisorctl update
supervisorctl status
netstat -nltup|grep kube-api
4.4.6 部署啟動所有apiserver機器
集群其他機器的部署,沒有不同的地方,所以略
4.5 部署controller-manager服務
apiserve、controller-manager、kube-scheduler三個服務所需的軟件在同一套壓縮包里面的，因此后兩個服務不需要在單獨解包
而且這三個服務是在同一個主機上，互相之間通過http://127.0.0.1,也不需要證書
4.5.1 創建controller-manager啟動腳本
cat >/opt/kubernetes/server/bin/kube-controller-manager.sh <<'EOF'
#!/bin/sh
./kube-controller-manager \
--cluster-cidr 172.7.0.0/16 \
--leader-elect true \
--log-dir /data/logs/kubernetes/kube-controller-manager \
--master http://127.0.0.1:8080 \
--service-account-private-key-file ./cert/ca-key.pem \
--service-cluster-ip-range 192.168.0.0/16 \
--root-ca-file ./cert/ca.pem \
--v 2
EOF
# 授權
chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh
4.5.2 創建supervisor配置
cat >/etc/supervisord.d/kube-conntroller-manager.ini <<EOF
[program:kube-controller-manager] ; 顯示的程序名
command=sh /opt/kubernetes/server/bin/kube-controller-manager.sh
numprocs=1 ; 啟動進程數 (def 1)
directory=/opt/kubernetes/server/bin
autostart=true ; 是否自啟 (default: true)
autorestart=true ; 是否自動重啟 (default: true)
startsecs=30 ; 服務運行多久判斷為成功(def. 1)
startretries=3 ; 啟動重試次數 (default 3)
exitcodes=0,2 ; 退出狀態碼 (default 0,2)
stopsignal=QUIT ; 退出信號 (default TERM)
stopwaitsecs=10 ; 退出延遲時間 (default 10)
user=root ; 運行用戶
redirect_stderr=true ; 重定向錯誤輸出到標准輸出(def false)
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log
stdout_logfile_maxbytes=64MB ; 日志文件大小 (default 50MB)
stdout_logfile_backups=4 ; 日志文件滾動個數 (default 10)
stdout_capture_maxbytes=1MB ; 設定capture管道的大小(default 0)
;子進程還有子進程,需要添加這個參數,避免產生孤兒進程
killasgroup=true
stopasgroup=true
EOF
4.5.3 啟動服務並檢查
mkdir -p /data/logs/kubernetes/kube-controller-manager
supervisorctl update
supervisorctl status
4.5.4 部署啟動所有集群
沒有不同的地方,所以略
4.6 部署kube-scheduler服務
4.6.1 創建啟動腳本
cat >/opt/kubernetes/server/bin/kube-scheduler.sh <<'EOF'
#!/bin/sh
./kube-scheduler \
--leader-elect \
--log-dir /data/logs/kubernetes/kube-scheduler \
--master http://127.0.0.1:8080 \
--v 2
EOF
# 授權
chmod +x /opt/kubernetes/server/bin/kube-scheduler.sh
4.6.2 創建supervisor配置
cat >/etc/supervisord.d/kube-scheduler.ini <<EOF
[program:kube-scheduler]
command=sh /opt/kubernetes/server/bin/kube-scheduler.sh
numprocs=1 ; 啟動進程數 (def 1)
directory=/opt/kubernetes/server/bin
autostart=true ; 是否自啟 (default: true)
autorestart=true ; 是否自動重啟 (default: true)
startsecs=30 ; 服務運行多久判斷為成功(def. 1)
startretries=3 ; 啟動重試次數 (default 3)
exitcodes=0,2 ; 退出狀態碼 (default 0,2)
stopsignal=QUIT ; 退出信號 (default TERM)
stopwaitsecs=10 ; 退出延遲時間 (default 10)
user=root ; 運行用戶
redirect_stderr=true ; 重定向錯誤輸出到標准輸出(def false)
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log
stdout_logfile_maxbytes=64MB ; 日志文件大小 (default 50MB)
stdout_logfile_backups=4 ; 日志文件滾動個數 (default 10)
stdout_capture_maxbytes=1MB ; 設定capture管道的大小(default 0)
;子進程還有子進程,需要添加這個參數,避免產生孤兒進程
killasgroup=true
stopasgroup=true
EOF
4.6.3 啟動服務並檢查
mkdir -p /data/logs/kubernetes/kube-scheduler
supervisorctl update
supervisorctl status
4.6.4 部署啟動所有集群
沒有不同的地方,所以略
4.7 檢查master節點部署情況
[root@hdss7-21 bin]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
5 部署4層反代去代理apiserver
master節點上的3套服務部署完成后,需要使用反向代理去統一兩個apiservser的對外端口
這里使用nginx+keepalived的高可用架構部署在7.11和7.12兩台機器上
5.1 部署nginx四層反代
使用7443端口代理apiserver的6443端口,使用keepalived管理VIP10.4.7.10
5.1.1 yum安裝程序
yum install nginx keepalived -y
5.1.2 配置NGINX
四層代理不能寫在默認的conf.d目錄下,因為這個目錄默認是數據http模塊的include
所以要么把四層代理寫到主配置文件最下面,要么模仿七層代理創建一個四層代理文件夾
# 1. 在nginx配置文件中增加四層代理配置文件夾
mkdir /etc/nginx/tcp.d/
echo 'include /etc/nginx/tcp.d/*.conf;' >>/etc/nginx/nginx.conf
# 寫入代理配置
cat >/etc/nginx/tcp.d/apiserver.conf <<EOF
stream {
upstream kube-apiserver {
server 10.4.7.21:6443 max_fails=3 fail_timeout=30s;
server 10.4.7.22:6443 max_fails=3 fail_timeout=30s;
}
server {
listen 7443;
proxy_connect_timeout 2s;
proxy_timeout 900s;
proxy_pass kube-apiserver;
}
}
EOF
5.1.3 啟動nginx
nginx -t
systemctl start nginx
systemctl enable nginx
5.2 配置keepalived
5.2.1 創建端口監測腳本
創建腳本
cat >/etc/keepalived/check_port.sh <<'EOF'
#!/bin/bash
#keepalived 監控端口腳本
#使用方法：等待keepalived傳入端口參數,檢查改端口是否存在並返回結果
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l`
if [ $PORT_PROCESS -eq 0 ];then
echo "Port $CHK_PORT Is Not Used,End."
exit 1
fi
else
echo "Check Port Cant Be Empty!"
fi
EOF
給與腳本執行權限
chmod +x /etc/keepalived/check_port.sh
5.2.2 創建keepalived主配置文件
主機定義為10.4.7.11,從機定義為10.4.7.12
注意:主配置文件添加了nopreempt參數,非搶占式,意味着VIP發生漂移后,主重新啟動后也不會奪回VIP,目的是為了穩定性
cat >/etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
router_id 10.4.7.11
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 251
priority 100
advert_int 1
mcast_src_ip 10.4.7.11
nopreempt
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.4.7.10
}
}
EOF
5.2.3 創建keepalived從配置文件
cat >/etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
router_id 10.4.7.12
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 7443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 251
mcast_src_ip 10.4.7.12
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.4.7.10
}
}
EOF
5.3.4 啟動keepalived並驗證
systemctl start keepalived
systemctl enable keepalived
ip addr|grep '10.4.7.10'
6 部署node節點
6.1 簽發kubelet證書
簽發證書,都在7.200上
6.1.1 創建生成證書csr的json配置文件
cd /opt/certs/
cat >/opt/certs/kubelet-csr.json <<EOF
{
"CN": "k8s-kubelet",
"hosts": [
"127.0.0.1",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23",
"10.4.7.24",
"10.4.7.25",
"10.4.7.26",
"10.4.7.27",
"10.4.7.28"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "zq",
"OU": "ops"
}
]
}
EOF
6.1.2 生成kubelet證書文件
cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-profile=server \
kubelet-csr.json | cfssl-json -bare kubelet
[root@hdss7-200 certs]# ll |grep kubelet
-rw-r--r-- 1 root root 1115 Apr 22 22:17 kubelet.csr
-rw-r--r-- 1 root root 452 Apr 22 22:17 kubelet-csr.json
-rw------- 1 root root 1679 Apr 22 22:17 kubelet-key.pem
-rw-r--r-- 1 root root 1460 Apr 22 22:17 kubelet.pem
6.2 創建kubelet服務
6.2.1 拷貝證書至node節點
cd /opt/kubernetes/server/bin/cert
scp hdss7-200:/opt/certs/kubelet.pem .
scp hdss7-200:/opt/certs/kubelet-key.pem .
6.2.2 創建kubelet配置
創建kubelet的配置文件kubelet.kubeconfig比較麻煩,需要四步操作才能完成
(1) set-cluster(設置集群參數)
使用ca證書創建集群myk8s,使用的apiserver信息是10.4.7.10這個VIP
cd /opt/kubernetes/server/conf/
kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=kubelet.kubeconfig
(2) set-credentials(設置客戶端認證參數)
使用client證書創建用戶k8s-node
kubectl config set-credentials k8s-node \
--client-certificate=/opt/kubernetes/server/bin/cert/client.pem \
--client-key=/opt/kubernetes/server/bin/cert/client-key.pem \
--embed-certs=true \
--kubeconfig=kubelet.kubeconfig
(3) set-context(綁定namespace)
創建myk8s-context,關聯集群myk8s和用戶k8s-node
kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=k8s-node \
--kubeconfig=kubelet.kubeconfig
(4) use-context
使用生成的配置文件向apiserver注冊,注冊信息會寫入etcd,所以只需要注冊一次即可
kubectl config use-context myk8s-context --kubeconfig=kubelet.kubeconfig
(5) 查看生成的kubelet.kubeconfig
[root@hdss7-21 conf]# cat kubelet.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: xxxxxxxx
server: https://10.4.7.10:7443
name: myk8s
contexts:
- context:
cluster: myk8s
user: k8s-node
name: myk8s-context
current-context: myk8s-context
kind: Config
preferences: {}
users:
- name: k8s-node
user:
client-certificate-data: xxxxxxxx
client-key-data: xxxxxxxx
可以看出來,這個配置文件里面包含了集群名字,用戶名字,集群認證的公鑰,用戶的公私鑰等
6.2.3 創建k8s-node.yaml配置文件
cat >/opt/kubernetes/server/conf/k8s-node.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: k8s-node
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
EOF
使用RBAC鑒權規則,創建了一個
ClusterRoleBinding的資源
此資源中定義了一個user叫k8s-node
給k8s-node用戶綁定了角色ClusterRole,角色名為system:node
使這個用戶具有成為集群運算節點角色的權限
由於這個用戶名,同時也是kubeconfig中指定的用戶,
所以通過kubeconfig配置啟動的kubelet節點,就能夠成為node節點
6.2.4 應用資源配置
應用資源配置,並查看結果
# 應用資源配置
kubectl create -f /opt/kubernetes/server/conf/k8s-node.yaml
# 查看集群角色和角色屬性
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node
NAME AGE
k8s-node 13s
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2020-04-22T14:38:09Z"
name: k8s-node
resourceVersion: "21217"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/k8s-node
uid: 597ffb0f-f92d-4eb5-aca2-2fe73397e2e4
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
#此時只是創建了相應的資源,還沒有具體的node,如下驗證
[root@hdss7-21 conf]# kubectl get nodes
No resources found.
6.2.5 創建kubelet啟動腳本
--hostname-override參數每個node節點都一樣,是節點的主機名,注意修改
cat >/opt/kubernetes/server/bin/kubelet.sh <<'EOF'
#!/bin/sh
./kubelet \
--hostname-override hdss7-21.host.com \
--anonymous-auth=false \
--cgroup-driver systemd \
--cluster-dns 192.168.0.2 \
--cluster-domain cluster.local \
--runtime-cgroups=/systemd/system.slice \
--kubelet-cgroups=/systemd/system.slice \
--fail-swap-on="false" \
--client-ca-file ./cert/ca.pem \
--tls-cert-file ./cert/kubelet.pem \
--tls-private-key-file ./cert/kubelet-key.pem \
--image-gc-high-threshold 20 \
--image-gc-low-threshold 10 \
--kubeconfig ../conf/kubelet.kubeconfig \
--log-dir /data/logs/kubernetes/kube-kubelet \
--pod-infra-container-image harbor.zq.com/public/pause:latest \
--root-dir /data/kubelet
EOF
# 創建目錄&授權
chmod +x /opt/kubernetes/server/bin/kubelet.sh
mkdir -p /data/logs/kubernetes/kube-kubelet
mkdir -p /data/kubelet
6.2.6 創建supervisor配置
cat >/etc/supervisord.d/kube-kubelet.ini <<EOF
[program:kube-kubelet]
command=sh /opt/kubernetes/server/bin/kubelet.sh
numprocs=1 ; 啟動進程數 (def 1)
directory=/opt/kubernetes/server/bin
autostart=true ; 是否自啟 (default: true)
autorestart=true ; 是否自動重啟 (default: true)
startsecs=30 ; 服務運行多久判斷為成功(def. 1)
startretries=3 ; 啟動重試次數 (default 3)
exitcodes=0,2 ; 退出狀態碼 (default 0,2)
stopsignal=QUIT ; 退出信號 (default TERM)
stopwaitsecs=10 ; 退出延遲時間 (default 10)
user=root ; 運行用戶
redirect_stderr=true ; 重定向錯誤輸出到標准輸出(def false)
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
stdout_logfile_maxbytes=64MB ; 日志文件大小 (default 50MB)
stdout_logfile_backups=4 ; 日志文件滾動個數 (default 10)
stdout_capture_maxbytes=1MB ; 設定capture管道的大小(default 0)
;子進程還有子進程,需要添加這個參數,避免產生孤兒進程
killasgroup=true
stopasgroup=true
EOF
6.2.7 啟動服務並檢查
supervisorctl update
supervisorctl status
[root@hdss7-21 server]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready <none> 65s v1.15.5
6.2.8 部署其他node節點
第一個節點部署完成后,其他節點就要簡單很多,只需拷貝kubelet.kubeconfig配置到本地后,創建啟動腳本並用`supervisord啟動即可
也可以不拷貝配置文件,就需要手動再執行創建配置文件的四步
# 拷貝證書
cd /opt/kubernetes/server/bin/cert
scp hdss7-200:/opt/certs/kubelet.pem .
scp hdss7-200:/opt/certs/kubelet-key.pem .
# 拷貝配置文件
cd /opt/kubernetes/server/conf/
scp hdss7-21:/opt/kubernetes/server/conf/kubelet.kubeconfig .
拷貝完配置后,剩下的步驟參考6.2.5 創建kubelet啟動腳本,除腳本中--hostname-override不同外,其他都一樣
6.2.9 檢查所有節點並給節點打上標簽
此操作非必須,因為只是打的一個標簽,方便識別而已
kubectl get nodes
kubectl label node hdss7-21.host.com node-role.kubernetes.io/master=
kubectl label node hdss7-21.host.com node-role.kubernetes.io/node=
[root@hdss7-22 cert]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready master,node 9m v1.15.5
hdss7-22.host.com Ready <none> 64s v1.15.5
6.3 創建kube-proxy服務
簽發證書在7.200上
6.3.1 簽發kube-proxy證書
(1) 創建生成證書csr的json配置文件
cd /opt/certs/
cat >/opt/certs/kube-proxy-csr.json <<EOF
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "zq",
"OU": "ops"
}
]
}
EOF
(2) 生成kube-proxy證書文件
cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-profile=client \
kube-proxy-csr.json |cfssl-json -bare kube-proxy-client
(3) 檢查生成的證書文件
[root@hdss7-200 certs]# ll |grep proxy
-rw-r--r-- 1 root root 1005 Apr 22 22:54 kube-proxy-client.csr
-rw------- 1 root root 1675 Apr 22 22:54 kube-proxy-client-key.pem
-rw-r--r-- 1 root root 1371 Apr 22 22:54 kube-proxy-client.pem
-rw-r--r-- 1 root root 267 Apr 22 22:54 kube-proxy-csr.json
6.3.2 拷貝證書文件至各節點
cd /opt/kubernetes/server/bin/cert
scp hdss7-200:/opt/certs/kube-proxy-client.pem .
scp hdss7-200:/opt/certs/kube-proxy-client-key.pem .
6.3.3 創建kube-proxy配置
同樣是四步操作,類似kubelet
(1) set-cluster
cd /opt/kubernetes/server/conf/
kubectl config set-cluster myk8s \
--certificate-authority=/opt/kubernetes/server/bin/cert/ca.pem \
--embed-certs=true \
--server=https://10.4.7.10:7443 \
--kubeconfig=kube-proxy.kubeconfig
(2) set-credentials
kubectl config set-credentials kube-proxy \
--client-certificate=/opt/kubernetes/server/bin/cert/kube-proxy-client.pem \
--client-key=/opt/kubernetes/server/bin/cert/kube-proxy-client-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
(3) set-context
kubectl config set-context myk8s-context \
--cluster=myk8s \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
(4) use-context
kubectl config use-context myk8s-context --kubeconfig=kube-proxy.kubeconfig
6.3.4 加載ipvs模塊以備kube-proxy啟動用
# 創建開機ipvs腳本
cat >/etc/ipvs.sh <<'EOF'
#!/bin/bash
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"
for i in $(ls $ipvs_mods_dir|grep -o "^[^.]*")
do
/sbin/modinfo -F filename $i &>/dev/null
if [ $? -eq 0 ];then
/sbin/modprobe $i
fi
done
EOF
# 執行腳本開啟ipvs
sh /etc/ipvs.sh
# 驗證開啟結果
[root@hdss7-21 conf]# lsmod |grep ip_vs
ip_vs_wrr 12697 0
ip_vs_wlc 12519 0
......略
6.3.5 創建kube-proxy啟動腳本
同上, --hostname-override參數在不同的node節點上不一樣,需修改
cat >/opt/kubernetes/server/bin/kube-proxy.sh <<'EOF'
#!/bin/sh
./kube-proxy \
--hostname-override hdss7-21.host.com \
--cluster-cidr 172.7.0.0/16 \
--proxy-mode=ipvs \
--ipvs-scheduler=nq \
--kubeconfig ../conf/kube-proxy.kubeconfig
EOF
# 授權
chmod +x /opt/kubernetes/server/bin/kube-proxy.sh
6.3.6 創建kube-proxy的supervisor配置
cat >/etc/supervisord.d/kube-proxy.ini <<'EOF'
[program:kube-proxy]
command=sh /opt/kubernetes/server/bin/kube-proxy.sh
numprocs=1 ; 啟動進程數 (def 1)
directory=/opt/kubernetes/server/bin
autostart=true ; 是否自啟 (default: true)
autorestart=true ; 是否自動重啟 (default: true)
startsecs=30 ; 服務運行多久判斷為成功(def. 1)
startretries=3 ; 啟動重試次數 (default 3)
exitcodes=0,2 ; 退出狀態碼 (default 0,2)
stopsignal=QUIT ; 退出信號 (default TERM)
stopwaitsecs=10 ; 退出延遲時間 (default 10)
user=root ; 運行用戶
redirect_stderr=true ; 重定向錯誤輸出到標准輸出(def false)
stdout_logfile=/data/logs/kubernetes/kube-proxy/proxy.stdout.log
stdout_logfile_maxbytes=64MB ; 日志文件大小 (default 50MB)
stdout_logfile_backups=4 ; 日志文件滾動個數 (default 10)
stdout_capture_maxbytes=1MB ; 設定capture管道的大小(default 0)
;子進程還有子進程,需要添加這個參數,避免產生孤兒進程
killasgroup=true
stopasgroup=true
EOF
6.3.7 啟動服務並檢查
mkdir -p /data/logs/kubernetes/kube-proxy
supervisorctl update
supervisorctl status
[root@hdss7-21 conf]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 47h
# 檢查ipvs,是否新增了配置
yum install ipvsadm -y
[root@hdss7-21 conf]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.1:443 nq
-> 10.4.7.21:6443 Masq 1 0 0
-> 10.4.7.22:6443 Masq 1 0 0
6.3.8 部署所有節點
首先需拷貝kube-proxy.kubeconfig 到 hdss7-22.host.com的conf目錄下
# 拷貝證書文件
cd /opt/kubernetes/server/bin/cert
scp hdss7-200:/opt/certs/kube-proxy-client.pem .
scp hdss7-200:/opt/certs/kube-proxy-client-key.pem .
# 拷貝配置文件
cd /opt/kubernetes/server/conf/
scp hdss7-21:/opt/kubernetes/server/conf/kube-proxy.kubeconfig .
其他不同的地方就一個主機名,都已經在前面說明了,略
7 驗證kubernetes集群
7.1 在任意一個節點上創建一個資源配置清單
cat >/root/nginx-ds.yaml <<'EOF'
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: nginx-ds
spec:
template:
metadata:
labels:
app: nginx-ds
spec:
containers:
- name: my-nginx
image: harbor.zq.com/public/nginx:v1.17.9
ports:
- containerPort: 80
EOF
7.2 應用資源配置，並檢查
7.2.1 應用資源配置
kubectl create -f /root/nginx-ds.yaml
[root@hdss7-22 conf]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ds-j777c 1/1 Running 0 8s
nginx-ds-nwsd6 1/1 Running 0 8s
7.2.2 在另一台node節點上檢查
kubectl get pods
kubectl get pods -o wide
curl 172.7.22.2
7.2.3 查看kubernetes是否搭建好
[root@hdss7-22 conf]# kubectl get cs
NAME STATUS MESSAGE ERROR
etcd-0 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
controller-manager Healthy ok
scheduler Healthy ok
[root@hdss7-21 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready master,node 6d1h v1.15.5
hdss7-22.host.com Ready <none> 6d1h v1.15.5
[root@hdss7-22 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ds-j777c 1/1 Running 0 6m45s
nginx-ds-nwsd6 1/1 Running 0 6m45s