二進制部署k8s集群（2）： 簽發etcd證書，安裝etcd集群

【前期准備】

下載 etcd 二進制安裝包：https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

下載 kubernetes 1.18.3 二進制安裝包：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183 

注意：打開鏈接有很多下載包，包含kubernetes-client 、kubernetes-server 、kubernetes-node ，下載其中一個64位的就行。

 

安裝證書簽發工具cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

cp cfssl_linux-amd64 /usr/local/bin/cfssl

cp cfssljson_linux-amd64 /usr/local/bin/cfssljson

cp cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

下載etcd安裝包

 

下載etcd-v3.4.9二進制包

下載etcd-v.3.4.9二進制安裝包，

其它版本下載地址： https://github.com/etcd-io/etcd/tags 

wget  https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

 

下載完成后，解壓etcd安裝包，復制解壓目錄里的etcd與etcdctl 兩個文件到 /usr/local/bin 目錄下面，並且賦予兩個文件可執行權限

tar xfv etcd-v3.4.9-linux-amd64.tar.gz
cd etcd-v3.4.9-linux-amd64
cp etcd /usr/local/bin 
cp etcdctl /usr/local/bin
#賦予執行權限
chmod +x /usr/local/bin/etcd
chmod +x /usr/local/bin/etcdctl

 

簽發etcd證書

創建證書存放目錄。

mkdir -p /opt/certs

 

【創建證書】

首先安裝cfssl 證書制作工具，安裝方法參考：https://www.cnblogs.com/yyee/p/13189331.html

在etcd01 (192.168.0.102)節點上創建證書。

(1) 創建根證書配置文件

vi  /opt/certs/ca-config.json

{
  "signing": { "default": { "expiry": "175200h" }, "profiles":{ "k8s-server": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth" ] }, "k8s-client": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "client auth" ] }, "k8s-server-client": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } }

 

signing：表示該證書可用於簽名其它證書（生成的 ca.pem 證書中 CA=TRUE）；

server auth：表示 client 可以用該該證書對 server 提供的證書進行驗證；

client auth：表示 server 可以用該該證書對 client 提供的證書進行驗證；

"expiry": "175200h"：證書有效期設置為 20 年； 

 

(2) 創建根證書請求文件 ca-csr.json

vi /opt/certs/ca-csr.json

{
  "CN": "k8s", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" , "O": "k8s", "OU": "system" } ] }

CN：Common Name：kube-apiserver 從證書中提取該字段作為請求的用戶名 (User Name)，

瀏覽器使用該字段驗證網站是否合法；

O：Organization：kube-apiserver 從證書中提取該字段作為請求用戶所屬的組 (Group)；

kube-apiserver 將提取的 User、Group 作為 RBAC 授權的用戶標識； 

 

(3) 創建etcd證書請求文件etcd-peer-csr.json

vi /opt/certs/etcd-peer-csr.json

{
  "CN": "k8s-etcd", "hosts": [ "192.168.0.101", "192.168.0.102", "192.168.0.103", "192.168.0.104", "192.168.0.105", "192.168.0.106" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "k8s", "OU": "system" } ] }

  

三個json文件編輯完在之后，/opt/certs 目錄有三個json文件。

  

(4) 生成 ca 證書和私鑰 

cd /opt/certs
#生成 ca 證書和私鑰
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

 

 

 生成了ca.csr, ca-key.pen, ca.pem三個私鑰與證書文件。

 

(5) 生成etcd用的證書文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=k8s-server-client etcd-peer-csr.json | cfssljson -bare etcd-peer

注意，-profile=k8s-server-client，表示客戶端與服務端要雙向通訊。| cfssljson -bare etcd-peer 表示生成證書文件名為 etcd-peer 。

這次生成了etcd-peer.csr， etcd-peer-key.pem，etcd-peer.pem 三個文件

 最終產生這幾個文件： ca-config.json， ca.csr， ca-csr.json ， ca-key.pem， ca.pem， etcd-peer.csr， etcd-peer-csr.json ，etcd-peer-key.pem，etcd-peer.pem 

  

(6) copy證書到其它兩個節點 

 把ca.pem, etcd-peer.pem, etcd-peer-key.pem三個證書拷貝到etcd02與etcd03節點的【/opt/etcd/certs】目錄，etcd只用到三個證書。

cd  /opt/certs
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.102:/opt/etcd/certs/
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.103:/opt/etcd/certs/

 

 

安裝etcd可以使用SSL證書安裝，也可以不使用SSL證書安裝。

安裝etcd (不使用SSL證書安裝)

將etcd集群安裝在三個節點上，三個實例節點信息為：

 

etcd實例名稱	IP地址	Hostname
etcd01	192.168.0.102	yyee-centos-2
ctcd02	192.168.0.103	yyee-centos-3
etcd03	192.168.0.104	yyee-centos-4

 

 

 

 

 

 

(1) 在三個節點上創建工作目錄

 

mkdir -p /var/lib/etcd/data

  

 (2) 編寫etcd啟動文件

編寫 etcd01, etcd02, etcd03 三個節點的etcd啟動文件，然后三個節點要同時啟動才能啟動etcd集群成功。

【編寫 etcd01節點的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \ --name=etcd01 \ --data-dir=/var/lib/etcd/data \ --listen-peer-urls=http://192.168.0.102:2380 \ --listen-client-urls=http://192.168.0.102:2379,http://127.0.0.1:2379 \ --initial-advertise-peer-urls=http://192.168.0.102:2380 \ --advertise-client-urls=http://192.168.0.102:2379,http://127.0.0.1 \ --initial-cluster=etcd01=http://192.168.0.102:2380,etcd02=http://192.168.0.103:2380,etcd03=http://192.168.0.104:2380 \ --initial-cluster-token=k8s-etcd-cluster \ --initial-cluster-state=new" Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target

 

【編寫 etcd02節點的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \ --name=etcd02 \ --data-dir=/var/lib/etcd/data \ --listen-peer-urls=http://192.168.0.103:2380 \ --listen-client-urls=http://192.168.0.103:2379,http://127.0.0.1:2379 \ --initial-advertise-peer-urls=http://192.168.0.103:2380 \ --advertise-client-urls=http://192.168.0.103:2379,http://127.0.0.1 \ --initial-cluster=etcd01=http://192.168.0.102:2380,etcd02=http://192.168.0.103:2380,etcd03=http://192.168.0.104:2380 \ --initial-cluster-token=k8s-etcd-cluster \ --initial-cluster-state=new" Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target

 

【編寫 etcd03節點的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \ --name=etcd03 \ --data-dir=/var/lib/etcd/data \ --listen-peer-urls=http://192.168.0.104:2380 \ --listen-client-urls=http://192.168.0.104:2379,http://127.0.0.1:2379 \ --initial-advertise-peer-urls=http://192.168.0.104:2380 \ --advertise-client-urls=http://192.168.0.104:2379,http://127.0.0.1 \ --initial-cluster=etcd01=http://192.168.0.102:2380,etcd02=http://192.168.0.103:2380,etcd03=http://192.168.0.104:2380 \ --initial-cluster-token=k8s-etcd-cluster \ --initial-cluster-state=new" Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target 

 

(3) 啟動etcd

 然后三個節點要同時執行訓動命令才能成功啟動etcd。

systemctl daemon-reload
systemctl enable etcd

#這條命令要在三個節點上同時執行，第一個執行的節點會最多等待30秒讓其它兩個節點加入集群。
systemctl start etcd  

 

啟動如果沒報錯的話就啟動成功了，查看集群狀態

etcdctl member list

 

 

 

查看etcd監聽端口

netstat -tunlp | grep etcd

 

 

  

安裝etcd (使用SSL證書安裝)

將etcd集群安裝在三個節點上，三個實例節點信息為：

 

etcd實例名稱	IP地址	Hostname
etcd01	192.168.0.102	yyee-centos-2
ctcd02	192.168.0.103	yyee-centos-3
etcd03	192.168.0.104	yyee-centos-4

 

 

 

 

 

 

(1) 在三個節點上創建工作目錄

 

mkdir -p /var/lib/etcd/data
mkdir -p /opt/certs

 

 

(2) copy證書到其它兩個節點 

 把192.168.0.102:/opt/certs/  目錄下的ca.pem, etcd-peer.pem, etcd-peer-key.pem三個證書文件拷貝到etcd02節點與etcd03節點上。

cd  /opt/certs
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.103:/opt/certs/
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.104:/opt/certs/

 

 (3) 編寫etcd啟動文件

編寫 etcd01, etcd02, etcd03 三個節點的etcd啟動文件，然后三個節點要同時啟動才能啟動etcd集群成功。

【編寫 etcd01節點的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \
 --name=etcd01 \
 --data-dir=/var/lib/etcd/data \
 --listen-peer-urls=https://192.168.0.102:2380 \
 --listen-client-urls=https://192.168.0.102:2379,http://127.0.0.1:2379 \
 --initial-advertise-peer-urls=https://192.168.0.102:2380 \
 --advertise-client-urls=https://192.168.0.102:2379 \
 --initial-cluster=etcd01=https://192.168.0.102:2380,etcd02=https://192.168.0.103:2380,etcd03=https://192.168.0.104:2380 \
 --initial-cluster-token=k8s-etcd-cluster \
 --initial-cluster-state=new \
 --cert-file=/opt/etcd/certs/etcd-peer.pem \
 --key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --peer-cert-file=/opt/etcd/certs/etcd-peer.pem \
 --peer-key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --trusted-ca-file=/opt/etcd/certs/ca.pem \
 --peer-trusted-ca-file=/opt/etcd/certs/ca.pem"


Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

 

【編寫 etcd02節點的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \
 --name=etcd02 \
 --data-dir=/var/lib/etcd/data \
 --listen-peer-urls=https://192.168.0.103:2380 \
 --listen-client-urls=https://192.168.0.103:2379,http://127.0.0.1:2379 \
 --initial-advertise-peer-urls=https://192.168.0.103:2380 \
 --advertise-client-urls=https://192.168.0.103:2379 \
 --initial-cluster=etcd01=https://192.168.0.102:2380,etcd02=https://192.168.0.103:2380,etcd03=https://192.168.0.104:2380 \
 --initial-cluster-token=k8s-etcd-cluster \
 --initial-cluster-state=new \
 --cert-file=/opt/etcd/certs/etcd-peer.pem \
 --key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --peer-cert-file=/opt/etcd/certs/etcd-peer.pem \
 --peer-key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --trusted-ca-file=/opt/etcd/certs/ca.pem \
 --peer-trusted-ca-file=/opt/etcd/certs/ca.pem"


Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

 

【編寫 etcd03節點的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \
 --name=etcd03 \
 --data-dir=/var/lib/etcd/data \
 --listen-peer-urls=https://192.168.0.104:2380 \
 --listen-client-urls=https://192.168.0.104:2379,http://127.0.0.1:2379 \
 --initial-advertise-peer-urls=https://192.168.0.104:2380 \
 --advertise-client-urls=https://192.168.0.104:2379 \
 --initial-cluster=etcd01=https://192.168.0.102:2380,etcd02=https://192.168.0.103:2380,etcd03=https://192.168.0.104:2380 \
 --initial-cluster-token=k8s-etcd-cluster \
 --initial-cluster-state=new \
 --cert-file=/opt/etcd/certs/etcd-peer.pem \
 --key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --peer-cert-file=/opt/etcd/certs/etcd-peer.pem \
 --peer-key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --trusted-ca-file=/opt/etcd/certs/ca.pem \
 --peer-trusted-ca-file=/opt/etcd/certs/ca.pem"


Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target 

 

(4) 啟動etcd

 然后三個節點要同時執行訓動命令才能成功啟動etcd。

systemctl daemon-reload
systemctl enable etcd

#這條命令要在三個節點上同時執行，第一個執行的節點會最多等待30秒讓其它兩個節點加入集群。
systemctl start etcd  

 

啟動如果沒報錯的話就啟動成功了，查看集群狀態

etcdctl member list

 

 

 

查看etcd監聽端口

netstat -tunlp | grep etcd