二进制部署k8s集群（2）： 签发etcd证书，安装etcd集群

【前期准备】

下载 etcd 二进制安装包：https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

下载 kubernetes 1.18.3 二进制安装包：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183 

注意：打开链接有很多下载包，包含kubernetes-client 、kubernetes-server 、kubernetes-node ，下载其中一个64位的就行。

 

安装证书签发工具cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

cp cfssl_linux-amd64 /usr/local/bin/cfssl

cp cfssljson_linux-amd64 /usr/local/bin/cfssljson

cp cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

下载etcd安装包

 

下载etcd-v3.4.9二进制包

下载etcd-v.3.4.9二进制安装包，

其它版本下载地址： https://github.com/etcd-io/etcd/tags 

wget  https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

 

下载完成后，解压etcd安装包，复制解压目录里的etcd与etcdctl 两个文件到 /usr/local/bin 目录下面，并且赋予两个文件可执行权限

tar xfv etcd-v3.4.9-linux-amd64.tar.gz
cd etcd-v3.4.9-linux-amd64
cp etcd /usr/local/bin 
cp etcdctl /usr/local/bin
#赋予执行权限
chmod +x /usr/local/bin/etcd
chmod +x /usr/local/bin/etcdctl

 

签发etcd证书

创建证书存放目录。

mkdir -p /opt/certs

 

【创建证书】

首先安装cfssl 证书制作工具，安装方法参考：https://www.cnblogs.com/yyee/p/13189331.html

在etcd01 (192.168.0.102)节点上创建证书。

(1) 创建根证书配置文件

vi  /opt/certs/ca-config.json

{
  "signing": { "default": { "expiry": "175200h" }, "profiles":{ "k8s-server": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth" ] }, "k8s-client": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "client auth" ] }, "k8s-server-client": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } }

 

signing：表示该证书可用于签名其它证书（生成的 ca.pem 证书中 CA=TRUE）；

server auth：表示 client 可以用该该证书对 server 提供的证书进行验证；

client auth：表示 server 可以用该该证书对 client 提供的证书进行验证；

"expiry": "175200h"：证书有效期设置为 20 年； 

 

(2) 创建根证书请求文件 ca-csr.json

vi /opt/certs/ca-csr.json

{
  "CN": "k8s", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" , "O": "k8s", "OU": "system" } ] }

CN：Common Name：kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)，

浏览器使用该字段验证网站是否合法；

O：Organization：kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；

kube-apiserver 将提取的 User、Group 作为 RBAC 授权的用户标识； 

 

(3) 创建etcd证书请求文件etcd-peer-csr.json

vi /opt/certs/etcd-peer-csr.json

{
  "CN": "k8s-etcd", "hosts": [ "192.168.0.101", "192.168.0.102", "192.168.0.103", "192.168.0.104", "192.168.0.105", "192.168.0.106" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "k8s", "OU": "system" } ] }

  

三个json文件编辑完在之后，/opt/certs 目录有三个json文件。

  

(4) 生成 ca 证书和私钥 

cd /opt/certs
#生成 ca 证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

 

 

 生成了ca.csr, ca-key.pen, ca.pem三个私钥与证书文件。

 

(5) 生成etcd用的证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=k8s-server-client etcd-peer-csr.json | cfssljson -bare etcd-peer

注意，-profile=k8s-server-client，表示客户端与服务端要双向通讯。| cfssljson -bare etcd-peer 表示生成证书文件名为 etcd-peer 。

这次生成了etcd-peer.csr， etcd-peer-key.pem，etcd-peer.pem 三个文件

 最终产生这几个文件： ca-config.json， ca.csr， ca-csr.json ， ca-key.pem， ca.pem， etcd-peer.csr， etcd-peer-csr.json ，etcd-peer-key.pem，etcd-peer.pem 

  

(6) copy证书到其它两个节点 

 把ca.pem, etcd-peer.pem, etcd-peer-key.pem三个证书拷贝到etcd02与etcd03节点的【/opt/etcd/certs】目录，etcd只用到三个证书。

cd  /opt/certs
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.102:/opt/etcd/certs/
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.103:/opt/etcd/certs/

 

 

安装etcd可以使用SSL证书安装，也可以不使用SSL证书安装。

安装etcd (不使用SSL证书安装)

将etcd集群安装在三个节点上，三个实例节点信息为：

 

etcd实例名称	IP地址	Hostname
etcd01	192.168.0.102	yyee-centos-2
ctcd02	192.168.0.103	yyee-centos-3
etcd03	192.168.0.104	yyee-centos-4

 

 

 

 

 

 

(1) 在三个节点上创建工作目录

 

mkdir -p /var/lib/etcd/data

  

 (2) 编写etcd启动文件

编写 etcd01, etcd02, etcd03 三个节点的etcd启动文件，然后三个节点要同时启动才能启动etcd集群成功。

【编写 etcd01节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \ --name=etcd01 \ --data-dir=/var/lib/etcd/data \ --listen-peer-urls=http://192.168.0.102:2380 \ --listen-client-urls=http://192.168.0.102:2379,http://127.0.0.1:2379 \ --initial-advertise-peer-urls=http://192.168.0.102:2380 \ --advertise-client-urls=http://192.168.0.102:2379,http://127.0.0.1 \ --initial-cluster=etcd01=http://192.168.0.102:2380,etcd02=http://192.168.0.103:2380,etcd03=http://192.168.0.104:2380 \ --initial-cluster-token=k8s-etcd-cluster \ --initial-cluster-state=new" Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target

 

【编写 etcd02节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \ --name=etcd02 \ --data-dir=/var/lib/etcd/data \ --listen-peer-urls=http://192.168.0.103:2380 \ --listen-client-urls=http://192.168.0.103:2379,http://127.0.0.1:2379 \ --initial-advertise-peer-urls=http://192.168.0.103:2380 \ --advertise-client-urls=http://192.168.0.103:2379,http://127.0.0.1 \ --initial-cluster=etcd01=http://192.168.0.102:2380,etcd02=http://192.168.0.103:2380,etcd03=http://192.168.0.104:2380 \ --initial-cluster-token=k8s-etcd-cluster \ --initial-cluster-state=new" Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target

 

【编写 etcd03节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \ --name=etcd03 \ --data-dir=/var/lib/etcd/data \ --listen-peer-urls=http://192.168.0.104:2380 \ --listen-client-urls=http://192.168.0.104:2379,http://127.0.0.1:2379 \ --initial-advertise-peer-urls=http://192.168.0.104:2380 \ --advertise-client-urls=http://192.168.0.104:2379,http://127.0.0.1 \ --initial-cluster=etcd01=http://192.168.0.102:2380,etcd02=http://192.168.0.103:2380,etcd03=http://192.168.0.104:2380 \ --initial-cluster-token=k8s-etcd-cluster \ --initial-cluster-state=new" Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target 

 

(3) 启动etcd

 然后三个节点要同时执行训动命令才能成功启动etcd。

systemctl daemon-reload
systemctl enable etcd

#这条命令要在三个节点上同时执行，第一个执行的节点会最多等待30秒让其它两个节点加入集群。
systemctl start etcd  

 

启动如果没报错的话就启动成功了，查看集群状态

etcdctl member list

 

 

 

查看etcd监听端口

netstat -tunlp | grep etcd

 

 

  

安装etcd (使用SSL证书安装)

将etcd集群安装在三个节点上，三个实例节点信息为：

 

etcd实例名称	IP地址	Hostname
etcd01	192.168.0.102	yyee-centos-2
ctcd02	192.168.0.103	yyee-centos-3
etcd03	192.168.0.104	yyee-centos-4

 

 

 

 

 

 

(1) 在三个节点上创建工作目录

 

mkdir -p /var/lib/etcd/data
mkdir -p /opt/certs

 

 

(2) copy证书到其它两个节点 

 把192.168.0.102:/opt/certs/  目录下的ca.pem, etcd-peer.pem, etcd-peer-key.pem三个证书文件拷贝到etcd02节点与etcd03节点上。

cd  /opt/certs
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.103:/opt/certs/
scp ca.pem  etcd-peer.pem  etcd-peer-key.pem  192.168.0.104:/opt/certs/

 

 (3) 编写etcd启动文件

编写 etcd01, etcd02, etcd03 三个节点的etcd启动文件，然后三个节点要同时启动才能启动etcd集群成功。

【编写 etcd01节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \
 --name=etcd01 \
 --data-dir=/var/lib/etcd/data \
 --listen-peer-urls=https://192.168.0.102:2380 \
 --listen-client-urls=https://192.168.0.102:2379,http://127.0.0.1:2379 \
 --initial-advertise-peer-urls=https://192.168.0.102:2380 \
 --advertise-client-urls=https://192.168.0.102:2379 \
 --initial-cluster=etcd01=https://192.168.0.102:2380,etcd02=https://192.168.0.103:2380,etcd03=https://192.168.0.104:2380 \
 --initial-cluster-token=k8s-etcd-cluster \
 --initial-cluster-state=new \
 --cert-file=/opt/etcd/certs/etcd-peer.pem \
 --key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --peer-cert-file=/opt/etcd/certs/etcd-peer.pem \
 --peer-key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --trusted-ca-file=/opt/etcd/certs/ca.pem \
 --peer-trusted-ca-file=/opt/etcd/certs/ca.pem"


Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

 

【编写 etcd02节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \
 --name=etcd02 \
 --data-dir=/var/lib/etcd/data \
 --listen-peer-urls=https://192.168.0.103:2380 \
 --listen-client-urls=https://192.168.0.103:2379,http://127.0.0.1:2379 \
 --initial-advertise-peer-urls=https://192.168.0.103:2380 \
 --advertise-client-urls=https://192.168.0.103:2379 \
 --initial-cluster=etcd01=https://192.168.0.102:2380,etcd02=https://192.168.0.103:2380,etcd03=https://192.168.0.104:2380 \
 --initial-cluster-token=k8s-etcd-cluster \
 --initial-cluster-state=new \
 --cert-file=/opt/etcd/certs/etcd-peer.pem \
 --key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --peer-cert-file=/opt/etcd/certs/etcd-peer.pem \
 --peer-key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --trusted-ca-file=/opt/etcd/certs/ca.pem \
 --peer-trusted-ca-file=/opt/etcd/certs/ca.pem"


Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

 

【编写 etcd03节点的 etcd.service 文件】

vi  /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/local/bin/etcd \
 --name=etcd03 \
 --data-dir=/var/lib/etcd/data \
 --listen-peer-urls=https://192.168.0.104:2380 \
 --listen-client-urls=https://192.168.0.104:2379,http://127.0.0.1:2379 \
 --initial-advertise-peer-urls=https://192.168.0.104:2380 \
 --advertise-client-urls=https://192.168.0.104:2379 \
 --initial-cluster=etcd01=https://192.168.0.102:2380,etcd02=https://192.168.0.103:2380,etcd03=https://192.168.0.104:2380 \
 --initial-cluster-token=k8s-etcd-cluster \
 --initial-cluster-state=new \
 --cert-file=/opt/etcd/certs/etcd-peer.pem \
 --key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --peer-cert-file=/opt/etcd/certs/etcd-peer.pem \
 --peer-key-file=/opt/etcd/certs/etcd-peer-key.pem \
 --trusted-ca-file=/opt/etcd/certs/ca.pem \
 --peer-trusted-ca-file=/opt/etcd/certs/ca.pem"


Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target 

 

(4) 启动etcd

 然后三个节点要同时执行训动命令才能成功启动etcd。

systemctl daemon-reload
systemctl enable etcd

#这条命令要在三个节点上同时执行，第一个执行的节点会最多等待30秒让其它两个节点加入集群。
systemctl start etcd  

 

启动如果没报错的话就启动成功了，查看集群状态

etcdctl member list

 

 

 

查看etcd监听端口

netstat -tunlp | grep etcd