etcd三節點安全集群搭建-pki安全認證

etcd安全集群搭建就是 pki安裝認證

1、環境：

三台centos7.5 主機

192.168.0.91

192.168.0.92

192.168.0.93

都關閉防火牆

都關閉selinux

配置免密登錄，參照：https://www.cnblogs.com/effortsing/p/10060748.html

都配置主機名

sed -i '$a\hostname=test1' /etc/sysconfig/network && hostnamectl set-hostname test1     

sed -i '$a\test1' /etc/hostname

cat >>/etc/hosts<< EOF
192.168.0.91 test1
192.168.0.92 test2
192.168.0.93 test3
192.168.0.94 test4
EOF


配置所有主機時間同步(非必須)

都重啟


2、 啟動etcd非安全集群

2.1、 安裝並啟動etcd

在3個節點上安裝etcd：

yum install -y etcd
systemctl start etcd && systemctl enable etcd


使用etcdctl訪問etcd並檢查其狀態驗證啟動成功。

etcdctl cluster-health
member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379



2.2、 修改配置啟動集群


目前這3個節點上的etcd並未形成集群，刪除原先配置文件，添加如下參數


etcd1配置

cat >/etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.91:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.0.91:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.91:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
EOF


etcd2配置

cat >/etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.92:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.0.92:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd2"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.92:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
EOF


etcd3配置

cat >/etc/etcd/etcd.conf <<EOF 
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.93:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.0.93:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd3"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.93:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
EOF


注意：ETCD_INITIAL_CLUSTER 選項決定了通過 etcdctl cluster-health 可以查看到節點的個數


集群的配置信息如節點url，token均存儲在數據目錄中，這些配置項僅在建立集群時生效。因此當修改已有etcd集群配置時（如新增節點，從http變為https通信等操作），

並不是簡單的修改配置文件就能完成，而是要通過etcdctl的集群管理工具通過復雜的步驟實現



刪除成員並啟動

systemctl stop etcd
rm -rf /var/lib/etcd/default.etcd
systemctl daemon-reload && systemctl restart etcd

如果不刪除成員目錄的話是無法啟動的，

注意三個節點要同時啟動才可以啟動成功



在任意一個節點上使用etcdctl驗證集群狀態:

etcdctl cluster-health

[root@etcd1 ~]# etcdctl cluster-health
member adff72f24ac33f4b is healthy: got healthy result from http://192.168.0.91:2379
member c883f9e325d8667d is healthy: got healthy result from http://192.168.0.93:2379
member c96f41ba37a00a16 is healthy: got healthy result from http://192.168.0.92:2379
cluster is healthy



3、集群之間通信介紹

集群服務中的通信一般包括兩種場景：

對外提供服務的通信，發生在集群外部的客戶端和集群某個節點之間，etcd默認端口為2379，例如 etcdctl 就屬於客戶端

集群內部的通信，發生在集群內部的任意兩個節點之間，etcd的默認端口為2380，

剛安裝完etcd可以看到配置文件里面都是http，這是不安全的，為了加強集群通信安全，需要使用https，下面就要介紹如何使用https來訪問集群



4、 創建RootCA

4.1、 安裝pki證書管理工具cfssl

安裝cfssl工具

只要把安裝包改下名字，移動到usr/local/bin/下，加上授權即可

通過網盤下載cfssl工具

鏈接：https://pan.baidu.com/s/1PGVlADPfCMhYEfYlMngDHQ 
提取碼：itrj 


鏈接：https://pan.baidu.com/s/1KsDKbbzwO82WegqPAlonyg 
提取碼：n8ce 


鏈接：https://pan.baidu.com/s/1dM8cJ38XAO_n6S-KKHZlqw 
提取碼：5n6m 


mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl*



4.2、配置PKI

證書分兩種情況：

服務器與客戶端之間的通信，這種情況下服務器的證書僅用於服務器認證，客戶端證書僅用於客戶端認證

服務器間的通信，這種情況下每個etcd既是服務器也是客戶端，因此其證書既要用於服務器認證，也要用於客戶端認證


創建PKI配置文件

mkdir /etc/etcd/pki

cd /etc/etcd/pki

cfssl print-defaults config > ca-config.json

vi ca-config.json

cat >ca-config.json <<EOF
{
    "signing": {
        "default": {
            "expiry": "168h"
        },
        "profiles": {
            "server": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF

在其中定義3個profile

server，作為服務器與客戶端通信時的服務器證書

client，作為服務器與客戶端通信時的客戶端證書

peer，作為服務器間通信時用的證書，既認證服務器也認證客戶端




4.3、 創建RootCA證書


cfssl print-defaults csr > rootca-csr.json
vi rootca-csr.json


修改后內容如下，由於CA證書不表示任何一台服務器，因此此處無需hosts字段


cat >rootca-csr.json<<EOF
{
    "CN": "ETCD Root CA",
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}
EOF

cfssl gencert -initca rootca-csr.json | cfssljson -bare rootca

ls rootca*
rootca.csr  rootca-csr.json  rootca-key.pem  rootca.pem


把根CA證書拷貝到集群的所有節點當中：

scp /etc/etcd/pki/rootca.pem root@192.168.0.92:/etc/etcd/pki/rootca.pem
scp /etc/etcd/pki/rootca.pem root@192.168.0.93:/etc/etcd/pki/rootca.pem


證書授權

chown -R etcd:etcd /etc/etcd/pki/*


根CA證書只有1個, 每個節點都保存，只保存證書即可。

服務器server證書1個，本實驗中為整個集群使用1個證書，每個服務器均保存該證書和私鑰。

客戶端證書1個, 本實驗環境中僅供etcdctl使用，因此在運行etcdctl的主機上保存證書和私鑰即可。實際工作中中每個訪問etcd的客戶端都應該有自己的客戶端證書和私鑰。

服務器peer證書3個, 每個節點保存自己的證書和私鑰




5、 集群外部開啟pki安全認證

注意：外部的意思在本篇就是使用 etcdtl來訪問，etcdctl 就是外部客戶端。如果k8s的apiserver訪問etcd，那么apiserver就是客戶端



5.1、 創建服務器證書


方式一、


集群成員用各自的證書

也就是說請求文件中hosts只寫本機ip地址

本文采用第一種方式

生產etcd1服務端證書

cfssl print-defaults csr > etcd1-csr.json
vi etcd1-csr.json 


cat > etcd1-csr.json<< EOF
{
    "CN": "ETCD Cluster-1",
    "hosts": [
        "192.168.0.91"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}
EOF

cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=server etcd1-csr.json | cfssljson -bare etcd1



生產etcd2服務端證書

cfssl print-defaults csr > etcd2-csr.json
vi etcd2-csr.json 


cat > etcd2-csr.json<< EOF
{
    "CN": "ETCD Cluster-2",
    "hosts": [
        "192.168.0.92"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}
EOF


cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=server etcd2-csr.json | cfssljson -bare etcd2


生產etcd3服務端證書

cfssl print-defaults csr > etcd3-csr.json
vi etcd3-csr.json 


cat > etcd3-csr.json<< EOF
{
    "CN": "ETCD Cluster-3",
    "hosts": [
        "192.168.0.93"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}
EOF

cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=server etcd3-csr.json | cfssljson -bare etcd3


復制證書

復制證書到對應節點

所有節點創建目錄

mkdir -p /etc/etcd/pki/

scp /etc/etcd/pki/etcd2*.pem root@192.168.0.92:/etc/etcd/pki/
scp /etc/etcd/pki/etcd3*.pem root@192.168.0.93:/etc/etcd/pki/



授權

給所有節點證書授權，否則啟動報錯

因為用root用戶生成的證書文件，證書權限為rw-------，etcd用戶沒有讀權限，而配置文件里面的ETCD_就代表etcd用戶，因此需要將其屬主修改為etcd。

chown -R etcd:etcd /etc/etcd/pki/*



方式二、

集群成員用統一的證書

也就是說請求文件中hosts填寫集群所有ip地址

注意 hosts也可以改成域名
 
所有使用證書的服務器都要寫到下面hosts列表里面，否則無法建立連接，以后添加新成員的話，hosts也要改

從上面可以看到hosts中有三個地址，如果以后要擴充集群節點，就需要修改hosts列表重新生成證書，重新分發到所有節點上，這樣容易出錯，也麻煩

生產環境一般把hosts寫成統一的對外域名。這里最好分開創建三個配置文件，每個配置文件里面填寫一個ip，不公用。以后擴容也方便。


cfssl print-defaults csr > etcd-csr.json
vi etcd-csr.json 


cat >etcd-csr.json<<EOF
{
    "CN": "ETCD Cluster",
    "hosts": [
        "192.168.0.91",
        "192.168.0.92",
        "192.168.0.93"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}
EOF


cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=server etcd-csr.json | cfssljson -bare etcd


所有節點創建目錄

mkdir -p /etc/etcd/pki/

scp /etc/etcd/pki/etcd*.pem root@192.168.0.92:/etc/etcd/pki/
scp /etc/etcd/pki/etcd*.pem root@192.168.0.93:/etc/etcd/pki/


給所有節點證書授權

因為用root用戶生成的證書文件，證書權限為rw-------，etcd用戶沒有讀權限，而配置文件里面的ETCD_就代表etcd用戶，因此需要將其屬主修改為etcd。

chown -R etcd:etcd /etc/etcd/pki/*





5.2、 修改etcd1配置並重啟

cat >/etc/etcd/etcd.conf << EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.91:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem"
EOF


重啟

systemctl daemon-reload && systemctl restart etcd


此時改變的僅僅時集群對外的服務方式，內部的通信方式並沒有改變，因此無需刪除實例，可直接重啟etcd。

重啟后，使用etcdctl指令訪問集群，如果在不指定–ca-file參數，結果會提示 https://192.168.0.91:2379 訪問失敗，因為其證書是不受信任的。


[root@test1 ~]# etcdctl cluster-health
failed to check the health of member 6c70a880257288f on https://192.168.0.91:2379: Get https://192.168.0.91:2379/health: x509: certificate signed by unknown authority
member 6c70a880257288f is unreachable: [https://192.168.0.91:2379] are all unreachable
member 3f7336e156287ed0 is healthy: got healthy result from http://192.168.0.93:2379
member 5bbe42788a239cc6 is healthy: got healthy result from http://192.168.0.92:2379
cluster is healthy


注意：ETCD_LISTEN_CLIENT_URLS中包含了http://127.0.0.1:2379, 因此直接指定該地址可以訪問etcd，但是ETCD_ADVERTISE_CLIENT_URLS中不包含http://127.0.0.1:2379, 因此etcd在給客戶端廣播集群節點的地址時，只會廣播https://192.168.56.41:2379, etcdctl緊接着用這個地址去查詢集群健康狀態時，但證書不受信任無法訪問。

加上–ca-file參數指定用於校驗的CA證書，即根CA證書后，訪問正常。

[root@test1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem cluster-health
member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379
member 3f7336e156287ed0 is healthy: got healthy result from http://192.168.0.93:2379
member 5bbe42788a239cc6 is healthy: got healthy result from http://192.168.0.92:2379
cluster is healthy


上面輸出可以看到，僅有1個節點啟動了https。對其余兩個節點重復本節操作即可。出於對rootca的安全考慮，服務器證書的生成操作在一台服務器上完成，生成后將其拷貝到相應節點即可。配置並重啟完所有節點后，應該可以看到所有節點的偵聽URL均為https協議。


5.3、 修改etcd2配置並重啟

cat >/etc/etcd/etcd.conf << EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd2"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem"
EOF


重啟

systemctl daemon-reload && systemctl restart etcd



5.4、 修改etcd3配置並重啟

cat >/etc/etcd/etcd.conf << EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd3"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd3.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd3-key.pem"
EOF


重啟

systemctl daemon-reload && systemctl restart etcd


查看健康狀態

[root@test1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem cluster-health
member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379
member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379
member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379
cluster is healthy


發現都變成了https模式



6、 客戶端驗證

6.1.1、 修改etcd1配置並重啟


啟動客戶端認證需要修改以下參數：

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"


cat > /etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.91:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF


重啟etcd1

systemctl daemon-reload && systemctl restart etcd


重啟etcd服務后發現即使指定了–ca-file參數，https節點仍然無法訪問。這次的錯誤是證書錯誤，因為客戶端沒有提供任何證書。

[root@test1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem cluster-health
failed to check the health of member 6c70a880257288f on https://192.168.0.91:2379: Get https://192.168.0.91:2379/health: remote error: tls: bad certificate
member 6c70a880257288f is unreachable: [https://192.168.0.91:2379] are all unreachable
member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379
member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379
cluster is healthy



6.1.2、 創建客戶端證書

修改后內容如下，etcdctl可能運行在多台節點上，因此不指定可以使用該證書的主機列表。

創建客戶端證書請求文件所需配置:

cfssl print-defaults csr > etcdctl-csr.json
vi etcdctl-csr.json

cat >etcdctl-csr.json<<EOF
{
    "CN": "ETCDCTL",
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}
EOF


cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=client etcdctl-csr.json | cfssljson -bare etcdctl



授權

chown -R etcd:etcd /etc/etcd/pki/*


復制證書

scp /etc/etcd/pki/etcdctl*.pem root@192.168.0.92:/etc/etcd/pki/
scp /etc/etcd/pki/etcdctl*.pem root@192.168.0.93:/etc/etcd/pki/


授權

復制過去要給對方節點授權

chown -R etcd:etcd /etc/etcd/pki/*


然后在etcdctl命令行中指定生成的證書和私鑰，才能成功訪問節點:

[root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health
member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379
member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379
member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379
cluster is healthy



6.2.1、 修改etcd2配置並重啟


啟動客戶端認證需要修改以下參數：

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"


cat > /etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd2"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF


重啟etcd2

systemctl daemon-reload && systemctl restart etcd


然后在etcdctl命令行中指定生成的客戶端證書和私鑰，訪問節點:

[root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health
member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379
member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379
member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379
cluster is healthy



6.3.1、 修改etcd3配置並重啟


啟動客戶端認證需要修改以下參數：

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"


cat > /etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd3"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd3.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd3-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF


重啟etcd3

systemctl daemon-reload && systemctl restart etcd


然后在etcdctl命令行中指定生成的客戶端證書和私鑰，訪問節點:

[root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health
member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379
member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379
member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379
cluster is healthy



7、集群內部開啟pki安全認證


方式一： 不重建集群開啟pki安全認證


7.1、先修改etcd3節點為安全通信


7.1.1、准備peer證書

注意：peer證書既是服務端證書又是客戶端證書，從下面參數 -profile=peer中可以看到

和server證書一樣，3個節點的peer證書其實也可以共用一個，考慮到以后擴容代理的麻煩，所以這里每個節點都配置自己的peer證書3個節點分別創建peer證書請求文件


生產peer1證書

cfssl print-defaults csr > etcd1-peer-csr.json
vi etcd1-peer-csr.json

cat >etcd1-peer-csr.json <<EOF
{
    "CN": "ETCD Peer on etcd1",
    "hosts": [
        "192.168.0.91"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}
EOF

cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=peer etcd1-peer-csr.json | cfssljson -bare etcd1-peer


生產peer2證書

cfssl print-defaults csr > etcd2-peer-csr.json
vi etcd2-peer-csr.json

cat >etcd2-peer-csr.json <<EOF
{
    "CN": "ETCD Peer on etcd2",
    "hosts": [
        "192.168.0.92"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}
EOF

cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=peer etcd2-peer-csr.json | cfssljson -bare etcd2-peer


生產peer3證書

cfssl print-defaults csr > etcd3-peer-csr.json
vi etcd3-peer-csr.json

cat >etcd3-peer-csr.json <<EOF
{
    "CN": "ETCD Peer on etcd3",
    "hosts": [
        "192.168.0.93"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
}
EOF

cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=peer etcd3-peer-csr.json | cfssljson -bare etcd3-peer


注意：peer證書既是服務端證書又是客戶端證書，從上面參數 -profile=peer中可以看到



7.1.2、復制證書

scp /etc/etcd/pki/etcd2-peer*.pem root@192.168.0.92:/etc/etcd/pki/
scp /etc/etcd/pki/etcd3-peer*.pem root@192.168.0.93:/etc/etcd/pki/


7.1.3、授權

所有節點授權，復制過去要記得給授權，否則啟動報錯

chown -R etcd:etcd /etc/etcd/pki/*



7.1.4、查看節點列表，獲取節點標識

[root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list
adff72f24ac33f4b: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true
c883f9e325d8667d: name=etcd3 peerURLs=http://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false
c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false


7.1.5、修改etcd3節點的peer url為https

[root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update c883f9e325d8667d https://192.168.0.93:2380
Updated member with ID c883f9e325d8667d in cluster


7.1.6、重新檢查節點列表和集群健康狀態

[root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list
adff72f24ac33f4b: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true
c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false
c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false

[root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health
member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379
member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379
member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379
cluster is healthy


可以看到etcd3的peer地址已經是https了，但實際上此時etcd3的偵聽地址沒有修改，https所需要的相關證書都沒有配置，https通信是不可能建立的，因此事實上此時與etcd3的通信仍然是通過http。

注意：如果發現peerURLs不是https，原因在於執行"修改etcd3節點的peer url為https步驟"的時候掉了步驟最后面的https://192.168.0.93:2380 或者ID不正確，重新執行幾遍即可


7.1.7、修改etcd3的peer工作端口為https


修改內容如下:


ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"

ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem"      
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"


cat >/etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd3"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd3.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd3-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"

ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem"      
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF


重啟

systemctl daemon-reload && systemctl restart etcd


查看集群狀態

[root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list
6c70a880257288f: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true
3f7336e156287ed0: name=etcd3 peerURLs=http://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false
5bbe42788a239cc6: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false

上述配置在etcd3啟動了服務器端的https通信，並且要求進行客戶端驗證，而作為客戶端的etcd1和etcd2還沒有相關配置，因此https通信仍然會失敗，與etcd3的通信仍然fallback到http上

因此需要修改etcd1和etcd2進行客戶端驗證


7.1.8、 在etcd1和etcd2上配置客戶端所需證書

涉及的參數主要是客戶端自身的證書和私鑰，以及用於驗證etcd3的根CA證書：

etcd1

ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"


執行：

cat > /etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.91:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"

#開啟集群內部服務端認證同時帶上客戶端證書
ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF

systemctl daemon-reload && systemctl restart etcd


etcd2

ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd2-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd2-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"


執行：

cat > /etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="http://192.168.0.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd2"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"

#開啟集群內部服務端認證同時帶上客戶端證書
ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd2-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd2-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF

systemctl daemon-reload && systemctl restart etcd


查看集群狀態

[root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list
6c70a880257288f: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true
3f7336e156287ed0: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false
5bbe42788a239cc6: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false

發現etcd3上的報錯隨即停


注意：如果先在節點上修改配置文件啟用https URL，再使用etcdctl指令修改集群的peer訪問端點，在兩步之間的時間里，實際上是客戶端使用http協議訪問服務器的https服務，

這段時間實際集群間的通信是失敗的。可在服務器上看到https請求被拒絕的錯誤：

[root@etcd3 ~]# systemctl status etcd -l

Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.92:43682"

Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.91:47588"




7.2、修改etcd2節點為安全通信


查看節點列表，獲取節點標識

[root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list
adff72f24ac33f4b: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true
c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false
c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false


修改etcd2節點的peer url為https

etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update adff72f24ac33f4b https://192.168.0.91:2380

執行結果：

[root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update 5bbe42788a239cc6 https://192.168.0.91:2380
Updated member with ID 5bbe42788a239cc6 in cluster


重新檢查節點列表和集群健康狀態

[root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list
adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false
c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true
c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false

[root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health
member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379
member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379
member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379
cluster is healthy

發現etcd2節點的peerURLs改成了https

注意：如果發現peerURLs不是https，原因在於執行"修改etcd3節點的peer url為https步驟"的時候掉了步驟最后面的https://192.168.0.93:2380 或者ID不正確，重新執行幾遍即可




修改etcd2的peer工作端口為https

ETCD_LISTEN_PEER_URLS="https://192.168.0.91:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.91:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"

ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem"      
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"


執行：

cat > /etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="https://192.168.0.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd2"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"

ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd2-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd2-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF


重啟

systemctl daemon-reload && systemctl restart etcd



7.2、修改etcd1節點為安全通信


查看節點列表，獲取節點標識

[root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list
adff72f24ac33f4b: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true
c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false
c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false


修改etcd1節點的peer url為https

etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update c96f41ba37a00a16 https://192.168.0.91:2380

執行結果：

[root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update adff72f24ac33f4b https://192.168.0.91:2380
membership: peerURL exists



重新檢查節點列表和集群健康狀態

[root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list
adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false
c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true
c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false

[root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health
member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379
member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379
member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379
cluster is healthy

發現etcd1節點 peerURLs變為https

注意：如果發現peerURLs不是https，原因在於執行"修改etcd3節點的peer url為https步驟"的時候掉了步驟最后面的https://192.168.0.93:2380 或者ID不正確，重新執行幾遍即可



修改etcd1的peer工作端口為https

ETCD_LISTEN_PEER_URLS="https://192.168.0.92:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.92:2380"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"

ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem"      
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"


執行：

cat > /etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="https://192.168.0.91:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"

ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF


重啟

systemctl daemon-reload && systemctl restart etcd


重新檢查節點列表和集群健康狀態

[root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list
adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false
c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true
c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false

[root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health
member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379
member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379
member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379
cluster is healthy


可以看到peerURLs改變為https模式


如果先在節點上修改配置文件啟用https URL，再使用etcdctl指令修改集群的peer訪問端點，會報如下錯誤,所以最好是先使用etcdct指令修改訪問端點，再修改服務器配置文件啟用https。

[root@etcd3 ~]# systemctl status etcd -l
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-01-26 01:43:20 EST; 4min 52s ago
 Main PID: 2525 (etcd)
   CGroup: /system.slice/etcd.service
           └─2525 /usr/bin/etcd --name=etcd3 --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=https://192.168.0.93:2379,http://127.0.0.1:2379

Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.92:43682" (error "remote error: tls: bad certificate", ServerName "")
Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.91:47588" (error "remote error: tls: bad certificate", ServerName "")
Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.92:43684" (error "remote error: tls: bad certificate", ServerName "")
Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.91:47590" (error "remote error: tls: bad certificate", ServerName "")




7.3、所有文件改成https並重啟

etcd1節點etcd配置文件

cat > /etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="https://192.168.0.91:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"

#開啟集群內部服務端認證並帶上客戶端證書
ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF

重啟

systemctl daemon-reload && systemctl restart etcd



etcd2節點etcd配置文件

cat >/etc/etcd/etcd.conf << EOF 
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="https://192.168.0.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd2"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"

#開啟集群內部服務端認證並帶上客戶端證書
ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd2-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd2-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF


重啟

systemctl daemon-reload && systemctl restart etcd


etcd3節點etcd配置文件

cat >/etc/etcd/etcd.conf << EOF
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd3"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#開啟集群外部服務端認證
ETCD_CERT_FILE="/etc/etcd/pki/etcd3.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd3-key.pem"

#開啟集群外部客戶端認證
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"

#開啟集群內部服務端認證並帶上客戶端證書
ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem"      
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"
EOF


重啟

systemctl daemon-reload && systemctl restart etcd



報錯解決：

[root@etcd1 ~]# systemctl status etcd -l
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-01-26 02:35:51 EST; 4min 18s ago
 Main PID: 3117 (etcd)
   CGroup: /system.slice/etcd.service
           └─3117 /usr/bin/etcd --name=etcd1 --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=https://192.168.0.91:2379,http://127.0.0.1:2379

Jan 26 02:35:51 etcd1 etcd[3117]: established a TCP streaming connection with peer c96f41ba37a00a16 (stream Message writer)
Jan 26 02:35:51 etcd1 etcd[3117]: established a TCP streaming connection with peer c883f9e325d8667d (stream MsgApp v2 writer)
Jan 26 02:35:51 etcd1 bash[3117]: WARNING: 2019/01/26 02:35:51 Failed to dial 192.168.0.91:2379: connection error: desc = "transport: 


查看錯誤： WARNING: 2019/01/26 02:35:51 Failed to dial 192.168.0.91:2379: connection error:


原因：

ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,k8s=https://192.168.0.92:2380,k8=https://192.168.0.93:2380"

糾正：

ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,k83=https://192.168.0.93:2380"

重啟

systemctl daemon-reload && systemctl restart etcd





方式二：重建集群啟用https


注意：這種方式會丟失所有數據，一般在新建集群時使用。一般不使用這種方式

集群節點的peer訪問端點存儲在數據目錄，因此修改ETCD_INITIAL_CLUSTER參數后，最簡單讓其生效的方法就是重建集群。


在所有節點上修改etcd配置文件，將peer的url修改為https，配置相關證書，以etcd3為例，涉及參數如下：

ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380"
ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"


[root@etcd3 ~]# cat /etc/etcd/etcd.conf 
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"      
ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379"
ETCD_NAME="etcd3"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379"
ETCD_INITIAL_CLUSTER="etcd4=https://192.168.0.94:2380,etcd1=https://192.168.0.91:2380,etcd3=https://192.168.0.93:2380,etcd2=https://192.168.0.92:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/pki/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"

ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem"      
ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem"



在所有節點上刪除已有實例，重啟etcd。

systemctl stop etcd
rm -rf /var/lib/etcd/default.etcd
systemctl daemon-reload && systemctl restart etcd


參照文檔：

https://www.jianshu.com/p/3015d514bae3
https://lprincewhn.github.io/2018/09/15/etcd-ha-pki-01.html
http://www.mamicode.com/info-detail-1737556.html
http://www.cnblogs.com/breg/p/5728237.html