規划：先安裝三台組建集群，然后擴容一個安全節點進來 1、環境： 三台centos7.5 主機 192.168.0.91 192.168.0.92 192.168.0.93 都關閉防火牆 都關閉selinux 配置免密登錄，參照：https://www.cnblogs.com/effortsing/p/10060748.html 都配置主機名 sed -i '$a\hostname=test1' /etc/sysconfig/network && hostnamectl set-hostname test1 sed -i '$a\test1' /etc/hostname cat >>/etc/hosts<< EOF 192.168.0.91 test1 192.168.0.92 test2 192.168.0.93 test3 192.168.0.94 test4 EOF 配置所有主機時間同步(非必須) 都退出xshell重新登錄，查看主機名 2、 啟動etcd非安全集群 2.1、 安裝並啟動etcd 在3個節點上安裝etcd： yum install -y etcd systemctl start etcd && systemctl enable etcd 使用etcdctl訪問etcd並檢查其狀態驗證啟動成功。 etcdctl cluster-health member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379 2.2、 修改配置啟動集群 目前這3個節點上的etcd並未形成集群，刪除原先配置文件，添加如下參數 etcd1配置 cat >/etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.91:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.0.91:2379,http://127.0.0.1:2379" ETCD_NAME="etcd1" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.91:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.91:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" EOF etcd2配置 cat >/etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.92:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.0.92:2379,http://127.0.0.1:2379" ETCD_NAME="etcd2" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.92:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.92:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" EOF etcd3配置 cat >/etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.93:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.0.93:2379,http://127.0.0.1:2379" ETCD_NAME="etcd3" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.93:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.93:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" EOF 注意：ETCD_INITIAL_CLUSTER 選項決定了通過 etcdctl cluster-health 可以查看到節點的個數 集群的配置信息如節點url，token均存儲在數據目錄中，這些配置項僅在建立集群時生效。因此當修改已有etcd集群配置時（如新增節點，從http變為https通信等操作）， 並不是簡單的修改配置文件就能完成，而是要通過etcdctl的集群管理工具通過復雜的步驟實現 刪除成員並啟動 systemctl stop etcd rm -rf /var/lib/etcd/default.etcd systemctl daemon-reload && systemctl restart etcd 如果不刪除成員目錄的話是無法啟動的， 注意三個節點要同時啟動才可以啟動成功 在任意一個節點上使用etcdctl驗證集群狀態: etcdctl cluster-health [root@etcd1 ~]# etcdctl cluster-health member adff72f24ac33f4b is healthy: got healthy result from http://192.168.0.91:2379 member c883f9e325d8667d is healthy: got healthy result from http://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from http://192.168.0.92:2379 cluster is healthy 3、集群之間通信介紹 集群服務中的通信一般包括兩種場景： 對外提供服務的通信，發生在集群外部的客戶端和集群某個節點之間，etcd默認端口為2379，例如 etcdctl 就屬於客戶端 集群內部的通信，發生在集群內部的任意兩個節點之間，etcd的默認端口為2380， 剛安裝完etcd可以看到配置文件里面都是http，這是不安全的，為了加強集群通信安全，需要使用https，下面就要介紹如何使用https來訪問集群 4、 創建RootCA 4.1、 安裝pki證書管理工具cfssl 安裝cfssl工具 只要把安裝包改下名字，移動到usr/local/bin/下，加上授權即可 通過網盤下載cfssl工具 鏈接：https://pan.baidu.com/s/1PGVlADPfCMhYEfYlMngDHQ 提取碼：itrj 鏈接：https://pan.baidu.com/s/1KsDKbbzwO82WegqPAlonyg 提取碼：n8ce 鏈接：https://pan.baidu.com/s/1dM8cJ38XAO_n6S-KKHZlqw 提取碼：5n6m mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson chmod +x /usr/local/bin/cfssl* 4.2、配置PKI 證書分兩種情況： 服務器與客戶端之間的通信，這種情況下服務器的證書僅用於服務器認證，客戶端證書僅用於客戶端認證 服務器間的通信，這種情況下每個etcd既是服務器也是客戶端，因此其證書既要用於服務器認證，也要用於客戶端認證 創建PKI配置文件 mkdir /etc/etcd/pki cd /etc/etcd/pki cfssl print-defaults config > ca-config.json vi ca-config.json cat >ca-config.json <<EOF { "signing": { "default": { "expiry": "168h" }, "profiles": { "server": { "expiry": "8760h", "usages": [ "signing", "key encipherment", "server auth" ] }, "client": { "expiry": "8760h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "8760h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF 在其中定義3個profile server，作為服務器與客戶端通信時的服務器證書 client，作為服務器與客戶端通信時的客戶端證書 peer，作為服務器間通信時用的證書，既認證服務器也認證客戶端 4.3、 創建RootCA證書 cfssl print-defaults csr > rootca-csr.json vi rootca-csr.json 修改后內容如下，由於CA證書不表示任何一台服務器，因此此處無需hosts字段 cat >rootca-csr.json<<EOF { "CN": "ETCD Root CA", "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -initca rootca-csr.json | cfssljson -bare rootca ls rootca* rootca.csr rootca-csr.json rootca-key.pem rootca.pem 把根CA證書拷貝到集群的所有節點當中： scp /etc/etcd/pki/rootca.pem root@192.168.0.92:/etc/etcd/pki/rootca.pem scp /etc/etcd/pki/rootca.pem root@192.168.0.93:/etc/etcd/pki/rootca.pem 證書授權 chown -R etcd:etcd /etc/etcd/pki/* 根CA證書只有1個, 每個節點都保存，只保存證書即可。 服務器server證書1個，本實驗中為整個集群使用1個證書，每個服務器均保存該證書和私鑰。 客戶端證書1個, 本實驗環境中僅供etcdctl使用，因此在運行etcdctl的主機上保存證書和私鑰即可。實際工作中中每個訪問etcd的客戶端都應該有自己的客戶端證書和私鑰。 服務器peer證書3個, 每個節點保存自己的證書和私鑰 5、 集群外部開啟pki安全認證 注意：外部的意思在本篇就是使用 etcdtl來訪問，etcdctl 就是外部客戶端。如果k8s的apiserver訪問etcd，那么apiserver就是客戶端 5.1、 創建服務器證書 方式一、 集群成員用各自的證書 也就是說請求文件中hosts只寫本機ip地址 本文采用第一種方式 生產etcd1服務端證書 cfssl print-defaults csr > etcd1-csr.json vi etcd1-csr.json cat > etcd1-csr.json<< EOF { "CN": "ETCD Cluster-1", "hosts": [ "192.168.0.91" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=server etcd1-csr.json | cfssljson -bare etcd1 生產etcd2服務端證書 cfssl print-defaults csr > etcd2-csr.json vi etcd2-csr.json cat > etcd2-csr.json<< EOF { "CN": "ETCD Cluster-2", "hosts": [ "192.168.0.92" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=server etcd2-csr.json | cfssljson -bare etcd2 生產etcd3服務端證書 cfssl print-defaults csr > etcd3-csr.json vi etcd3-csr.json cat > etcd3-csr.json<< EOF { "CN": "ETCD Cluster-3", "hosts": [ "192.168.0.93" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=server etcd3-csr.json | cfssljson -bare etcd3 復制證書 復制證書到對應節點 所有節點創建目錄 mkdir -p /etc/etcd/pki/ scp /etc/etcd/pki/etcd2*.pem root@192.168.0.92:/etc/etcd/pki/ scp /etc/etcd/pki/etcd3*.pem root@192.168.0.93:/etc/etcd/pki/ 授權 給所有節點證書授權，否則啟動報錯 因為用root用戶生成的證書文件，證書權限為rw-------，etcd用戶沒有讀權限，而配置文件里面的ETCD_就代表etcd用戶，因此需要將其屬主修改為etcd。 chown -R etcd:etcd /etc/etcd/pki/* 方式二、 集群成員用統一的證書 也就是說請求文件中hosts填寫集群所有ip地址 注意 hosts也可以改成域名 所有使用證書的服務器都要寫到下面hosts列表里面，否則無法建立連接，以后添加新成員的話，hosts也要改 從上面可以看到hosts中有三個地址，如果以后要擴充集群節點，就需要修改hosts列表重新生成證書，重新分發到所有節點上，這樣容易出錯，也麻煩 生產環境一般把hosts寫成統一的對外域名。這里最好分開創建三個配置文件，每個配置文件里面填寫一個ip，不公用。以后擴容也方便。 cfssl print-defaults csr > etcd-csr.json vi etcd-csr.json cat >etcd-csr.json<<EOF { "CN": "ETCD Cluster", "hosts": [ "192.168.0.91", "192.168.0.92", "192.168.0.93" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=server etcd-csr.json | cfssljson -bare etcd 所有節點創建目錄 mkdir -p /etc/etcd/pki/ scp /etc/etcd/pki/etcd*.pem root@192.168.0.92:/etc/etcd/pki/ scp /etc/etcd/pki/etcd*.pem root@192.168.0.93:/etc/etcd/pki/ 給所有節點證書授權 因為用root用戶生成的證書文件，證書權限為rw-------，etcd用戶沒有讀權限，而配置文件里面的ETCD_就代表etcd用戶，因此需要將其屬主修改為etcd。 chown -R etcd:etcd /etc/etcd/pki/* 5.2、 修改etcd1配置並重啟 cat >/etc/etcd/etcd.conf << EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.91:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379" ETCD_NAME="etcd1" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.91:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 此時改變的僅僅時集群對外的服務方式，內部的通信方式並沒有改變，因此無需刪除實例，可直接重啟etcd。 重啟后，使用etcdctl指令訪問集群，如果在不指定–ca-file參數，結果會提示 https://192.168.0.91:2379 訪問失敗，因為其證書是不受信任的。 [root@test1 ~]# etcdctl cluster-health failed to check the health of member 6c70a880257288f on https://192.168.0.91:2379: Get https://192.168.0.91:2379/health: x509: certificate signed by unknown authority member 6c70a880257288f is unreachable: [https://192.168.0.91:2379] are all unreachable member 3f7336e156287ed0 is healthy: got healthy result from http://192.168.0.93:2379 member 5bbe42788a239cc6 is healthy: got healthy result from http://192.168.0.92:2379 cluster is healthy 注意：ETCD_LISTEN_CLIENT_URLS中包含了http://127.0.0.1:2379, 因此直接指定該地址可以訪問etcd，但是ETCD_ADVERTISE_CLIENT_URLS中不包含http://127.0.0.1:2379, 因此etcd在給客戶端廣播集群節點的地址時，只會廣播https://192.168.56.41:2379, etcdctl緊接着用這個地址去查詢集群健康狀態時，但證書不受信任無法訪問。 加上–ca-file參數指定用於校驗的CA證書，即根CA證書后，訪問正常。 [root@test1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem cluster-health member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379 member 3f7336e156287ed0 is healthy: got healthy result from http://192.168.0.93:2379 member 5bbe42788a239cc6 is healthy: got healthy result from http://192.168.0.92:2379 cluster is healthy 上面輸出可以看到，僅有1個節點啟動了https。對其余兩個節點重復本節操作即可。出於對rootca的安全考慮，服務器證書的生成操作在一台服務器上完成，生成后將其拷貝到相應節點即可。配置並重啟完所有節點后，應該可以看到所有節點的偵聽URL均為https協議。 5.3、 修改etcd2配置並重啟 cat >/etc/etcd/etcd.conf << EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.92:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379" ETCD_NAME="etcd2" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.92:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 5.4、 修改etcd3配置並重啟 cat >/etc/etcd/etcd.conf << EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.93:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379" ETCD_NAME="etcd3" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.93:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd3.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd3-key.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 查看健康狀態 [root@test1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem cluster-health member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379 member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379 member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 發現都變成了https模式 6、 客戶端驗證 6.1.1、 修改etcd1配置並重啟 啟動客戶端認證需要修改以下參數： ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" cat > /etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.91:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379" ETCD_NAME="etcd1" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.91:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟etcd1 systemctl daemon-reload && systemctl restart etcd 重啟etcd服務后發現即使指定了–ca-file參數，https節點仍然無法訪問。這次的錯誤是證書錯誤，因為客戶端沒有提供任何證書。 [root@test1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem cluster-health failed to check the health of member 6c70a880257288f on https://192.168.0.91:2379: Get https://192.168.0.91:2379/health: remote error: tls: bad certificate member 6c70a880257288f is unreachable: [https://192.168.0.91:2379] are all unreachable member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379 member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 6.1.2、 創建客戶端證書 修改后內容如下，etcdctl可能運行在多台節點上，因此不指定可以使用該證書的主機列表。 創建客戶端證書請求文件所需配置: cfssl print-defaults csr > etcdctl-csr.json vi etcdctl-csr.json cat >etcdctl-csr.json<<EOF { "CN": "ETCDCTL", "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=client etcdctl-csr.json | cfssljson -bare etcdctl 授權 chown -R etcd:etcd /etc/etcd/pki/* 復制證書 scp /etc/etcd/pki/etcdctl*.pem root@192.168.0.92:/etc/etcd/pki/ scp /etc/etcd/pki/etcdctl*.pem root@192.168.0.93:/etc/etcd/pki/ 授權 復制過去要給對方節點授權 chown -R etcd:etcd /etc/etcd/pki/* 然后在etcdctl命令行中指定生成的證書和私鑰，才能成功訪問節點: [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379 member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379 member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 6.2.1、 修改etcd2配置並重啟 啟動客戶端認證需要修改以下參數： ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" cat > /etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.92:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379" ETCD_NAME="etcd2" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.92:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟etcd2 systemctl daemon-reload && systemctl restart etcd 然后在etcdctl命令行中指定生成的客戶端證書和私鑰，訪問節點: [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379 member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379 member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 6.3.1、 修改etcd3配置並重啟 啟動客戶端認證需要修改以下參數： ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" cat > /etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.93:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379" ETCD_NAME="etcd3" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.93:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=http://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd3.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd3-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟etcd3 systemctl daemon-reload && systemctl restart etcd 然后在etcdctl命令行中指定生成的客戶端證書和私鑰，訪問節點: [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member 6c70a880257288f is healthy: got healthy result from https://192.168.0.91:2379 member 3f7336e156287ed0 is healthy: got healthy result from https://192.168.0.93:2379 member 5bbe42788a239cc6 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 7、集群內部開啟pki安全認證 方式一： 不重建集群開啟pki安全認證 7.1、先修改etcd3節點為安全通信 7.1.1、准備peer證書 注意：peer證書既是服務端證書又是客戶端證書，從下面參數 -profile=peer中可以看到 和server證書一樣，3個節點的peer證書其實也可以共用一個，考慮到以后擴容代理的麻煩，所以這里每個節點都配置自己的peer證書3個節點分別創建peer證書請求文件 生產peer1證書 cfssl print-defaults csr > etcd1-peer-csr.json vi etcd1-peer-csr.json cat >etcd1-peer-csr.json <<EOF { "CN": "ETCD Peer on etcd1", "hosts": [ "192.168.0.91" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=peer etcd1-peer-csr.json | cfssljson -bare etcd1-peer 生產peer2證書 cfssl print-defaults csr > etcd2-peer-csr.json vi etcd2-peer-csr.json cat >etcd2-peer-csr.json <<EOF { "CN": "ETCD Peer on etcd2", "hosts": [ "192.168.0.92" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=peer etcd2-peer-csr.json | cfssljson -bare etcd2-peer 生產peer3證書 cfssl print-defaults csr > etcd3-peer-csr.json vi etcd3-peer-csr.json cat >etcd3-peer-csr.json <<EOF { "CN": "ETCD Peer on etcd3", "hosts": [ "192.168.0.93" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=peer etcd3-peer-csr.json | cfssljson -bare etcd3-peer 注意：peer證書既是服務端證書又是客戶端證書，從上面參數 -profile=peer中可以看到 7.1.2、復制證書 scp /etc/etcd/pki/etcd2-peer*.pem root@192.168.0.92:/etc/etcd/pki/ scp /etc/etcd/pki/etcd3-peer*.pem root@192.168.0.93:/etc/etcd/pki/ 7.1.3、授權 所有節點授權，復制過去要記得給授權，否則啟動報錯 chown -R etcd:etcd /etc/etcd/pki/* 7.1.4、查看節點列表，獲取節點標識 [root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true c883f9e325d8667d: name=etcd3 peerURLs=http://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false 7.1.5、修改etcd3節點的peer url為https [root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update c883f9e325d8667d https://192.168.0.93:2380 Updated member with ID c883f9e325d8667d in cluster 7.1.6、重新檢查節點列表和集群健康狀態 [root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false [root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379 member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 可以看到etcd3的peer地址已經是https了，但實際上此時etcd3的偵聽地址沒有修改，https所需要的相關證書都沒有配置，https通信是不可能建立的，因此事實上此時與etcd3的通信仍然是通過http。 注意：如果發現peerURLs不是https，原因在於執行"修改etcd3節點的peer url為https步驟"的時候掉了步驟最后面的https://192.168.0.93:2380 或者ID不正確，重新執行幾遍即可 7.1.7、修改etcd3的peer工作端口為https 修改內容如下: ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" cat >/etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379" ETCD_NAME="etcd3" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd3.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd3-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 查看集群狀態 [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 6c70a880257288f: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true 3f7336e156287ed0: name=etcd3 peerURLs=http://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false 5bbe42788a239cc6: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false 上述配置在etcd3啟動了服務器端的https通信，並且要求進行客戶端驗證，而作為客戶端的etcd1和etcd2還沒有相關配置，因此https通信仍然會失敗，與etcd3的通信仍然fallback到http上 因此需要修改etcd1和etcd2進行客戶端驗證 7.1.8、 在etcd1和etcd2上配置客戶端所需證書 涉及的參數主要是客戶端自身的證書和私鑰，以及用於驗證etcd3的根CA證書： etcd1 ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" 執行： cat > /etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.91:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379" ETCD_NAME="etcd1" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.91:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" #開啟集群內部服務端認證同時帶上客戶端證書 ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF systemctl daemon-reload && systemctl restart etcd etcd2 ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd2-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd2-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" 執行： cat > /etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.92:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379" ETCD_NAME="etcd2" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.92:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=http://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" #開啟集群內部服務端認證同時帶上客戶端證書 ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd2-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd2-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF systemctl daemon-reload && systemctl restart etcd 查看集群狀態 [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 6c70a880257288f: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true 3f7336e156287ed0: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false 5bbe42788a239cc6: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false 發現etcd3上的報錯隨即停 注意：如果先在節點上修改配置文件啟用https URL，再使用etcdctl指令修改集群的peer訪問端點，在兩步之間的時間里，實際上是客戶端使用http協議訪問服務器的https服務， 這段時間實際集群間的通信是失敗的。可在服務器上看到https請求被拒絕的錯誤： [root@etcd3 ~]# systemctl status etcd -l Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.92:43682" Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.91:47588" 7.2、修改etcd2節點為安全通信 查看節點列表，獲取節點標識 [root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false 修改etcd2節點的peer url為https etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update adff72f24ac33f4b https://192.168.0.91:2380 執行結果： [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update 5bbe42788a239cc6 https://192.168.0.91:2380 Updated member with ID 5bbe42788a239cc6 in cluster 重新檢查節點列表和集群健康狀態 [root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false [root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379 member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 發現etcd2節點的peerURLs改成了https 注意：如果發現peerURLs不是https，原因在於執行"修改etcd3節點的peer url為https步驟"的時候掉了步驟最后面的https://192.168.0.93:2380 或者ID不正確，重新執行幾遍即可 修改etcd2的peer工作端口為https ETCD_LISTEN_PEER_URLS="https://192.168.0.91:2380" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.91:2380" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" 執行： cat > /etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.92:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379" ETCD_NAME="etcd2" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.92:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379" ETCD_INITIAL_CLUSTER="etcd1=http://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd2-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd2-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 7.2、修改etcd1節點為安全通信 查看節點列表，獲取節點標識 [root@etcd1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=http://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false 修改etcd1節點的peer url為https etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update c96f41ba37a00a16 https://192.168.0.91:2380 執行結果： [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update adff72f24ac33f4b https://192.168.0.91:2380 membership: peerURL exists 重新檢查節點列表和集群健康狀態 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379 member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 發現etcd1節點 peerURLs變為https 注意：如果發現peerURLs不是https，原因在於執行"修改etcd3節點的peer url為https步驟"的時候掉了步驟最后面的https://192.168.0.93:2380 或者ID不正確，重新執行幾遍即可 修改etcd1的peer工作端口為https ETCD_LISTEN_PEER_URLS="https://192.168.0.92:2380" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.92:2380" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" 執行： cat > /etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.91:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379" ETCD_NAME="etcd1" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.91:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 重新檢查節點列表和集群健康狀態 [root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false [root@etcd3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379 member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 可以看到peerURLs改變為https模式 如果先在節點上修改配置文件啟用https URL，再使用etcdctl指令修改集群的peer訪問端點，會報如下錯誤,所以最好是先使用etcdct指令修改訪問端點，再修改服務器配置文件啟用https。 [root@etcd3 ~]# systemctl status etcd -l ● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-01-26 01:43:20 EST; 4min 52s ago Main PID: 2525 (etcd) CGroup: /system.slice/etcd.service └─2525 /usr/bin/etcd --name=etcd3 --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=https://192.168.0.93:2379,http://127.0.0.1:2379 Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.92:43682" (error "remote error: tls: bad certificate", ServerName "") Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.91:47588" (error "remote error: tls: bad certificate", ServerName "") Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.92:43684" (error "remote error: tls: bad certificate", ServerName "") Jan 26 01:48:12 etcd3 etcd[2525]: rejected connection from "192.168.0.91:47590" (error "remote error: tls: bad certificate", ServerName "") 7.3、所有文件改成https並重啟 etcd1節點etcd配置文件 cat > /etc/etcd/etcd.conf <<EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.91:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379" ETCD_NAME="etcd1" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.91:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" #開啟集群內部服務端認證並帶上客戶端證書 ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd etcd2節點etcd配置文件 cat >/etc/etcd/etcd.conf << EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.92:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379" ETCD_NAME="etcd2" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.92:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" #開啟集群內部服務端認證並帶上客戶端證書 ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd2-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd2-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd etcd3節點etcd配置文件 cat >/etc/etcd/etcd.conf << EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379" ETCD_NAME="etcd3" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd3.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd3-key.pem" #開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" #開啟集群內部服務端認證並帶上客戶端證書 ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 報錯解決： [root@etcd1 ~]# systemctl status etcd -l ● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-01-26 02:35:51 EST; 4min 18s ago Main PID: 3117 (etcd) CGroup: /system.slice/etcd.service └─3117 /usr/bin/etcd --name=etcd1 --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=https://192.168.0.91:2379,http://127.0.0.1:2379 Jan 26 02:35:51 etcd1 etcd[3117]: established a TCP streaming connection with peer c96f41ba37a00a16 (stream Message writer) Jan 26 02:35:51 etcd1 etcd[3117]: established a TCP streaming connection with peer c883f9e325d8667d (stream MsgApp v2 writer) Jan 26 02:35:51 etcd1 bash[3117]: WARNING: 2019/01/26 02:35:51 Failed to dial 192.168.0.91:2379: connection error: desc = "transport: 查看錯誤： WARNING: 2019/01/26 02:35:51 Failed to dial 192.168.0.91:2379: connection error: 原因： ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,k8s=https://192.168.0.92:2380,k8=https://192.168.0.93:2380" 糾正： ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,k83=https://192.168.0.93:2380" 重啟 systemctl daemon-reload && systemctl restart etcd 方式二：重建集群啟用https 注意：這種方式會丟失所有數據，一般在新建集群時使用。一般不使用這種方式 集群節點的peer訪問端點存儲在數據目錄，因此修改ETCD_INITIAL_CLUSTER參數后，最簡單讓其生效的方法就是重建集群。 在所有節點上修改etcd配置文件，將peer的url修改為https，配置相關證書，以etcd3為例，涉及參數如下： ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" [root@etcd3 ~]# cat /etc/etcd/etcd.conf ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379" ETCD_NAME="etcd3" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379" ETCD_INITIAL_CLUSTER="etcd4=https://192.168.0.94:2380,etcd1=https://192.168.0.91:2380,etcd3=https://192.168.0.93:2380,etcd2=https://192.168.0.92:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/pki/etcd.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" 在所有節點上刪除已有實例，重啟etcd。 systemctl stop etcd rm -rf /var/lib/etcd/default.etcd systemctl daemon-reload && systemctl restart etcd 8、etcd節點擴容 8.1、安裝etcd 本文通篇注意：對於新加入的成員，新成員自己每次修改配置文件后如果重啟失敗，那么新成員本身需要刪除殘留成員目錄，且lesder節點需要剔除新成員才可以啟動。 假如要擴容etcd4節點 etc4節點設置主機名、關閉防火牆、關閉selinux sed -i '$a\hostname=test4' /etc/sysconfig/network && hostnamectl set-hostname test4 sed -i '$a\test4' /etc/hostname cat >>/etc/hosts<< EOF 192.168.0.91 test1 192.168.0.92 test2 192.168.0.93 test3 192.168.0.94 test1 EOF 重啟 重啟目的是讓主機名永久生效 reboot 配置免密登錄 配置ntp服務器保證所有節點時間一致 ，參照 https://www.cnblogs.com/effortsing/p/10011459.html(這一步非必須) etcd4節點安裝、啟動etcd yum install -y etcd systemctl start etcd && systemctl enable etcd 查看狀態 etcdctl cluster-health member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379 不安裝etcd也可以添加新成員，這是添加了也沒有用 注意：安裝完etcd后就可以在leader上添加新成員了，但是這個時候不要添加，嚴格按照步驟做，否則會報各種錯誤。 8.2、以集群模式啟動 停掉etcd systemctl stop etcd 修改配置文件 cat > /etc/etcd/etcd.conf<< EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.94:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.0.94:2379,http://127.0.0.1:2379" ETCD_NAME="etcd4" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.94:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.94:2379" ETCD_INITIAL_CLUSTER="etcd4=http://192.168.0.94:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" EOF 注意：ETCD_INITIAL_CLUSTER_STATE="new" 這個new代表新成員，沒有這句話是無法加入的 刪除原來成員 必須刪除原來成員，否則導致沖突，無法啟動 rm -rf /var/lib/etcd/default.etcd 啟動 systemctl daemon-reload && systemctl restart etcd 8.3、leader節點上添加新成員 原集群的三個節點，其中有一個是leader節點，可以通過查看成員列表看到，isLeader=true，必須在leader節點上添加，否則報錯。 如果出現如下錯誤，說明之前添加過該成員，但是沒有剔除，所以再次添加該成員時候會如下報錯； 如果沒有出現如下錯誤，說明之前沒有添加過該成員。之前實驗的集群中添加過etcd4節點，但是沒有剔除etcd4，后來把etcd4虛擬機刪除后重新克隆了一份，所以會報如下錯誤。 etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member add etcd4 http://192.168.0.94:2380 如果出現如下錯誤： [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member add etcd4 http://192.168.0.94:2380 client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member https://192.168.0.93:2379 has no leader ; error #1: client: etcd member https://192.168.0.91:2379 has no leader ; error #2: client: etcd member https://192.168.0.92:2379 has no leader ; error #3: EOF 如果添加失敗有兩種情況 第一種情況： 是因為原有集群中已經有etcd4節點了，需要刪除該節點后才可以添加，刪除操作看下面： 先查看成員列表 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false bc721669bdca5256[unstarted]: peerURLs=http://192.168.0.94:2380 c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=true 發現果然存在etcd4成員，這是之前殘留的，剔除掉，否則沖突導致無法添加 刪除etcd4節點 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member remove bc721669bdca5256 Removed member bc721669bdca5256 from cluster 再次查看成員列表 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false 發現沒有了etcd4節點 第二種情況： 查看leader節點日志如下 [root@test1 pki]# journalctl -xe Jan 28 08:55:18 test1 etcd[2267]: failed to find member 6c70a880257288f in cluster a03ca7b6ecf1d2d4 Jan 28 08:55:18 test1 etcd[2267]: failed to find member 6c70a880257288f in cluster a03ca7b6ecf1d2d4 Jan 28 08:55:18 test1 etcd[2267]: streaming request ignored (ID mismatch got 5bbe42788a239cc6 want 6c70a880257288f) Jan 28 08:55:18 test1 etcd[2267]: streaming request ignored (ID mismatch got 5bbe42788a239cc6 want 6c70a880257288f) Jan 28 08:55:18 test1 etcd[2267]: failed to find member 6c70a880257288f in cluster a03ca7b6ecf1d2d4 再查看之前執行的命令過程 [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 6c70a880257288f: name=etcd1 peerURLs=http://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false 3f7336e156287ed0: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true 5bbe42788a239cc6: name=etcd2 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.92:2379 isLeader=false 62f2353f81e89de3[unstarted]: peerURLs=http://192.168.0.94:2380 [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update 6c70a880257288f https://192.168.0.4:2380 Updated member with ID 6c70a880257288f in cluster [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 6c70a880257288f: name=etcd1 peerURLs=https://192.168.0.4:2380 clientURLs=https://192.168.0.91:2379 isLeader=false 3f7336e156287ed0: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true 5bbe42788a239cc6: name=etcd2 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.92:2379 isLeader=false 62f2353f81e89de3[unstarted]: peerURLs=http://192.168.0.94:2380 [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member remove 62f2353f81e89de3 Removed member 62f2353f81e89de3 from cluster [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 6c70a880257288f: name=etcd1 peerURLs=https://192.168.0.4:2380 clientURLs=https://192.168.0.91:2379 isLeader=false 3f7336e156287ed0: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=true 5bbe42788a239cc6: name=etcd2 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.92:2379 isLeader=false [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member add etcd4 http://192.168.0.94:2380 client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member https://192.168.0.92:2379 has no leader ; error #1: client: etcd member https://192.168.0.91:2379 has no leader ; error #2: client: etcd member https://192.168.0.93:2379 has no leader [root@test1 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update 6c70a880257288f https://192.168.0.4:2380 Updated member with ID 6c70a880257288f in cluster 上面是之前執行的命令，找出update 6c70a880257288f https://192.168.0.4:2380，這里看出來把etcd1的id更新成了 https://192.168.0.4:2380，ip也沒寫全， 然后又剔除了62f2353f81e89de3，然后又開始添加etcd4節點，之后就開始報錯。 解決： 原理：把etcd1節點剔除重新添加即可，但是我沒有做成。 leader節點上剔除etcd1 [root@test3 pki]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member remove 6c70a880257288f Removed member 6c70a880257288f from cluster 添加etcd1 [root@test3 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member add etcd1 http://192.168.0.91:2380 Added member named etcd1 with ID f62393f31ba7a865 to cluster ETCD_NAME="etcd1" ETCD_INITIAL_CLUSTER="etcd3=https://192.168.0.93:2380,etcd2=https://192.168.0.91:2380,etcd1=http://192.168.0.91:2380" ETCD_INITIAL_CLUSTER_STATE="existing" 暫時先不用etcd1了 再次添加新成員 etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member add etcd4 http://192.168.0.94:2380 執行結果： [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member add etcd4 http://192.168.0.94:2380 Added member named etcd4 with ID 7cdd9649a07e40fb to cluster ETCD_NAME="etcd4" ETCD_INITIAL_CLUSTER="etcd4=http://192.168.0.94:2380,etcd1=https://192.168.0.91:2380,etcd3=https://192.168.0.93:2380,etcd2=https://192.168.0.92:2380" ETCD_INITIAL_CLUSTER_STATE="existing" 到此添加成功 查看節點列表和集群健康狀態 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 7cdd9649a07e40fb[unstarted]: peerURLs=http://192.168.0.94:2380 adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=false c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=true [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379 member 7cdd9649a07e40fb is unreachable: no available published client urls member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 剛添加完新成員看到集群時不健康的，需要修改etcd4配置文件，進行如下操作 8.4、糾正集群為健康狀態 創建證書目錄 mkdir -p /etc/etcd/pki/ 拷貝根證書並授權 scp root@192.168.0.91:/etc/etcd/pki/rootca* /etc/etcd/pki/ 生成etcd4服務端證書 cfssl print-defaults csr > etcd4-csr.json vi etcd4-csr.json cat >etcd4-csr.json <<EOF { "CN": "ETCD Peer on etcd4", "hosts": [ "192.168.0.94" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=server etcd4-csr.json | cfssljson -bare etcd4 拷貝服務端證書並授權 scp /etc/etcd/pki/etcd4*.pem root@192.168.0.94:/etc/etcd/pki/ 生成peer4證書 cfssl print-defaults csr > etcd4-peer-csr.json vi etcd4-csr.json cat >etcd4-peer-csr.json<<EOF { "CN": "ETCD Peer on etcd4", "hosts": [ "192.168.0.94" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ] } EOF cfssl gencert -ca=rootca.pem -ca-key=rootca-key.pem -config=ca-config.json -profile=peer etcd4-peer-csr.json | cfssljson -bare etcd4-peer 拷貝peer4證書並授權 scp /etc/etcd/pki/etcd4-peer*.pem root@192.168.0.94:/etc/etcd/pki/ 給所有證書授權 chown -R etcd:etcd /etc/etcd/pki/* 修改etcd4配置文件 把添加新成員那一步生成的結果寫到配置文件里面 注意：可不跨越步驟，直接進行下一步操作，否則無法啟動，會報各種錯誤 注意：添加PEER參數改變了集群內部通信方式需要刪除實例后重啟 cat > /etc/etcd/etcd.conf << EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="http://192.168.0.94:2380" ETCD_LISTEN_CLIENT_URLS="http://192.168.0.94:2379,http://127.0.0.1:2379" ETCD_NAME="etcd4" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.94:2380" ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.94:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380,etcd4=http://192.168.0.94:2380" ETCD_INITIAL_CLUSTER_STATE="existing" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #帶上集群內部客戶端證書同時開啟集群內部服務端認證 ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd4-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd4-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 注意：本次修改不能修改peerURLs、clientURLs為https模式，試過多次，修改后總是無法啟動。遵循之前的過程，逐步替換為https模式， 注意：本次修改配置文件里面必須要帶上peer證書和開啟服務端認證，否則無法啟動，會報如下錯誤。因為從添加新成員產生的結果看到其他節點都是安全訪問模式， 所以配置文件里面要帶上客戶端證書才可以訪問安全端口，peer證書既是服務端證書又是客戶端證書 啟動 systemctl daemon-reload && systemctl restart etcd 如果啟動失敗： [root@etcd4 ~]# systemctl daemon-reload && systemctl restart etcd Job for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details. [root@etcd4 ~]# journalctl -xe Jan 27 22:08:31 etcd4 etcd[1638]: listening for client requests on 127.0.0.1:2379 Jan 27 22:08:31 etcd4 etcd[1638]: listening for client requests on 192.168.0.94:2379 Jan 27 22:08:31 etcd4 etcd[1638]: open /etc/etcd/pki/etcd4-peer-key.pem: permission denied Jan 27 22:08:31 etcd4 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE Jan 27 22:08:31 etcd4 systemd[1]: Failed to start Etcd Server. 發現看到倒數第三行錯誤：open /etc/etcd/pki/etcd4-peer-key.pem: permission denied 是因為權限不足問題導致啟動失敗 添加授權 chown -R etcd:etcd /etc/etcd/pki/* 再次重啟： systemctl daemon-reload && systemctl restart etcd 啟動成功 本文通篇注意：對於新加入的成員，每次修改配置文件后如果啟動失敗就需要刪除本節點殘留成員目錄，而且需要從lesder節點剔除新成員，否則會有沖突，導致無法啟動成功， 查看節點列表和集群健康狀態 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 7cdd9649a07e40fb[unstarted]: peerURLs=http://192.168.0.94:2380 adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member 7cdd9649a07e40fb is unreachable: no available published client urls member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379 member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 如果發現集群仍然是不健康狀態，就需要刪除殘留成員目錄並重啟，如果是健康的，跳過下一步 查看日志報錯 [root@etcd4 ~]# systemctl status etcd ● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2019-01-27 21:50:28 EST; 17min ago Main PID: 1533 (etcd) CGroup: /system.slice/etcd.service └─1533 /usr/bin/etcd --name=etcd4 --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://192.168.0.94:2379,ht... Jan 27 22:08:16 etcd4 etcd[1533]: request cluster ID mismatch (got 9488eae2b4328f45 want 29ae782d95021b85) Jan 27 22:08:16 etcd4 etcd[1533]: request cluster ID mismatch (got 9488eae2b4328f45 want 29ae782d95021b85) Jan 27 22:08:16 etcd4 etcd[1533]: request cluster ID mismatch (got 9488eae2b4328f45 want 29ae782d95021b85) Jan 27 22:08:16 etcd4 etcd[1533]: request cluster ID mismatch (got 9488eae2b4328f45 want 29ae782d95021b85) Jan 27 22:08:16 etcd4 etcd[1533]: request cluster ID mismatch (got 9488eae2b4328f45 want 29ae782d95021b85) Jan 27 22:08:16 etcd4 etcd[1533]: request cluster ID mismatch (got 9488eae2b4328f45 want 29ae782d95021b85) 發現上面錯誤說明數據目錄中的成員和請求要加入的成員不匹配，刪除本節點數據目錄重啟即可 刪除殘留成員目錄並重啟 systemctl stop etcd rm -rf /var/lib/etcd/default.etcd systemctl daemon-reload && systemctl restart etcd 如果刪除后無法啟動就查看日志報錯，對應解決即可 例如下面錯誤： [root@test1 pki]# systemctl daemon-reload && systemctl restart etcd Jan 28 09:57:01 test1 etcd[3687]: couldn't find local name "etcd4" in the initial cluster configuration 從錯誤中很容易解決的，就是配置文件里面的名義定義錯誤了， 如果解決完錯誤還是無法啟動，那就剔除掉新成員重新添加，重復這個過程，直至提添加成功。 再次查看節點列表和集群健康狀態 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 7cdd9649a07e40fb: name=etcd4 peerURLs=http://192.168.0.94:2380 clientURLs=http://192.168.0.94:2379 isLeader=false adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member 7cdd9649a07e40fb is healthy: got healthy result from http://192.168.0.94:2379 member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379 member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 可以看到集群已經是健康狀態，但是peerURLs、clientURLs 都還不是https安全模式，下面就要修改為安全模式 先看下日志 [root@etcd4 ~]# journalctl -xe Jan 27 11:52:53 etcd4 etcd[3657]: could not get cluster response from https://192.168.0.92:2380: Get https://192.168.0.92:2380/members Jan 27 11:52:53 etcd4 etcd[3657]: could not get cluster response from https://192.168.0.93:2380: Get https://192.168.0.93:2380/members Jan 27 11:52:53 etcd4 etcd[3657]: cannot fetch cluster info from peer urls: could not retrieve cluster information from the given urls Jan 27 11:52:53 etcd4 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE Jan 27 11:52:53 etcd4 systemd[1]: Failed to start Etcd Server. 出現上面報錯沒有關系，繼續進行下面操作：修改為安全模式 8.5、開啟集群peer安全模式 修改etcd4節點的peer url為https etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update 7cdd9649a07e40fb https://192.168.0.4:2380 執行結果： [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member update 7cdd9649a07e40fb https://192.168.0.4:2380 Updated member with ID 7cdd9649a07e40fb in cluster 修改etcd4的peer工作端口為https；同時修改client工作端口為https，修改client url為https，帶上證書 cat >/etc/etcd/etcd.conf << EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.94:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.94:2379,http://127.0.0.1:2379" ETCD_NAME="etcd4" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.94:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.94:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380,etcd4=https://192.168.0.94:2380" ETCD_INITIAL_CLUSTER_STATE="existing" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #集群內部互相通信用的證書 ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd4-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd4-peer-key.pem" # 開啟集群內部客戶端認證 ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" # 開啟集群外部服務端認證 ETCD_CERT_FILE="/etc/etcd/pki/etcd4.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd4-key.pem" # 開啟集群外部客戶端認證 ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 啟動 systemctl daemon-reload && systemctl restart etcd 查看節點列表和集群健康狀態 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 7cdd9649a07e40fb: name=etcd4 peerURLs=https://192.168.0.94:2380 clientURLs=http://192.168.0.94:2379 isLeader=false adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member 7cdd9649a07e40fb is healthy: got healthy result from https://192.168.0.94:2379 member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379 member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 如果看到etcd4節點peerURLs已經是https模式了 注意：如果發現peerURLs不是https，原因在於執行"修改etcd3節點的peer url為https步驟"的時候掉了步驟最后面的https://192.168.0.93:2380 或者ID不正確，重新執行幾遍即可 在etcd4節點上查看日志 [root@etcd4 ~]# journalctl -xe Jan 27 13:25:06 etcd4 etcd[3926]: rejected connection from "192.168.0.91:55218" (error "tls: first record does not look like a TLS han Jan 27 13:25:06 etcd4 etcd[3926]: rejected connection from "192.168.0.91:55220" (error "tls: first record does not look like a TLS han Jan 27 13:25:06 etcd4 etcd[3926]: rejected connection from "192.168.0.93:42674" (error "tls: first record does not look like a TLS han Jan 27 13:25:06 etcd4 etcd[3926]: rejected connection from "192.168.0.93:42676" (error "tls: first record does not look like a TLS han 讓然有報錯，需要修改所有節點一致 8.6、修改所有節點配置文件一致並重啟 主要修改http改成https：ETCD_INITIAL_CLUSTER="etcd4=https://192.168.0.94:2380" 修改etcd1 cat > /etc/etcd/etcd.conf<< EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.91:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.91:2379,http://127.0.0.1:2379" ETCD_NAME="etcd1" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.91:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.91:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380,etcd4=https://192.168.0.94:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/pki/etcd1.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd1-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd1-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd1-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 修改etcd2 cat > /etc/etcd/etcd.conf<< EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.92:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.92:2379,http://127.0.0.1:2379" ETCD_NAME="etcd2" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.92:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.92:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380,etcd4=https://192.168.0.94:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/pki/etcd2.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd2-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd2-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd2-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 修改etcd3 cat > /etc/etcd/etcd.conf<< EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379" ETCD_NAME="etcd3" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380,etcd4=https://192.168.0.94:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/pki/etcd3.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd3-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 修改etcd4 cat > /etc/etcd/etcd.conf<< EOF ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.93:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.93:2379,http://127.0.0.1:2379" ETCD_NAME="etcd3" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.93:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.93:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.91:2380,etcd2=https://192.168.0.92:2380,etcd3=https://192.168.0.93:2380,etcd4=https://192.168.0.94:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_CERT_FILE="/etc/etcd/pki/etcd4.pem" ETCD_KEY_FILE="/etc/etcd/pki/etcd4-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" ETCD_PEER_CERT_FILE="/etc/etcd/pki/etcd3-peer.pem" ETCD_PEER_KEY_FILE="/etc/etcd/pki/etcd3-peer-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/rootca.pem" EOF 重啟 systemctl daemon-reload && systemctl restart etcd 查看節點列表和集群健康狀態 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true c775cff37a58077c: name=etcd4 peerURLs=https://192.168.0.94:2380 clientURLs=https://192.168.0.94:2379 isLeader=false c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem cluster-health member adff72f24ac33f4b is healthy: got healthy result from https://192.168.0.91:2379 member c775cff37a58077c is healthy: got healthy result from https://192.168.0.94:2379 member c883f9e325d8667d is healthy: got healthy result from https://192.168.0.93:2379 member c96f41ba37a00a16 is healthy: got healthy result from https://192.168.0.92:2379 cluster is healthy 可以看到全部是https模式 在etcd4節點上查看日志 [root@etcd4 ~]# journalctl -xe Jan 27 16:57:45 etcd4 etcd[5902]: rejected connection from "192.168.0.93:39910" (error "EOF", ServerName "") Jan 27 16:57:45 etcd4 etcd[5902]: rejected connection from "192.168.0.93:39914" (error "EOF", ServerName "") Jan 27 16:57:46 etcd4 etcd[5902]: peer c883f9e325d8667d became active Jan 27 16:57:46 etcd4 etcd[5902]: established a TCP streaming connection with peer c883f9e325d8667d (stream MsgApp v2 reader) Jan 27 16:57:46 etcd4 etcd[5902]: established a TCP streaming connection with peer c883f9e325d8667d (stream Message reader) 仍然報錯，暫時未解決，不影響使用 9、刪除成員 注意：刪除成員是針對新成員來說的，對於原有集群成員，不能隨便刪除，因為已經有很多數據。如果必須刪除，需要做備份遷移后才可以刪除 例如要刪除etcd4成員 先查看成員列表 [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member list 1da3d0181b2c051: name=etcd4 peerURLs=http://192.168.0.94:2380 clientURLs=http://192.168.0.94:2379 isLeader=false adff72f24ac33f4b: name=etcd1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true c883f9e325d8667d: name=etcd3 peerURLs=https://192.168.0.93:2380 clientURLs=https://192.168.0.93:2379 isLeader=false c96f41ba37a00a16: name=etcd2 peerURLs=https://192.168.0.92:2380 clientURLs=https://192.168.0.92:2379 isLeader=false 剔除成員 etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member remove 1da3d0181b2c051 執行結果： [root@etcd1 ~]# etcdctl --ca-file /etc/etcd/pki/rootca.pem --cert-file /etc/etcd/pki/etcdctl.pem --key-file /etc/etcd/pki/etcdctl-key.pem member remove 1da3d0181b2c051 Removed member 1da3d0181b2c051 from cluster 停掉etcd服務 systemctl stop etcd 刪除成員目錄 rm -rf /var/lib/etcd/default.etcd 如果上面不按順序來，再次添加完成員客戶端是無法啟動的，會報如下錯誤，看里面提示 the data-dir used by this member must be removed. [root@etcd4 ~]# systemctl daemon-reload && systemctl restart etcd [root@etcd4 ~]# systemctl status etcd -l ● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: inactive (dead) since Sun 2019-01-27 14:38:29 CST; 1s ago Process: 4503 ExecStart=/bin/bash -c GOMAXPROCS=$(nproc) /usr/bin/etcd --name="${ETCD_NAME}" --data-dir="${ETCD_DATA_DIR}" --listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" (code=exited, status=0/SUCCESS) Main PID: 4503 (code=exited, status=0/SUCCESS) Jan 27 14:38:29 etcd4 etcd[4503]: the data-dir used by this member must be removed. Jan 27 14:38:29 etcd4 etcd[4503]: aborting publish because server is stopped Jan 27 14:38:29 etcd4 etcd[4503]: stopping peer c96f41ba37a00a16... Jan 27 14:38:29 etcd4 etcd[4503]: stopped streaming with peer c96f41ba37a00a16 (writer) Jan 27 14:38:29 etcd4 etcd[4503]: stopped streaming with peer c96f41ba37a00a16 (writer) Jan 27 14:38:29 etcd4 etcd[4503]: stopped HTTP pipelining with peer c96f41ba37a00a16 Jan 27 14:38:29 etcd4 etcd[4503]: stopped streaming with peer c96f41ba37a00a16 (stream MsgApp v2 reader) Jan 27 14:38:29 etcd4 etcd[4503]: stopped streaming with peer c96f41ba37a00a16 (stream Message reader) Jan 27 14:38:29 etcd4 etcd[4503]: stopped peer c96f41ba37a00a16 Jan 27 14:38:29 etcd4 systemd[1]: Started Etcd Server. 參照文檔： https://www.jianshu.com/p/3015d514bae3 https://lprincewhn.github.io/2018/09/15/etcd-ha-pki-01.html http://www.mamicode.com/info-detail-1737556.html http://www.cnblogs.com/breg/p/5728237.html