本文所有命令均在 TLS 環境下運行,如需參考,請自行更改為您的環境(節點IP,證書路徑),無證書環境請刪除證書相關指令
本文所有命令均在 etcdctl 默認api ,即 etcd api v2 下操作,v3 指令略有改動可能不匹配,詳情請查閱官方文檔:https://etcd.io/docs/
Etcd 使用
-
舉例:創建、查詢、刪除 key ( /test/ok,值為 11)
# Etcd 錄入數據示例 ETCDCTL_API=3 etcdctl \ --endpoints=https://172.16.10.70:2379 \ --cacert=/etc/kubernetes/ssl/ca.pem \ --cert=/etc/etcd/ssl/etcd.pem \ --key=/etc/etcd/ssl/etcd-key.pem \ put /test/ok 11
# Etcd 查詢數據示例 ETCDCTL_API=3 etcdctl \ --endpoints=https://172.16.10.70:2379 \ --cacert=/etc/kubernetes/ssl/ca.pem \ --cert=/etc/etcd/ssl/etcd.pem \ --key=/etc/etcd/ssl/etcd-key.pem \ get /test/ok
# Etcd 刪除數據示例 ETCDCTL_API=3 etcdctl \ --endpoints=https://172.16.10.70:2379 \ --cacert=/etc/kubernetes/ssl/ca.pem \ --cert=/etc/etcd/ssl/etcd.pem \ --key=/etc/etcd/ssl/etcd-key.pem \ del /test/ok
通過 Curl 來維護 Etcd
-
查看版本
curl ‐k ‐‐cert /etc/etcd/ssl/etcd.pem ‐‐key /etc/etcd/ssl/etcd‐key.pem https://127.0.0.1:2379/version
-
查看 Etcd 暴露出來的 prometheus 指標,在 prometheus 對其監控時可調用
curl ‐k ‐‐cert /etc/etcd/ssl/etcd.pem ‐‐key /etc/etcd/ssl/etcd‐key.pem https://127.0.0.1:2379/metrics
通過 Etcdctl 查看 版本
-
查看 etcd、etcd api v2 版本
etcdctl -v
-
查看 etcd、etcd api v3 版本
ETCDCTL_API=3 etcdctl version
查看集群健康狀態
etcdctl \ --endpoints=https://172.16.10.70:2379 \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/etcd/ssl/etcd.pem \ --key-file=/etc/etcd/ssl/etcd-key.pem \ cluster-health
查看集群成員、哪個是leader節點
etcdctl \ --endpoints=https://172.16.10.70:2379 \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/etcd/ssl/etcd.pem \ --key-file=/etc/etcd/ssl/etcd-key.pem \ member list
刪除 Etcd 節點
-
查詢節點 ID
etcdctl \ ‐‐endpoints=https://172.16.10.70:2379 \ ‐‐ca‐file=/etc/kubernetes/ssl/ca.pem \ ‐‐cert‐file=/etc/etcd/ssl/etcd.pem \ ‐‐key‐file=/etc/etcd/ssl/etcd‐key.pem \
member list
340acbd004e6bcdb: name=etcd3 peerURLs=https://172.16.10.72:2380 clientURLs=https://172.16.10.72:2379
isLeader=false
9784cb04cceb3a48: name=etcd1 peerURLs=https://172.16.10.70:2380 clientURLs=https://172.16.10.70:2379
isLeader=true
ba343177666dd96e: name=etcd2 peerURLs=https://172.16.10.71:2380 clientURLs=https://172.16.10.71:2379
isLeader=false
-
刪除節點,如刪除 Eecd3
etcdctl \ ‐‐endpoints=https://172.16.10.70:2379 \ ‐‐ca‐file=/etc/kubernetes/ssl/ca.pem \ ‐‐cert‐file=/etc/etcd/ssl/etcd.pem \ ‐‐key‐file=/etc/etcd/ssl/etcd‐key.pem \ member remove 340acbd004e6bcdb
-
修改配置文件 etcd.conf,修改參數 ETCD_INITIAL_CLUSTER 並移除節點信息,重啟etcd服務
加入 Etcd 節點
已存在的 Etcd 節點故障重新添加(例 etcd3 重新添加)
1)在群集中刪除故障節點
-
在任意一 etcd 節點服務器查詢該節點 ID,通過ID刪除故障節點,操作步驟如上
-
刪除目標節點的數據
# 停止目標節點 etcd 服務 systemctl stop etcd # 刪除前備份 cd /var/lib/ && mkdir ‐p etcd_bak && tar ‐czvf etcd_bak/etcd_`date +%Y%m%d%H%M%S`.tar.gz etcd # 刪除節點數據 rm ‐rf /var/lib/etcd/*
2)編輯目標節點配置文件,將 --initial-cluster-state值為 existing (否則會生成新的ID,與原ID不匹配將無法加入集群)
vim /etc/etcd/etcd.conf [member] ETCD_NAME=etcd3 ETCD_DATA_DIR="/var/lib/etcd/" ETCD_SNAPSHOT_COUNT="100" ETCD_HEARTBEAT_INTERVAL="100" ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://172.16.10.72:2380" ETCD_LISTEN_CLIENT_URLS="https://172.16.10.72:2379,https://127.0.0.1:2379" ETCD_MAX_SNAPSHOTS="5" ETCD_MAX_WALS="5" # [cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.10.72:2380" ETCD_INITIAL_CLUSTER="etcd1=https://172.16.10.70:2380,etcd2=https://172.16.10.71:2380,etcd3=https://172.16.10.72:2380" ETCD_INITIAL_CLUSTER_STATE="existing" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://172.16.10.72:2379" # [security] ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem" ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/kubernetes/ssl/ca.pem" ETCD_AUTO_TLS="true" ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/ssl/ca.pem" ETCD_PEER_AUTO_TLS="true"
3)加入節點至集群,需輸入目標節點的 etcd name 和 PEER_URLS
etcdctl \ ‐‐endpoints=https://172.16.10.70:2379 \ ‐‐ca‐file=/etc/kubernetes/ssl/ca.pem \ ‐‐cert‐file=/etc/etcd/ssl/etcd.pem \ ‐‐key‐file=/etc/etcd/ssl/etcd‐key.pem \ member add etcd3 https://172.16.10.72:2380
4)啟動目標節點 etcd 服務
systemctl start etcd && systemctl status etcd
5)查看集群健康狀態
etcdctl \ ‐‐endpoints=https://172.16.10.70:2379 \ ‐‐ca‐file=/etc/kubernetes/ssl/ca.pem \ ‐‐cert‐file=/etc/etcd/ssl/etcd.pem \ ‐‐key‐file=/etc/etcd/ssl/etcd‐key.pem \ cluster‐health
對 Etcd 進行快照備份
ETCDCTL_API=3 etcdctl \ ‐‐endpoints=https://172.16.10.70:2379 \ ‐‐cacert=/etc/kubernetes/ssl/ca.pem \ ‐‐cert=/etc/etcd/ssl/etcd.pem \ ‐‐key=/etc/etcd/ssl/etcd‐key.pem \ snapshot save /tmp/snapshot_`date +%Y%m%d%H%M%S`.db ETCDCTL_API=3:表示使用etcd的v3版本的API接口 注:一定要添加ETCDCTL_API=3才能正常備份;如果不添加將無法備份
通過快照備份恢復 Etcd集群;(每個節點都要執行)
-
停止 Etcd 服務
systemctl stop etcd
-
備份並刪除當前 Etcd 數據
cd /var/lib/ && mkdir ‐p etcd_bak && tar ‐czvf etcd_bak/etcd_`date +%Y%m%d%H%M%S`.tar.gz etcd ‐‐remov e‐files
-
還原快照鏡像
ETCDCTL_API=3 etcdctl \ ‐‐cacert=/etc/kubernetes/ssl/ca.pem \ ‐‐cert=/etc/etcd/ssl/etcd.pem \ ‐‐key=/etc/etcd/ssl/etcd‐key.pem \ ‐‐name etcd1 \ ‐‐data‐dir=/var/lib/etcd \ ‐‐initial‐cluster etcd1=https://172.16.10.70:2380,etcd2=https://172.16.10.71:2380,etcd3=https://172.16.10.72:2380 \ ‐‐initial‐cluster‐token k8s‐etcd‐cluster \ ‐‐initial‐advertise‐peer‐urls https://172.16.10.70:2380 \ snapshot restore /tmp/2019‐12‐18_snapshot.db
‐‐name:表示當前etcd節點的名字(非主機名) ‐‐data‐dir:表示當前 etcd 節點的數據目錄 ‐‐initial‐cluster:集群中所有節點的peer訪問地址;例:etcd1=https:///172.16.10.70:2380,etcd2=https:///172.16.10.71:2380,etcd3=https:///172.16.10.72:2380
‐‐initial‐cluster‐token:集群中各節點通信的token ‐‐initial‐advertise‐peer‐urls:當前節點對其它節點的通信地址
-
啟動 所有 Etcd 節點服務器
systemctl start etcd
-
查看集群健康狀態
etcdctl \ ‐‐endpoints=https://172.16.10.70:2379 \ ‐‐ca‐file=/etc/kubernetes/ssl/ca.pem \ ‐‐cert‐file=/etc/etcd/ssl/etcd.pem \ ‐‐key‐file=/etc/etcd/ssl/etcd‐key.pem \ cluster‐health
沒有進行快照備份,通過數據目錄的 db 恢復
注意:此方法恢復數據可能不完整,僅建議極端環境下使用,常規數據恢復請使用快照
-
如果當前 Etcd 集群故障,且沒有快照備份文件,可通過數據目錄的 db 恢復數據;
-
從數據目錄 db 復制而來數據源,沒有完整性hash,需要 --skip-hash-check=true 參數跳過完整性檢查。
ETCDCTL_API=3 etcdctl \ --cacert=/etc/kubernetes/ssl/ca.pem \ --cert=/etc/etcd/ssl/etcd.pem \ --key=/etc/etcd/ssl/etcd-key.pem \ --name etcd3 \ --data-dir=/var/lib/etcd \ --initial-cluster etcd1=https://172.16.10.70:2380,etcd2=https://172.16.10.71:2380,etcd3=https://172.16.10.72:2380 \ --initial-cluster-token k8s-etcd-cluster \ --initial-advertise-peer-urls https://172.16.10.72:2380 \ --skip-hash-check=true \ snapshot restore /var/lib/etcd_bak/etcd/member/snap/db --name:表示當前etcd節點的名字(非主機名) --data-dir:表示當前 etcd 節點的數據目錄 --initial-cluster:集群中所有節點的peer訪問地址;例:https://172.16.10.70:2380,etcd2=https://172.16.10.71:2380,etcd3=https://172.16.10.72:2380
--initial-cluster-token:集群中各節點通信的token --initial-advertise-peer-urls:當前節點對其它節點的通信地址