ETCD 集群部署

目錄

• 部署ETCD集群
• Etcd 解析
  • 方法一 修改hosts文件
  • 方法二 增加bind解析
• 下載和分發etcd二進制文件
• 創建etcd證書和私鑰
• 創建etcd的啟動文件
• 測試 ETCD 集群狀態

下面的操作依托於上一篇文章

部署前期准備工作

部署ETCD集群

這里使用的ETCD為三節點高可用集群，步驟如下

• 下載和分發etcd二進制文件
• 創建etcd集群各節點的x509證書，用於加密客戶端(如kubectl)與etcd集群、etcd集群之間的數據流
• 創建etcd的system unit文件，配置服務參數
• 檢查集群工作狀態

注意: 沒有特殊說明都在node01節點操作

Etcd 解析

本次使用etcd單獨的域名解析

方法一 修改hosts文件

在所有機器上操作， /etc/hosts 文件最后增加下面內容，也可以替換成自己的IP

10.0.20.11 etcd01 etcd01.k8s.com
10.0.20.12 etcd02 etcd02.k8s.com
10.0.20.13 etcd03 etcd03.k8s.com

方法二 增加bind解析

如果使用的內網 DNS bind 做內網解析增加下面解析

etcd01  IN  A   10.0.20.11
etcd02  IN  A   10.0.20.12
etcd03  IN  A   10.0.20.13

下載和分發etcd二進制文件

二進制文件在 部署前期准備工作 文章中已經下載好，直接使用；

分發二進制文件到ETCD集群節點

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-v3.3.13-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done

創建etcd證書和私鑰

cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.0.20.11",
    "10.0.20.12",
    "10.0.20.13",
    "etcd01.k8s.com",
    "etcd02.k8s.com",
    "etcd03.k8s.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF


#host字段指定授權使用該證書的etcd節點IP或域名列表，需要將etcd集群的3個節點都添加其中

生成證書和私鑰

cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ls etcd*pem

分發證書和私鑰到etcd各個節點

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/etcd/cert"
    scp etcd*.pem root@${node_ip}:/etc/etcd/cert/
  done

創建etcd的啟動文件

這里相對應的etcd 的配置，就保存在啟動文件中

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > etcd.service.template <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \\
  --data-dir=${ETCD_DATA_DIR} \\
  --wal-dir=${ETCD_WAL_DIR} \\
  --name=##NODE_NAME## \\
  --cert-file=/etc/etcd/cert/etcd.pem \\
  --key-file=/etc/etcd/cert/etcd-key.pem \\
  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-cert-file=/etc/etcd/cert/etcd.pem \\
  --peer-key-file=/etc/etcd/cert/etcd-key.pem \\
  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --listen-peer-urls=https://##NODE_IP##:2380 \\
  --initial-advertise-peer-urls=https://##NODE_IP##:2380 \\
  --listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \\
  --advertise-client-urls=https://##NODE_IP##:2379 \\
  --initial-cluster-token=etcd-cluster-0 \\
  --initial-cluster=${ETCD_NODES} \\
  --initial-cluster-state=new \\
  --auto-compaction-mode=periodic \\
  --auto-compaction-retention=1 \\
  --max-request-bytes=33554432 \\
  --quota-backend-bytes=6442450944 \\
  --heartbeat-interval=250 \\
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

配置說明 (此處不需要修改任何配置)

• WorkDirectory、–data-dir 指定etcd工作目錄和數據存儲為${ETCD_DATA_DIR},需要在啟動前創建這個目錄 (后面跟着我操作就可以，會有創建步驟)
• –wal-dir 指定wal目錄，為了提高性能，一般使用SSD和–data-dir不同的盤
• –name 指定節點名稱，當–initial-cluster-state值為new時，–name的參數值必須位於–initial-cluster列表中
• –cert-file、–key-file ETCD server與client通信時使用的證書和私鑰
• –trusted-ca-file 簽名client證書的CA證書，用於驗證client證書
• –peer-cert-file、–peer-key-file ETCD與peer通信使用的證書和私鑰
• –peer-trusted-ca-file 簽名peer證書的CA證書，用於驗證peer證書

拆分三個配置文件，並修改信息

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${ETCD_NAMES[i]}/" -e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service 
  done

etcd*.service

分發生成的etcd啟動文件到對應的服務器

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service
  done

啟動etcd服務

etcd首次進程啟動會等待其他節點加入etcd集群，執行啟動命令會卡頓一會，為正常現象

遠程創建對應 ETCD 的數據目錄等

source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &
  done

測試 ETCD 集群狀態

檢查啟動結果

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status etcd|grep Active"
  done

輸出結果:

[root@node01 work]# for node_ip in ${MASTER_IPS[@]}
>   do
>     echo ">>> ${node_ip}"
>     ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
>     --endpoints=https://${node_ip}:2379 \
>     --cacert=/etc/kubernetes/cert/ca.pem \
>     --cert=/etc/etcd/cert/etcd.pem \
>     --key=/etc/etcd/cert/etcd-key.pem endpoint health
>   done
>>> 10.0.20.11
https://10.0.20.11:2379 is healthy: successfully committed proposal: took = 1.609991ms
>>> 10.0.20.12
https://10.0.20.12:2379 is healthy: successfully committed proposal: took = 1.117871ms
>>> 10.0.20.13
https://10.0.20.13:2379 is healthy: successfully committed proposal: took = 1.49139ms

通過下面命令查看當前etcd集群leader

source /opt/k8s/bin/environment.sh
ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
  -w table --cacert=/etc/kubernetes/cert/ca.pem \
  --cert=/etc/etcd/cert/etcd.pem \
  --key=/etc/etcd/cert/etcd-key.pem \
  --endpoints=${ETCD_ENDPOINTS} endpoint status

輸出結果如下：

[root@node01 work]# source /opt/k8s/bin/environment.sh
[root@node01 work]# ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
>   -w table --cacert=/etc/kubernetes/cert/ca.pem \
>   --cert=/etc/etcd/cert/etcd.pem \
>   --key=/etc/etcd/cert/etcd-key.pem \
>   --endpoints=${ETCD_ENDPOINTS} endpoint status
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://etcd01.k8s.com:2379 | 6330dc0a28f62066 |  3.3.13 |   16 kB |     false |        35 |         14 |
| https://etcd02.k8s.com:2379 | 77bc4da10f4c40bb |  3.3.13 |   16 kB |      true |        35 |         14 |
| https://etcd03.k8s.com:2379 | d2573d5cc998d0f0 |  3.3.13 |   16 kB |     false |        35 |         14 |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+

---

---

如果對ETCD集群安裝不熟悉的，可以參考文章 CentOS 7 ETCD集群配置大全