CentOS 7 ETCD集群配置大全

本文轉載自查看原文 2019-11-07 12:48 1393 kubernetes/ Linux

前言
環境准備
安裝
靜態集群
- 配置
- 啟動測試
  - 查看集群狀態
生成TLS證書
- etcd證書創建
靜態TLS集群
- etcd 配置
- 啟動測試
  - 檢查TLS集群狀態
ETCD 動態集群基於DNS的SRV解析自動發現
ETCD TLS動態集群基於DNS的SRV解析自動發現
報錯解決
- 1. 證書報錯 bad certificate
  - 解決
- 2. DNS 的 SRV 解析報錯 cannot find local etcd member "etcd1" in SRV records
  - http不帶證書解析如下
    - bind 的解析
    - dnsmasq 的解析
  - https帶證書解析如下
    - bind 的解析
    - dnsmasq 的解析

前言

Etcd 是 CoreOS 基於 Raft 開發的分布式 key-value 存儲，可用於服務發現、共享配置以及一致性保障（如數據庫選主、分布式鎖等）

本次環境，是用於k8s集群，由於在二進制部署 k8s 中，由於 Etcd 集群導致各種各樣的問題，特意抽出時間來研究 Etcd 集群。

Etcd 集群配置分為三種:

靜態發現
Etcd 動態發現
DNS 動態發現通過DNS的SRV解析動態發現集群

本次主要基於靜態發現和 DNS動態發現兩種，並結合自簽的TLS證書來創建集群。

環境准備

此環境實際用於 k8s 中的ETCD集群使用，用於本次文檔

主機名	角色	IP	系統版本	內核版本
node01.k8s.com	node01	192.168.1.91	CentOS 7.7	5.1.4-1.el7.elrepo.x86_64
node02.k8s.com	node02	192.168.1.92	CentOS 7.7	5.1.4-1.el7.elrepo.x86_64
node03.k8s.com	node03	192.168.1.93	CentOS 7.7	5.1.4-1.el7.elrepo.x86_64

安裝

在三台機器上均執行

[root@node01 ~]# yum install etcd -y
[root@node01 ~]# rpm -qa etcd
etcd-3.3.11-2.el7.centos.x86_64

創建Etcd所需目錄，在三台機器上均執行

mkdir /data/k8s/etcd/{data,wal} -p
mkdir -p /etc/kubernetes/cert
chown -R etcd.etcd /data/k8s/etcd

靜態集群

配置

node01 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.91:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.91:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.91:2379"

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node02 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.92:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.92:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd2"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.92:2379"

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node03 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.93:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.93:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd3"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.93:2379"

ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

啟動測試

[root@node01 etcd]# systemctl start etcd
[root@node01 etcd]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-07 09:28:54 CST; 5s ago
 Main PID: 1546 (etcd)
    Tasks: 8
   Memory: 41.3M
   CGroup: /system.slice/etcd.service
           └─1546 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=http://192.168.1.91:2379

Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 [term: 1] received a MsgVote message with higher term from 9c64fba479c5e94 [term: 2]
Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 became follower at term 2
Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 [logterm: 1, index: 3, vote: 0] cast MsgVote for 9c64fba479c5e94 [logterm: 1, index: 3] at term 2
Nov 07 09:28:54 node01.k8s.com etcd[1546]: raft.node: 3b8b38de05e2c497 elected leader 9c64fba479c5e94 at term 2
Nov 07 09:28:54 node01.k8s.com etcd[1546]: published {Name:etcd1 ClientURLs:[http://192.168.1.91:2379]} to cluster 19456f0bfd57284e
Nov 07 09:28:54 node01.k8s.com etcd[1546]: ready to serve client requests
Nov 07 09:28:54 node01.k8s.com etcd[1546]: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!
Nov 07 09:28:54 node01.k8s.com systemd[1]: Started Etcd Server.
Nov 07 09:28:54 node01.k8s.com etcd[1546]: set the initial cluster version to 3.3
Nov 07 09:28:54 node01.k8s.com etcd[1546]: enabled capabilities for version 3.3

查看 /var/log/message 日志中，會有日下體現：

Nov  7 09:28:53 node02 etcd: added member 9c64fba479c5e94 [http://192.168.1.92:2380] to cluster 19456f0bfd57284e
Nov  7 09:28:53 node02 etcd: added member 3b8b38de05e2c497 [http://192.168.1.91:2380] to cluster 19456f0bfd57284e
Nov  7 09:28:53 node02 etcd: added member 76ea8679db7365b3 [http://192.168.1.93:2380] to cluster 19456f0bfd57284e

查看集群狀態

[root@node01 etcd]# ETCDCTL_API=3 etcdctl --endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 endpoint health
http://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.103545ms
http://192.168.1.93:2379 is healthy: successfully committed proposal: took = 2.122478ms
http://192.168.1.91:2379 is healthy: successfully committed proposal: took = 2.690215ms

[root@node01 etcd]# etcdctl --endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 cluster-health
member 9c64fba479c5e94 is healthy: got healthy result from http://192.168.1.92:2379
member 3b8b38de05e2c497 is healthy: got healthy result from http://192.168.1.91:2379
member 76ea8679db7365b3 is healthy: got healthy result from http://192.168.1.93:2379
cluster is healthy

生成TLS證書

使用自簽證書

CA(Certificate Authority)是自簽名的根證書，用來簽名后續創建的其他證書。本文章使用CloudFlare的PKI工具cfssl創建所有證書。

etcd證書創建

整個證書的創建過程均在 node01 上操作;

安裝cfssl工具集

mkdir -p /opt/k8s/cert && cd /opt/k8s
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
mv cfssl_linux-amd64 /opt/k8s/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssljson_linux-amd64 /opt/k8s/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /opt/k8s/bin/cfssl-certinfo
chmod +x /opt/k8s/bin/*
echo 'export PATH=/opt/k8s/bin:$PATH' >> ~/.bash_profile
source ~/.bash_profile

生成證書

創建根證書 (CA)

CA證書是集群所有節點共享的，只需要創建一個CA證書，后續創建的所有證書都是由它簽名

創建配置文件

CA配置文件用於配置根證書的使用場景(profile)和具體參數

(usage、過期時間、服務端認證、客戶端認證、加密等)

cd /opt/k8s/work
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF


######################
signing 表示該證書可用於簽名其它證書，生成的ca.pem證書找中CA=TRUE
server auth 表示client可以用該證書對server提供的證書進行驗證
client auth 表示server可以用該證書對client提供的證書進行驗證

創建證書簽名請求文件

cd /opt/k8s/work
cat > ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF



#######################
CN CommonName,kube-apiserver從證書中提取該字段作為請求的用戶名(User Name)，瀏覽器使用該字段驗證網站是否合法
O Organization,kube-apiserver 從證書中提取該字段作為請求用戶和所屬組(Group)
kube-apiserver將提取的User、Group作為RBAC授權的用戶和標識

生成CA證書和私鑰

cd /opt/k8s/work
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls ca*

創建etcd證書和私鑰

cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.1.91",
    "192.168.1.92",
    "192.168.1.93",
    "k8s.com",
    "etcd1.k8s.com",
    "etcd2.k8s.com",
    "etcd3.k8s.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

#host字段指定授權使用該證書的etcd節點IP或域名列表，需要將etcd集群的3個節點都添加其中

生成證書和私鑰

cd /opt/k8s/work

cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
    
ls etcd*pem -l
-rw------- 1 root root 1675 Nov  7 09:52 etcd-key.pem
-rw-r--r-- 1 root root 1444 Nov  7 09:52 etcd.pem

etcd 使用的TLS證書創建完成

分發證書到各節點上

要做所有節點上創建對應的目錄

mkdir /data/k8s/etcd/{data,wal} -p
mkdir -p /etc/kubernetes/cert
chown -R etcd.etcd /data/k8s/etcd

分發證書

cd /opt/k8s/work
scp ca*.pem ca-config.json 192.168.1.91:/etc/kubernetes/cert
scp ca*.pem ca-config.json 192.168.1.92:/etc/kubernetes/cert 
scp ca*.pem ca-config.json 192.168.1.93:/etc/kubernetes/cert
scp etcd*pem 192.168.1.91:/etc/etcd/cert/
scp etcd*pem 192.168.1.92:/etc/etcd/cert/
scp etcd*pem 192.168.1.93:/etc/etcd/cert/

在所有節點上執行:

chown -R etcd.etcd /etc/etcd/cert

靜態TLS集群

etcd 配置

node01 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.91:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.91:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.91:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.91:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"

node02 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.92:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd2"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.92:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.92:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"

node03 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.93:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd3"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.93:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.93:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"

啟動測試

[root@node01 work]# systemctl start etcd
[root@node01 work]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-07 10:15:58 CST; 5s ago
 Main PID: 2078 (etcd)
    Tasks: 8
   Memory: 28.9M
   CGroup: /system.slice/etcd.service
           └─2078 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=https://192.168.1.91:2379

Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe [term: 1] received a MsgVote message with higher term from af05139f75a68867 [term: 2]
Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe became follower at term 2
Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe [logterm: 1, index: 3, vote: 0] cast MsgVote for af05139f75a68867 [logterm: 1, index: 3] at term 2
Nov 07 10:15:58 node01.k8s.com etcd[2078]: raft.node: 2a40d8ba966d12fe elected leader af05139f75a68867 at term 2
Nov 07 10:15:58 node01.k8s.com etcd[2078]: published {Name:etcd1 ClientURLs:[https://192.168.1.91:2379]} to cluster f3e9c54e1aafb3c1
Nov 07 10:15:58 node01.k8s.com etcd[2078]: ready to serve client requests
Nov 07 10:15:58 node01.k8s.com etcd[2078]: serving client requests on 192.168.1.91:2379
Nov 07 10:15:58 node01.k8s.com systemd[1]: Started Etcd Server.
Nov 07 10:15:58 node01.k8s.com etcd[2078]: set the initial cluster version to 3.3
Nov 07 10:15:58 node01.k8s.com etcd[2078]: enabled capabilities for version 3.3

查看 /var/log/message 日志中，會有日下體現：

Nov  7 10:15:57 node01 etcd: added member 2a40d8ba966d12fe [https://192.168.1.91:2380] to cluster f3e9c54e1aafb3c1
Nov  7 10:15:57 node01 etcd: added member af05139f75a68867 [https://192.168.1.92:2380] to cluster f3e9c54e1aafb3c1
Nov  7 10:15:57 node01 etcd: added member c3bab7c20fba3f60 [https://192.168.1.93:2380] to cluster f3e9c54e1aafb3c1

檢查TLS集群狀態

ETCDCTL_API=3 etcdctl \
    --endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \
    --cacert=/etc/kubernetes/cert/ca.pem \
    --cert=/etc/etcd/cert/etcd.pem \
    --key=/etc/etcd/cert/etcd-key.pem endpoint health

# 輸出
https://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.317022ms
https://192.168.1.91:2379 is healthy: successfully committed proposal: took = 1.59958ms
https://192.168.1.93:2379 is healthy: successfully committed proposal: took = 1.453049ms

etcdctl \
--endpoint=https://etcd1.k8s.com:2379 \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/etcd/cert/etcd.pem \
--key-file=/etc/etcd/cert/etcd-key.pem cluster-health

# 輸出
member 40a8f19a5db99534 is healthy: got healthy result from https://etcd2.k8s.com:2379
member 9888555207dbf0e0 is healthy: got healthy result from https://etcd3.k8s.com:2379
member a0d541999e9eb3b3 is healthy: got healthy result from https://etcd1.k8s.com:2379
cluster is healthy

ETCD 動態集群基於DNS的SRV解析自動發現

需要局域網內部有DNS服務器

添加SRV解析

目前常用的內部DNS服務有兩種，bind、dnsmasq

在下面都會列出具體的配置，但只需要配置其中之一即可；

方法一: 使用`bind`配置SRV解析

如果內部沒有bind服務，可以參考部署文檔文章: https://www.cnblogs.com/winstom/p/11806962.html

使用域名為 : k8s.com，在bind的zone文件中添加如下解析:

etcd1   IN      A       192.168.1.91
etcd2   IN      A       192.168.1.92
etcd3   IN      A       192.168.1.93
_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd1
_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd2
_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd3
_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd1
_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd2
_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd3

修改之后重新加載配置文件:

[root@jenkins named]# named-checkzone k8s.com k8s.com.zone 
zone k8s.com/IN: loaded serial 0
OK
[root@jenkins named]# rndc reload
server reload successful

方法二: 使用`dnsmasq`配置SRV解析

如果內部沒有dnsmasq服務，可以參考部署文檔文章: https://www.cnblogs.com/winstom/p/11809066.html

使用域名為 : k8s.com，具體修改如下:

在/etc/dnsmasq_hosts新增下面內容

192.168.1.91 etcd1 etcd1.k8s.com
192.168.1.92 etcd2 etcd2.k8s.com
192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析內容

srv-host=_etcd-server._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-server._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-server._tcp.k8s.com,etcd3.k8s.com,2380,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd1.k8s.com,2379,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd2.k8s.com,2379,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd3.k8s.com,2379,0,100

修改之后重啟服務 systemctl restart dnsmasq

驗證SRV解析是否正常

查詢SRV記錄

[root@node01 ~]# dig @192.168.1.122 +noall +answer SRV _etcd-server._tcp.k8s.com
_etcd-server._tcp.k8s.com. 3600	IN	SRV	2380 0 100 etcd2.k8s.com.
_etcd-server._tcp.k8s.com. 3600	IN	SRV	2380 0 100 etcd1.k8s.com.
_etcd-server._tcp.k8s.com. 3600	IN	SRV	2380 0 100 etcd3.k8s.com.

查詢域名解析結果

[root@node01 ~]# dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com
etcd1.k8s.com.		86400	IN	A	192.168.1.91
etcd2.k8s.com.		86400	IN	A	192.168.1.92
etcd3.k8s.com.		86400	IN	A	192.168.1.93

如上述顯示，則表示SRV解析正常

配置ETCD

node01 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.91:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.91:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd1.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://etcd1.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node02 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.92:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.92:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd2"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd2.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://etcd2.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node03 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="http://192.168.1.93:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.93:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd3"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd3.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://etcd3.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

啟動並測試

啟動

[root@node01 etcd]# systemctl start etcd
[root@node01 etcd]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-07 11:25:29 CST; 4s ago
 Main PID: 14203 (etcd)
    Tasks: 8
   Memory: 16.9M
   CGroup: /system.slice/etcd.service
           └─14203 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=http://192.168.1.91:2379

Nov 07 11:25:29 node01.k8s.com etcd[14203]: d79e9ae86b2a1de1 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
Nov 07 11:25:29 node01.k8s.com etcd[14203]: d79e9ae86b2a1de1 became leader at term 2
Nov 07 11:25:29 node01.k8s.com etcd[14203]: raft.node: d79e9ae86b2a1de1 elected leader d79e9ae86b2a1de1 at term 2
Nov 07 11:25:29 node01.k8s.com etcd[14203]: published {Name:etcd1 ClientURLs:[http://etcd1.k8s.com:2379 http://etcd1.k8s.com:4001]} to cluster 42cecf80e3791d6c
Nov 07 11:25:29 node01.k8s.com etcd[14203]: ready to serve client requests
Nov 07 11:25:29 node01.k8s.com etcd[14203]: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!
Nov 07 11:25:29 node01.k8s.com systemd[1]: Started Etcd Server.
Nov 07 11:25:29 node01.k8s.com etcd[14203]: setting up the initial cluster version to 3.3
Nov 07 11:25:29 node01.k8s.com etcd[14203]: set the initial cluster version to 3.3
Nov 07 11:25:29 node01.k8s.com etcd[14203]: enabled capabilities for version 3.3

日志 vim /var/log/messages 表現如下:

Nov  7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at 0=http://etcd3.k8s.com:2380
Nov  7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at 1=http://etcd2.k8s.com:2380
Nov  7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at etcd1=http://etcd1.k8s.com:2380
Nov  7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380
Nov  7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380
Nov  7 11:25:28 node01 etcd: name = etcd1
Nov  7 11:25:28 node01 etcd: data dir = /data/k8s/etcd/data
Nov  7 11:25:28 node01 etcd: member dir = /data/k8s/etcd/data/member
Nov  7 11:25:28 node01 etcd: dedicated WAL dir = /data/k8s/etcd/wal
Nov  7 11:25:28 node01 etcd: heartbeat = 100ms
Nov  7 11:25:28 node01 etcd: election = 1000ms
Nov  7 11:25:28 node01 etcd: snapshot count = 100000
Nov  7 11:25:28 node01 etcd: advertise client URLs = http://etcd1.k8s.com:2379,http://etcd1.k8s.com:4001
Nov  7 11:25:28 node01 etcd: initial advertise peer URLs = http://etcd1.k8s.com:2380
Nov  7 11:25:28 node01 etcd: initial cluster = 0=http://etcd3.k8s.com:2380,1=http://etcd2.k8s.com:2380,etcd1=http://etcd1.k8s.com:2380

測試：

[root@node01 etcd]# etcdctl --endpoints=http://192.168.1.91:2379 cluster-health
member 184beca37ca32d75 is healthy: got healthy result from http://etcd2.k8s.com:2379
member d79e9ae86b2a1de1 is healthy: got healthy result from http://etcd1.k8s.com:2379
member f7662e609b7e4013 is healthy: got healthy result from http://etcd3.k8s.com:2379
cluster is healthy

ETCD TLS動態集群基於DNS的SRV解析自動發現

需要局域網內部有DNS服務器

添加SRV解析

目前常用的內部DNS服務有兩種，bind、dnsmasq

在下面都會列出具體的配置，但只需要配置其中之一即可；

方法一: 使用`bind`配置SRV解析

如果內部沒有bind服務，可以參考部署文檔文章: https://www.cnblogs.com/winstom/p/11806962.html

使用域名為 : k8s.com，在bind的zone文件中添加如下解析:

etcd1   IN      A       192.168.1.91
etcd2   IN      A       192.168.1.92
etcd3   IN      A       192.168.1.93
_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd1
_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd2
_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd3
_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd1
_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd2
_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd3

修改之后重新加載配置文件:

[root@jenkins named]# named-checkzone k8s.com k8s.com.zone 
zone k8s.com/IN: loaded serial 0
OK
[root@jenkins named]# rndc reload
server reload successful

方法二: 使用`dnsmasq`配置SRV解析

如果內部沒有dnsmasq服務，可以參考部署文檔文章: https://www.cnblogs.com/winstom/p/11809066.html

使用域名為 : k8s.com，具體修改如下:

在/etc/dnsmasq_hosts新增下面內容

192.168.1.91 etcd1 etcd1.k8s.com
192.168.1.92 etcd2 etcd2.k8s.com
192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析內容

srv-host=_etcd-server-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd1.k8s.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd2.k8s.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd3.k8s.com,2379,0,100

修改之后重啟服務 systemctl restart dnsmasq

驗證SRV解析是否正常

查詢SRV記錄

[root@node01 etcd]# dig @192.168.1.122 +noall +answer SRV _etcd-server-ssl._tcp.k8s.com
_etcd-server-ssl._tcp.k8s.com. 3600 IN	SRV	2380 0 100 etcd3.k8s.com.
_etcd-server-ssl._tcp.k8s.com. 3600 IN	SRV	2380 0 100 etcd2.k8s.com.
_etcd-server-ssl._tcp.k8s.com. 3600 IN	SRV	2380 0 100 etcd1.k8s.com.

查詢域名解析結果

[root@node01 ~]# dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com
etcd1.k8s.com.		86400	IN	A	192.168.1.91
etcd2.k8s.com.		86400	IN	A	192.168.1.92
etcd3.k8s.com.		86400	IN	A	192.168.1.93

ETCD 配置

node01 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.91:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.91:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd1"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd1.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd1.k8s.com:2379,https://etcd1.k8s.com:4001"
ETCD_DISCOVERY_SRV="k8s.com"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"

node02 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.92:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.92:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd2"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd2.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd2.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"

node03 配置文件

ETCD_DATA_DIR="/data/k8s/etcd/data"
ETCD_WAL_DIR="/data/k8s/etcd/wal"
ETCD_LISTEN_PEER_URLS="https://192.168.1.93:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.93:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
ETCD_NAME="etcd3"
ETCD_SNAPSHOT_COUNT="100000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd3.k8s.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd3.k8s.com:2379"
ETCD_DISCOVERY_SRV="k8s.com"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"
ETCD_PEER_AUTO_TLS="true"

啟動測試

啟動

[root@node03 etcd]# systemctl restart etcd
[root@node03 etcd]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-11-07 12:38:37 CST; 4s ago
 Main PID: 13460 (etcd)
    Tasks: 8
   Memory: 16.6M
   CGroup: /system.slice/etcd.service
           └─13460 /usr/bin/etcd --name=etcd3 --data-dir=/data/k8s/etcd/data --listen-client-urls=https://192.168.1.93:2379

Nov 07 12:38:36 node03.k8s.com etcd[13460]: established a TCP streaming connection with peer 40a8f19a5db99534 (stream Message writer)
Nov 07 12:38:36 node03.k8s.com etcd[13460]: established a TCP streaming connection with peer 40a8f19a5db99534 (stream MsgApp v2 writer)
Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 [term: 92] received a MsgVote message with higher term from a0d541999e9eb3b3 [term: 98]
Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 became follower at term 98
Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 [logterm: 92, index: 9, vote: 0] cast MsgVote for a0d541999e9eb3b3 [logterm: 92, index: 9] at term 98
Nov 07 12:38:37 node03.k8s.com etcd[13460]: raft.node: 9888555207dbf0e0 elected leader a0d541999e9eb3b3 at term 98
Nov 07 12:38:37 node03.k8s.com etcd[13460]: published {Name:etcd3 ClientURLs:[https://etcd3.k8s.com:2379]} to cluster f445a02ce3dc6a02
Nov 07 12:38:37 node03.k8s.com etcd[13460]: ready to serve client requests
Nov 07 12:38:37 node03.k8s.com etcd[13460]: serving client requests on 192.168.1.93:2379
Nov 07 12:38:37 node03.k8s.com systemd[1]: Started Etcd Server.

日志體現

Nov  7 12:38:36 node01 etcd: added member 40a8f19a5db99534 [https://etcd2.k8s.com:2380] to cluster f445a02ce3dc6a02
Nov  7 12:38:36 node01 etcd: starting peer 40a8f19a5db99534...
Nov  7 12:38:36 node01 etcd: started HTTP pipelining with peer 40a8f19a5db99534
Nov  7 12:38:36 node01 etcd: started streaming with peer 40a8f19a5db99534 (writer)
Nov  7 12:38:36 node01 etcd: started peer 40a8f19a5db99534
Nov  7 12:38:36 node01 etcd: added peer 40a8f19a5db99534
Nov  7 12:38:36 node01 etcd: added member 9888555207dbf0e0 [https://etcd3.k8s.com:2380] to cluster f445a02ce3dc6a02
Nov  7 12:38:36 node01 etcd: starting peer 9888555207dbf0e0...
Nov  7 12:38:36 node01 etcd: started HTTP pipelining with peer 9888555207dbf0e0
Nov  7 12:38:36 node01 etcd: started peer 9888555207dbf0e0
Nov  7 12:38:36 node01 etcd: added peer 9888555207dbf0e0
Nov  7 12:38:36 node01 etcd: added member a0d541999e9eb3b3 [https://etcd1.k8s.com:2380] to cluster f445a02ce3dc6a02

測試集群狀態：

ETCDCTL_API=3 etcdctl --endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \
    --cacert=/etc/kubernetes/cert/ca.pem \
    --cert=/etc/etcd/cert/etcd.pem \
    --key=/etc/etcd/cert/etcd-key.pem endpoint health

# 輸出
https://etcd1.k8s.com:2379 is healthy: successfully committed proposal: took = 4.269468ms
https://etcd3.k8s.com:2379 is healthy: successfully committed proposal: took = 1.58797ms
https://etcd2.k8s.com:2379 is healthy: successfully committed proposal: took = 1.622151ms

etcdctl \
--endpoint=https://etcd1.k8s.com:2379 \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/etcd/cert/etcd.pem \
--key-file=/etc/etcd/cert/etcd-key.pem cluster-health

# 輸出
member 40a8f19a5db99534 is healthy: got healthy result from https://etcd2.k8s.com:2379
member 9888555207dbf0e0 is healthy: got healthy result from https://etcd3.k8s.com:2379
member a0d541999e9eb3b3 is healthy: got healthy result from https://etcd1.k8s.com:2379
cluster is healthy

報錯解決

1. 證書報錯 bad certificate

日志中報錯:

Nov  7 12:37:03 node01 etcd: rejected connection from "192.168.1.93:46294" (error "remote error: tls: bad certificate", ServerName "k8s.com")

解決

報錯的意思是在生成ETCD的TLS證書的時候，沒有把對應的域名加進去

在創建ETCD的TLS證書請求的文件中加入對應的域名

cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.1.91",
    "192.168.1.92",
    "192.168.1.93",
    "k8s.com",          # 這里的域名查看是否正確
    "etcd1.k8s.com",
    "etcd2.k8s.com",
    "etcd3.k8s.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

2. `DNS` 的 `SRV` 解析報錯 cannot find local etcd member "etcd1" in SRV records

DNS 如果配置有問題，會有如下報錯:

etcd: error setting up initial cluster: cannot find local etcd member "etcd1" in SRV records

這里是表示DNS在配置SRV解析的時候報錯，請仔細查看解析配置:

SRV解析分為兩種，一種是http不帶證書的解析，一種是https帶證書的解析，是有區別的，如果配置錯誤就會包上述錯誤

http不帶證書解析如下

bind 的解析

編輯 /var/named/k8s.com.zone 文件

etcd1   IN      A       192.168.1.91
etcd2   IN      A       192.168.1.92
etcd3   IN      A       192.168.1.93
_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd1
_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd2
_etcd-server._tcp.k8s.com.  IN      SRV     10    10       2380     etcd3
_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd1
_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd2
_etcd-client._tcp.k8s.com.  IN      SRV     10    10       2379     etcd3

dnsmasq 的解析

在/etc/dnsmasq_hosts新增下面內容

192.168.1.91 etcd1 etcd1.k8s.com
192.168.1.92 etcd2 etcd2.k8s.com
192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析內容

srv-host=_etcd-server._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-server._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-server._tcp.k8s.com,etcd3.k8s.com,2380,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-client._tcp.k8s.com,etcd3.k8s.com,2380,0,100

https帶證書解析如下

bind 的解析

編輯 /var/named/k8s.com.zone 文件

etcd1   IN      A       192.168.1.91
etcd2   IN      A       192.168.1.92
etcd3   IN      A       192.168.1.93
_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd1
_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd2
_etcd-server-ssl._tcp.k8s.com.  IN      SRV     10    10       2380     etcd3
_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd1
_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd2
_etcd-client-ssl._tcp.k8s.com.  IN      SRV     10    10       2379     etcd3

dnsmasq 的解析

在/etc/dnsmasq_hosts新增下面內容

192.168.1.91 etcd1 etcd1.k8s.com
192.168.1.92 etcd2 etcd2.k8s.com
192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析內容

srv-host=_etcd-server-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100
srv-host=_etcd-server-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd1.k8s.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd2.k8s.com,2379,0,100
srv-host=_etcd-client-ssl._tcp.k8s.com,etcd3.k8s.com,2379,0,100

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 etcd集群配置 centos下etcd集群安裝 CentOS下安裝etcd集群 centos8 搭建etcd集群配置安全證書的Etcd集群 Kubernetes集群搭建之Etcd集群配置篇 Etcd v3.1.15 在Linux CentOS7 集群配置（單機改端口實現） centos7部署etcd集群 kubeadm配置高可用etcd集群 etcd集群

CentOS 7 ETCD集群配置大全

前言

環境准備

安裝

靜態集群

配置

node01 配置文件

node02 配置文件

node03 配置文件

啟動測試

查看集群狀態

生成TLS證書

etcd證書創建

安裝cfssl工具集

生成證書

分發證書到各節點上

靜態TLS集群

etcd 配置

node01 配置文件

node02 配置文件

node03 配置文件

啟動測試

檢查TLS集群狀態

ETCD 動態集群基於DNS的SRV解析自動發現

添加SRV解析

方法一: 使用bind配置SRV解析

方法二: 使用dnsmasq配置SRV解析

驗證SRV解析是否正常

配置ETCD

node01 配置文件

node02 配置文件

node03 配置文件

啟動並測試

ETCD TLS動態集群基於DNS的SRV解析自動發現

添加SRV解析

方法一: 使用bind配置SRV解析

方法二: 使用dnsmasq配置SRV解析

驗證SRV解析是否正常

ETCD 配置

node01 配置文件

node02 配置文件

node03 配置文件

啟動測試

報錯解決

1. 證書報錯 bad certificate

解決

2. DNS 的 SRV 解析報錯 cannot find local etcd member "etcd1" in SRV records

http不帶證書解析如下

bind 的解析

dnsmasq 的解析

https帶證書解析如下

bind 的解析

dnsmasq 的解析

免責聲明！

方法一: 使用`bind`配置SRV解析

方法二: 使用`dnsmasq`配置SRV解析

方法一: 使用`bind`配置SRV解析

方法二: 使用`dnsmasq`配置SRV解析

2. `DNS` 的 `SRV` 解析報錯 cannot find local etcd member "etcd1" in SRV records