本篇安裝單個etcd，然后進行擴容etcd節點至2個，環境配置如果做了的話就跳過 實驗架構 test1: 192.168.0.91 etcd test2: 192.168.0.92 無 test3: 192.168.0.93 無 1、環境配置 # 如下操作在所有節點操作 修改主機名 # 注意修改 各自節點對應的 主機名 sed -i '$a\hostname=test1' /etc/hostname sed -i '$a\hostname=test1' /etc/sysconfig/network && hostnamectl set-hostname test1 配置hosts解析 cat >>/etc/hosts<<EOF 192.168.0.91 test1 192.168.0.92 test2 192.168.0.93 test3 EOF 禁用selinux sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux sed -i 's/enforcing/disabled/g' /etc/selinux/config 關閉swap # 注釋/etc/fstab文件里swap相關的行 sed -i 's/\/dev\/mapper\/centos-swap/#\/dev\/mapper\/centos-swap/g' /etc/fstab 關掉防火牆 systemctl stop firewalld && systemctl disable firewalld 退出xshell重新登錄，查看主機名 開啟forward iptables -P FORWARD ACCEPT 配置轉發相關參數 cat >> /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 vm.swappiness=0 EOF 加載系統參數 sysctl --system 加載ipvs相關內核模塊 # 如果重新開機，需要重新加載 modprobe ip_vs modprobe ip_vs_rr modprobe ip_vs_wrr modprobe ip_vs_sh modprobe nf_conntrack_ipv4 lsmod | grep ip_vs 安裝etcd 下面幾步都在test1 節點操作 下載安裝包 useradd etcd mkdir -p /server/software/k8s mkdir -p /opt/k8s/bin cd /server/software/k8s wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz tar -xf etcd-v3.2.18-linux-amd64.tar.gz mv etcd-v3.2.18-linux-amd64/etcd* /opt/k8s/bin chmod +x /opt/k8s/bin/* ln -s /opt/k8s/bin/etcd /usr/bin/etcd ln -s /opt/k8s/bin/etcdctl /usr/bin/etcdctl etcd --version 2、安裝CFSSL證書生成工具 只在test1節點操作 mkdir -pv /server/software/k8s cd /server/software/k8s wget下載cfssl工具 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 安裝cfssl工具 # 只要把安裝包改下名字，移動到usr/local/bin/下，加上授權即可 mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson chmod +x /usr/local/bin/cfssl* 3、創建PKI配置文件 # 只 在test1節點操作 # 作用：生成其他組件ca證書時需要用到(除了根證書)CA 配置文件 mkdir -p $HOME/ssl && cd $HOME/ssl cat >ca-config.json<<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF 注意：PKI配置文件中的profiles中同時定義了 server、clietns，表明使用這個PKI創建的證書既可以作為服務器驗證用，也可以作為客戶端驗證用 這里對PKI安全認證不做過多解釋， PKI安全認證請參照：https://www.cnblogs.com/effortsing/p/10332492.html 4、生成 ca 根證書 # 只在test1節點操作 # ca 證書作用：生成其他組件證書時需要用到根證書 cd $HOME/ssl cat >ca-csr.json<<EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ], "ca": { "expiry": "87600h" } } EOF 生成證書 cfssl gencert -initca ca-csr.json | cfssljson -bare ca 查看生成的證書 [root@test1 ssl]# ls ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem 5、添加證書到受信任列表（選做） # 在 test1 節點操作 # 添加ca證書到linux系統受信任列表,這樣在執行命令的時候就不用帶上證書路徑了。 # 添加信任后： etcdctl cluster-health = etcdctl cluster-health /etc/kubernetes/cert/ca.pem ，就是省了個證書 # 如果沒有添加ca證書到linux系統受信任列表，后面執行etcdctl cluster-health 會報如下錯誤。 cat ca.pem >> /etc/pki/tls/certs/ca-bundle.crt 6、管理證書 # 把根證書和私鑰復制到一個目錄里面 mkdir -p /etc/kubernetes/cert/ cp ca*.pem /etc/kubernetes/cert/ chmod 777 /etc/kubernetes/* 5、生成etcd的ca證書和私鑰 # 只在test1節點上操作 cd $HOME/ssl cat >etcd-csr.json<<EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.0.92", "192.168.0.93", "192.168.0.91" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "etcd", "OU": "Etcd Security" } ] } EOF 生成證書 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd 查看生成的證書和私鑰 [root@test1 ssl]# ls ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem etcd.csr etcd-csr.json etcd-key.pem etcd.pem 6、添加證書到受信任列表（選做） # 在 test1 節點操作 # 添加ca證書到linux系統受信任列表 cat etcd.pem >> /etc/pki/tls/certs/ca-bundle.crt 7、管理證書 把etcd證書復制到一個目錄里面 mkdir -p /etc/etcd/cert/ cp etcd*.pem /etc/etcd/cert/ chmod 777 /etc/etcd/cert/* 8、啟動etcd 8.1、 配置etcd啟動腳本 # 配置 環境變量 cat >> /etc/profile << EOF export ETCD_NAME=$(hostname) export INTERNAL_IP=$(hostname -i | awk '{print $NF}') export ECTD_CLUSTER='test1=https://192.168.0.91:2380' EOF source /etc/profile 8.2、配置啟動文件 本文配置文件開啟了集群外部服務端、客戶端、認證，以及集群內部之間服務端、客戶端認證。所以客戶端etcdctl訪問時候需要帶上客戶端證書 mkdir -p /data/etcd cat> /etc/systemd/system/etcd.service<< EOF [Service] Type=notify WorkingDirectory=/data/etcd EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/opt/k8s/bin/etcd \\ --name ${ETCD_NAME} \\ --cert-file=/etc/etcd/cert/etcd.pem \\ --key-file=/etc/etcd/cert/etcd-key.pem \\ --peer-cert-file=/etc/etcd/cert/etcd.pem \\ --peer-key-file=/etc/etcd/cert/etcd-key.pem \\ --trusted-ca-file=/etc/kubernetes/cert/ca.pem \\ --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\ --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\ --listen-peer-urls https://${INTERNAL_IP}:2380 \\ --listen-client-urls https://${INTERNAL_IP}:2379,http://127.0.0.1:2379 \\ --advertise-client-urls https://${INTERNAL_IP}:2379 \\ --initial-cluster-token my-etcd-token \\ --initial-cluster $ECTD_CLUSTER \\ --initial-cluster-state new \\ --data-dir=/data/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF 8.3、啟動etctd systemctl daemon-reload #一定要執行，否則報錯 systemctl start etcd systemctl status etcd systemctl enable etcd 9、查看集群成員和安全狀態 必須得帶上證書，涉及到服務端、客戶端認證 [root@test1 ~]# etcdctl --ca-file /etc/kubernetes/cert/ca.pem --cert-file /etc/etcd/cert/etcd.pem --key-file /etc/etcd/cert/etcd-key.pem member list 42f7141ed6110de1: name=test1 peerURLs=https://192.168.0.91:2380 clientURLs=https://192.168.0.91:2379 isLeader=true [root@test1 ~]# etcdctl --ca-file /etc/kubernetes/cert/ca.pem --cert-file /etc/etcd/cert/etcd.pem --key-file /etc/etcd/cert/etcd-key.pem cluster-health member 42f7141ed6110de1 is healthy: got healthy result from https://192.168.0.91:2379 cluster is healthy 可以看到peerURLs已經是https模式了，由於test1節點是新建的集群，所以屬於重建集群開啟pki安全認證； 這里對pki安全認證不多做解釋，具體請參照：https://www.cnblogs.com/effortsing/p/10332492.html 報錯解決： 刪除etcd數據目錄重新啟動 參照文檔： http://www.maogx.win/posts/35/ http://www.maogx.win/ https://juejin.im/user/59ffa2836fb9a0451c39c64f/posts https://blog.csdn.net/fy573060627/article/details/52872740