Kubernetes二進制安裝

Master：192.168.11.220

Node1：192.168.11.221

Node2：192.168.11.222

 

一、創建集群所需要的CA證書和秘鑰

為確保安全，kubernetes 系統各組件需要使用 x509 證書對通信進行加密和認證。CA (Certificate Authority) 是自簽名的根證書，用來簽名后續創建的其它證書。這里使用 CloudFlare 的 PKI 工具集 cfssl 創建所有證書。

1)#安裝cfssl工具集
[root@k8s-master ]# mkdir -p /opt/k8s/work && cd /opt/k8s/work
[root@k8s-master work]# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
[root@k8s-master work]# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
[root@k8s-master work]# curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
[root@k8s-master work]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

2)#創建根證書(CA)
CA證書是集群所有節點共享的，只需要創建一個CA證書，后續創建的所有證書都由它簽名。
2.1)創建配置文件
CA配置文件用於配置根證書的使用場景(profile)和具體參數(usage，過期時間、服務端認證、客戶端認證、加密等)，后續在簽名其他證書時需要制定特定場景。
[root@k8s-master work]# cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

配置說明：
signing：表示該證書可用於簽名其它證書，生成的ca.pem證書中CA=TRUE；
server auth：表示clent可以用該證書對server提供的證書進行驗證；
client auth：表示server可以用該證書對client提供的證書進行驗證；

2.2)創建證書簽名請求文件
[root@k8s-master work]# cat > ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GuangDong",
      "L": "GuangDong",
    }
  ]
}
EOF

配置說明：
CN:Common Name，kube-apiserver從證書中提取該字段作為請求的用戶名(User Name),瀏覽器使用該字段驗證網站是否合法

2.3）生成CA證書和私鑰
[root@k8s-master work]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@k8s-master work]# ls ca*
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
[root@k8s-master work]# mkdir -p /etc/kubernetes/cert
[root@k8s-master work]# cp ca*.pem ca-config.json /etc/kubernetes/cert

#將證書和私鑰下發到所有節點
[root@k8s-master work]# for node_all_ip in 192.168.11.221 192.168.11.222
do
   echo ">>> ${node_all_ip}"
   ssh root@${node_all_ip} "mkdir -p /etc/kubernetes/cert"
   scp ca*.pem ca-config.json root@${node_all_ip}:/etc/kubernetes/cert
done

 

 二、部署etcd集群

 

1)下載etcd二進制文件
[root@k8s-master work]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz
[root@k8s-master work]# tar -xvf etcd-v3.4.3-linux-amd64.tar.gz
[root@k8s-master work]# mkdir -p /opt/k8s/bin
[root@k8s-master work]# cp etcd-v3.4.3-linux-amd64/etcd* /opt/k8s/bin/
[root@k8s-master work]# chmod +x /opt/k8s/bin/*
[root@k8s-master work]# for node_all_ip in 192.168.11.221 192.168.11.222
do
   echo ">>> ${node_all_ip}"
   ssh root@${node_all_ip} "mkdir -p /opt/k8s/bin"
   scp etcd-v3.4.3-linux-amd64/etcd* root@${node_all_ip}:/opt/k8s/bin/
   ssh root@${node_all_ip} "chmod +x /opt/k8s/bin/*"
done

2)創建etcd證書和私鑰
創建證書簽名請求：
[root@k8s-master work]# cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "192.168.11.220",
    "192.168.11.221",
    "192.168.11.222"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GuangDong",
      "L": "GuangDong"
    }
  ]
}
EOF

配置說明：
hosts:制定授權使用該證書的etcd節點IP或域名列表，需要將etcd集群的三個節點IP都列在其中；

2.1)生成證書和私鑰
[root@k8s-master work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \
   -ca-key=/opt/k8s/work/ca-key.pem \
   -config=/opt/k8s/work/ca-config.json \
   -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
[root@k8s-master work]# ls etcd*pem
etcd-key.pem  etcd.pem

[root@k8s-master work]# mkdir -p /etc/etcd/cert
[root@k8s-master work]# cp etcd*.pem /etc/etcd/cert/
[root@k8s-master work]# for node_all_ip in 192.168.11.221 192.168.11.222
do
   echo ">>> ${node_all_ip}"
   ssh root@${node_all_ip} "mkdir -p /etc/etcd/cert"
   scp etcd*.pem root@${node_all_ip}:/etc/etcd/cert/
done

#創建etcd的systemd unit模板腳本
[root@k8s-master work]# cat etcd.sh
#!/bin/bash
#example: ./etcd.sh etcd01 192.168.11.220 etcd01=https://192.168.11.220:2380,etcd02=https://192.168.11.221:2380,etcd03=https://192.168.11.222:2380

NODE_ETCD_NAME=$1
NODE_ETCD_IP=$2
ETCD_NODES=$3
ETCD_DATA_DIR=/data/k8s/etcd/data
ETCD_WAL_DIR=/data/k8s/etcd/wal

if [ ! -d "/data/k8s/etcd/data /data/k8s/etcd/wal" ];then
  mkdir -p /data/k8s/etcd/data /data/k8s/etcd/wal
fi

cat > /etc/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
   
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \\
  --data-dir=${ETCD_DATA_DIR} \\
  --wal-dir=${ETCD_WAL_DIR} \\
  --name=${NODE_ETCD_NAME} \\
  --cert-file=/etc/etcd/cert/etcd.pem \\
  --key-file=/etc/etcd/cert/etcd-key.pem \\
  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-cert-file=/etc/etcd/cert/etcd.pem \\
  --peer-key-file=/etc/etcd/cert/etcd-key.pem \\
  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --listen-peer-urls=https://${NODE_ETCD_IP}:2380 \\
  --initial-advertise-peer-urls=https://${NODE_ETCD_IP}:2380 \\
  --listen-client-urls=https://${NODE_ETCD_IP}:2379,http://127.0.0.1:2379 \\
  --advertise-client-urls=https://${NODE_ETCD_IP}:2379 \\
  --initial-cluster-token=etcd-cluster-0 \\
  --initial-cluster=${ETCD_NODES} \\
  --initial-cluster-state=new \\
  --auto-compaction-mode=periodic \\
  --auto-compaction-retention=1 \\
  --max-request-bytes=33554432 \\
  --quota-backend-bytes=6442450944 \\
  --heartbeat-interval=250 \\
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
   
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

#啟動etcd，此時會卡住不動，是因為etcd需要選舉才能正常運行，所以要在另外兩個節點也執行以下命令，記得修改$1和$2參數
[root@k8s-master work]# ./etcd.sh etcd01 192.168.11.220 etcd01=https://192.168.11.220:2380,etcd02=https://192.168.11.221:2380,etcd03=https://192.168.11.222:2380
for node_all_ip in 192.168.11.221 192.168.11.222 
do
  echo ">>> ${node_all_ip}"
  scp /opt/k8s/work/etcd.sh root@${node_all_ip}:/opt/k8s/
done

node1: sh /opt/k8s/etcd.sh etcd02 192.168.11.221 etcd01=https://192.168.11.220:2380,etcd02=https://192.168.11.221:2380,etcd03=https://192.168.11.222:2380
node2: sh /opt/k8s/etcd.sh etcd03 192.168.11.222 etcd01=https://192.168.11.220:2380,etcd02=https://192.168.11.221:2380,etcd03=https://192.168.11.222:2380

#當所有節點都執行完成后檢查，狀態是否都正常
[root@k8s-master work]# ETCDCTL_API=3 /opt/k8s/bin/etcdctl --endpoints="https://192.168.11.220:2379,https://192.168.11.221:2379,https://192.168.11.222:2379" \
 --cacert=/etc/kubernetes/cert/ca.pem \
 --cert=/etc/etcd/cert/etcd.pem \
 --key=/etc/etcd/cert/etcd-key.pem endpoint health
輸出內容：
https://192.168.11.221:2379 is healthy: successfully committed proposal: took = 25.077164ms
https://192.168.11.220:2379 is healthy: successfully committed proposal: took = 38.10606ms
https://192.168.11.222:2379 is healthy: successfully committed proposal: took = 38.785388ms

#查看當前etcd集群工中的leader
[root@k8s-master work]# for node_all_ip in 192.168.11.220 192.168.11.221 192.168.11.222
do
ETCDCTL_API=3 /opt/k8s/bin/etcdctl -w table --cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/etcd/cert/etcd.pem --key=/etc/etcd/cert/etcd-key.pem \
--endpoints=https://${node_all_ip}:2379 endpoint status
done
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.11.220:2379 | a6bbfb193c776e5c |   3.4.3 |   25 kB |      true |      false |       458 |         21 |                 21 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.11.221:2379 | 7b37d21aaf69f7d2 |   3.4.3 |   20 kB |     false |      false |       458 |         21 |                 21 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.11.222:2379 | 711ad351fe31c699 |   3.4.3 |   20 kB |     false |      false |       458 |         21 |                 21 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
由上面結果可見，當前的leader節點為192.168.11.220

#如果報錯：請檢查所有服務器時間是否同步
rejected connection from "192.168.11.220:58360" (error "tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid", ServerName "")
#如果遇到ETCD出現連接失敗狀況，導致創建實例失敗，則etcd啟動文件中的 --initial-cluster-state=new 改為 existing，重啟則正常