一 證書分類
服務器證書:
server cert :用戶客戶端驗證服務端的身份。
客戶端證書:
client cert:用戶服務端驗證客戶端的身份。
對等證書:
peer cert: 用戶成員之間的身份驗證,例如etcd,(該證書,它即是server cert,又是client cert)。
二 k8s集群證書的分類
ETCD:
需要server cert標識自己,也需要client cert與其它etcd集群通訊,因此需要一個peer cert(對等證書)。
master節點:
需要標識api-server的server cert,也需要client cert連接到etcd集群,還需要一個對等證書peer跟client交互。
kubelet:
需要標識服務kubelet的server cert,也需要client cert 去請求api-server,因此還需要一個對等證書peer cert。
kubelet,kube-proxy,calico:
需要client cert
三 生成的 CA 證書和秘鑰文件
目錄位置:/etc/kubernetes/ssl
使用證書的組件
CA 證書說明講解
可信任證書頒發機構
例如:我們的身份證,它為什么大家都認可它信任它,因為它是公安局頒發的所有被信任,這里的CA機構就相當於公安局。
CA證書結構:
ca-config.json (CA配置文件)
{
"signing": { #表示該證書可用於簽名其它證書;生成的ca.pem證書中CA=TRUE。
"default": {
"expiry": "8760h" #默認證書過期時間(小時,用戶可自己調大過期時間)。
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth", #表示client可以用該CA對server提供的證書進行驗證。
"client auth" #表示server可以用該CA對client提供的證書進行驗證。
],
"expiry": "8760h" #默認證書過期時間(小時,用戶可自己調大過期時間)。
}
}
}
}
ca-csr.json(證書簽名請求)
{
"CN": "kubernetes", #CN-Common Name:kube-apiserver從證書中提取該字段作為請求的用戶名 (User Name);瀏覽器使用該字段驗證網站是否合法
"key": {
"algo": "rsa", #加密算法
"size": 2048 #加密的位數,您可以不改也可以加大
},
"names": [
{
"C": "CN", #國家
"ST": "BeiJing", #城市
"L": "BeiJing", #組織
"O": "k8s", #O-Organization:kube-apiserver從證書中提取該字段作為請求用戶所屬的組 (Group);
"OU": "System" #組織單元
}
]
}
四 證書生成操作
4.1 證書頒發工具
安裝cfssl
這里我們選擇CloudFlare的PKI工具集cfssl ,它比OpenSSL要簡單
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
export PATH=/usr/local/bin:$PATH
4.2 創建CA(Certificate Authority)
4.2.1 創建ca配置文件:(CA證書的說明可參考上面CA證書講解部分)
mkdir /root/ssl
cd /root/ssl
cfssl print-defaults config > config.json
cfssl print-defaults csr > csr.json
# 根據config.json文件的格式創建如下的ca-config.json文件
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
4.2.2 創建CA證書簽名請求文件
cat > ca-csr.json <<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
],
"ca": {
"expiry": "87600h"
}
}
EOF
4.2.3 生成CA證書文和私鑰
$cfssl gencert -initca ca-csr.json | cfssljson -bare ca
$ll
total 20
drwxr-xr-x 2 root root 93 Aug 3 10:24 ./
drwxr-xr-x 5 root root 60 Aug 3 09:11 ../
-rw-r--r-- 1 root root 292 Aug 3 10:23 ca-config.json
-rw-r--r-- 1 root root 1001 Aug 3 10:24 ca.csr #證書簽名請求文件
-rw-r--r-- 1 root root 253 Aug 3 10:23 ca-csr.json
-rw------- 1 root root 1679 Aug 3 10:24 ca-key.pem #證書私鑰
-rw-r--r-- 1 root root 1359 Aug 3 10:24 ca.pem #證書
4.3 創建kubernetes證書
4.3.1 創建kubernetes證書簽名請求文件kubernetes-csr.json
注意:
. 如果 hosts 字段不為空則需要指定授權使用該證書的 IP 或域名列表,由於該證書后續被 etcd 集群和 kubernetes master 集群使用,所以上面分別指定了 etcd 集群kubernetes master 集群的主機 IP 和 kubernetes 服務的服務 IP(一般是 kube-apiserver 指定的 service-cluster-ip-range 網段的第一個IP,如 10.254.0.1)。
. 以下物理節點的IP也可以更換為主機名
cat > kubernetes-csr.json <<EOF
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"172.20.0.112",
"172.20.0.113",
"172.20.0.114",
"172.20.0.115",
"10.254.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
4.3.2 生成kubernetes證書和秘鑰
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
$ ll kubernetes*
-rw-r--r-- 1 root root 1269 Aug 3 10:36 kubernetes.csr #證書簽名請求文件
-rw-r--r-- 1 root root 578 Aug 3 10:36 kubernetes-csr.json
-rw------- 1 root root 1679 Aug 3 10:36 kubernetes-key.pem #證書私鑰
-rw-r--r-- 1 root root 1635 Aug 3 10:36 kubernetes.pem #證書
4.4 創建Admin證書
4.4.1 創建admin證書簽名請求文件admin-csr.json
注意:
. kube-apiserver 使用 RBAC 對客戶端(如 kubelet、kube-proxy、Pod)請求進行授權;
. kube-apiserver 預定義了一些 RBAC 使用的 RoleBindings,如 cluster-admin 將 Group system:masters 與 Role cluster-admin 綁定,該 Role 授予了調用kube-apiserver 的所有 API的權限;
. O 指定該證書的 Group 為 system:masters,kubelet 使用該證書訪問 kube-apiserver 時 ,由於證書被 CA 簽名,所以認證通過,同時由於證書用戶組為經過預授權的 system:masters,所以被授予訪問所有 API 的權限;
注意證書權限的關系:
. 這個admin 證書,是將來生成管理員用的kube config 配置文件用的,現在我們一般建議使用RBAC 來對kubernetes 進行角色權限控制, kubernetes 將證書中的CN 字段 作為User, O 字段作為 Group。
. 安裝完集群后您可以查看該admin用戶的集群權限,以下命令可查看,查看到 clusterrolebinding cluster-admin 的 subjects 的 kind 是 Group,name 是 system:masters。 roleRef 對象是 ClusterRole cluster-admin。 意思是凡是 system:masters Group的 user 或者 serviceAccount 都擁有 cluster-admin 的角色。 因此我們在使用 kubectl 命令時候,才擁有整個集群的管理權限:
$ kubectl get clusterrolebinding cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2017-04-11T11:20:42Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "52"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin
uid: e61b97b2-1ea8-11e7-8cd7-f4e9d49f8ed0
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:masters
4.4.1 創建admin證書簽名請求文件admin-csr.json
cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
4.4.2 生成admin證書和私鑰:
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
$ ll admin*
-rw-r--r-- 1 root root 1009 Aug 3 10:48 admin.csr
-rw-r--r-- 1 root root 229 Aug 3 10:45 admin-csr.json
-rw------- 1 root root 1675 Aug 3 10:48 admin-key.pem
-rw-r--r-- 1 root root 1399 Aug 3 10:48 admin.pem
4.5 創建kube-proxy證書
4.5.1 創建kube-proxy證書簽名請求文件kube-proxy-csr.json
注意:
. CN 指定該證書的 User 為 system:kube-proxy;
. kube-apiserver 預定義的 RoleBinding system:node-proxier 將User system:kube-proxy 與 Role system:node-proxier 綁定,該 Role 授予了調用 kube-apiserver Proxy 相關 API 的權限;
cat > kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
4.5.2 生成kube-proxy證書和秘鑰
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ll kube-proxy*
-rw-r--r-- 1 root root 1009 Aug 3 10:55 kube-proxy.csr
-rw-r--r-- 1 root root 230 Aug 3 10:54 kube-proxy-csr.json
-rw------- 1 root root 1679 Aug 3 10:55 kube-proxy-key.pem
-rw-r--r-- 1 root root 1403 Aug 3 10:55 kube-proxy.pem
五 校驗證書
如 kubernets證書
5.1 使用OpenSSL命令校驗:
openssl x509 -noout -text -in kubernetes.pem
檢查字段:
. 確認 Issuer 字段的內容和 ca-csr.json 一致;
. 確認 Subject 字段的內容和 kubernetes-csr.json 一致;
. 確認 X509v3 Subject Alternative Name 字段的內容和 kubernetes-csr.json 一致;
. 確認 X509v3 Key Usage、Extended Key Usage 字段的內容和 ca-config.json 中 kubernetes profile 一致;
$ openssl x509 -noout -text -in kubernetes.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
65:d7:41:64:d0:f7:62:78:3a:da:e3:1c:72:25:16:84:b7:bb:9f:8b
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = CN, ST = BeiJing, L = BeiJing, O = k8s, OU = System, CN = kubernetes
Validity
Not Before: Aug 3 02:31:00 2021 GMT
Not After : Aug 1 02:31:00 2031 GMT
Subject: C = CN, ST = BeiJing, L = BeiJing, O = k8s, OU = System, CN = kubernetes
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:d2:d8:2c:15:96:6d:97:03:1c:26:1a:17:06:38:
d5:7f:e1:4f:12:3a:f6:52:bd:4e:ae:0a:12:7a:ac:
c9:8a:23:38:2b:5c:3f:46:47:9f:95:15:7a:4d:69:
f3:0a:87:74:5c:28:2e:e5:6c:fe:55:79:51:d1:5a:
9e:ef:fd:a3:65:67:cf:66:f3:fb:d6:4a:29:98:11:
a8:8a:e7:9a:8f:05:bd:4c:23:e9:52:93:43:5a:09:
fc:39:48:ac:f9:04:15:a1:92:1e:af:4d:26:99:7b:
d7:ee:3c:b0:71:bc:a2:95:da:d4:78:dc:44:52:ad:
f3:47:a2:d2:4d:57:25:e5:56:59:ed:c5:dd:2c:5b:
d6:66:f1:7e:0b:4d:61:03:6a:8b:a0:df:54:71:f3:
48:c8:51:02:fc:4b:a2:4d:71:8c:41:af:0b:b7:f3:
61:09:a9:c4:49:7c:e0:68:d6:a6:ea:e3:bd:27:95:
2f:f7:a6:59:56:28:ea:b0:e2:b8:3e:31:74:97:ee:
04:08:70:ae:00:4f:21:a9:37:77:dc:fb:76:c6:78:
c5:fa:a3:46:20:b4:f6:5f:79:d1:16:fc:5f:64:c7:
57:68:34:49:3b:99:40:91:ce:2c:2d:de:fe:fc:fa:
83:5e:5e:73:6c:f1:46:6a:8f:64:f1:b8:6f:92:1b:
66:89
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
6F:FE:15:72:91:75:2F:F5:E5:50:A9:26:8F:76:C7:8F:26:BD:55:F0
X509v3 Authority Key Identifier:
keyid:3F:8C:95:EE:E9:3A:B1:84:FC:0D:BA:1B:F1:A8:AA:FD:B1:C4:C0:E2
X509v3 Subject Alternative Name:
DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:172.20.0.112, IP Address:172.20.0.113, IP Address:172.20.0.114, IP Address:172.20.0.115, IP Address:10.254.0.1
Signature Algorithm: sha256WithRSAEncryption
40:6e:12:57:13:87:17:47:6c:5a:6a:bb:9f:8c:06:04:2c:7e:
71:5c:37:f7:6b:87:9c:23:ab:bf:54:df:53:be:f7:a1:3b:46:
b2:87:77:0a:fd:99:ef:7d:44:33:75:b1:bf:fc:ca:1a:20:8a:
c2:d7:e8:bc:e1:ea:7e:fe:7f:bb:58:68:03:a1:5d:9b:62:7e:
9a:48:27:36:42:be:14:41:9f:1a:41:53:31:0f:cf:a7:a8:70:
08:45:99:b9:09:82:fd:00:02:32:52:f5:62:c2:92:96:35:51:
25:ca:cc:08:d5:c4:e8:d2:97:50:93:9a:48:5e:6b:f9:d8:91:
58:4c:fb:24:56:aa:37:1b:78:94:ca:5f:a0:7c:84:91:99:a7:
8e:e5:37:83:50:b8:31:23:2d:bb:74:48:57:aa:5c:b3:5f:37:
15:b4:7d:5d:45:44:63:f5:c0:4b:83:a1:de:4b:a2:12:fe:2b:
e1:5c:58:6a:a6:74:b9:72:b7:3e:ff:89:4d:7c:18:d1:b5:6d:
ea:92:f3:67:53:ad:93:78:72:ab:9f:d8:89:e1:62:ce:d7:56:
57:cc:0b:f6:12:11:ab:17:ff:2e:76:19:c4:fe:36:e0:fe:74:
cc:f8:9f:ee:cd:27:00:21:fc:61:79:48:e5:27:74:bb:bd:e4:
26:8c:8d:50
5.2 使用cfssl-certinfo命令
$ cfssl-certinfo -cert kubernetes.pem
{
"subject": {
"common_name": "kubernetes",
"country": "CN",
"organization": "k8s",
"organizational_unit": "System",
"locality": "BeiJing",
"province": "BeiJing",
"names": [
"CN",
"BeiJing",
"BeiJing",
"k8s",
"System",
"kubernetes"
]
},
"issuer": {
"common_name": "kubernetes",
"country": "CN",
"organization": "k8s",
"organizational_unit": "System",
"locality": "BeiJing",
"province": "BeiJing",
"names": [
"CN",
"BeiJing",
"BeiJing",
"k8s",
"System",
"kubernetes"
]
},
"serial_number": "581408424675478050081912100441044253482994605963",
"sans": [
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"127.0.0.1",
"172.20.0.112",
"172.20.0.113",
"172.20.0.114",
"172.20.0.115",
"10.254.0.1"
],
"not_before": "2021-08-03T02:31:00Z",
"not_after": "2031-08-01T02:31:00Z",
"sigalg": "SHA256WithRSA",
"authority_key_id": "3F:8C:95:EE:E9:3A:B1:84:FC:D:BA:1B:F1:A8:AA:FD:B1:C4:C0:E2",
"subject_key_id": "6F:FE:15:72:91:75:2F:F5:E5:50:A9:26:8F:76:C7:8F:26:BD:55:F0",
"pem": "-----BEGIN CERTIFICATE-----\nMIIEizCCA3OgAwIBAgIUZddBZND3Yng62uMcciUWhLe7n4swDQYJKoZIhvcNAQEL\nBQAwZTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0Jl\naUppbmcxDDAKBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwpr\ndWJlcm5ldGVzMB4XDTIxMDgwMzAyMzEwMFoXDTMxMDgwMTAyMzEwMFowZTELMAkG\nA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0JlaUppbmcxDDAK\nBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwprdWJlcm5ldGVz\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0tgsFZZtlwMcJhoXBjjV\nf+FPEjr2Ur1OrgoSeqzJiiM4K1w/RkeflRV6TWnzCod0XCgu5Wz+VXlR0Vqe7/2j\nZWfPZvP71kopmBGoiueajwW9TCPpUpNDWgn8OUis+QQVoZIer00mmXvX7jywcbyi\nldrUeNxEUq3zR6LSTVcl5VZZ7cXdLFvWZvF+C01hA2qLoN9UcfNIyFEC/EuiTXGM\nQa8Lt/NhCanESXzgaNam6uO9J5Uv96ZZVijqsOK4PjF0l+4ECHCuAE8hqTd33Pt2\nxnjF+qNGILT2X3nRFvxfZMdXaDRJO5lAkc4sLd7+/PqDXl5zbPFGao9k8bhvkhtm\niQIDAQABo4IBMTCCAS0wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUF\nBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBRv/hVykXUv9eVQ\nqSaPdsePJr1V8DAfBgNVHSMEGDAWgBQ/jJXu6TqxhPwNuhvxqKr9scTA4jCBrQYD\nVR0RBIGlMIGiggprdWJlcm5ldGVzghJrdWJlcm5ldGVzLmRlZmF1bHSCFmt1YmVy\nbmV0ZXMuZGVmYXVsdC5zdmOCHmt1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rl\ncoIka3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FshwR/AAABhwSs\nFABwhwSsFABxhwSsFAByhwSsFABzhwQK/gABMA0GCSqGSIb3DQEBCwUAA4IBAQBA\nbhJXE4cXR2xaarufjAYELH5xXDf3a4ecI6u/VN9TvvehO0ayh3cK/ZnvfUQzdbG/\n/MoaIIrC1+i84ep+/n+7WGgDoV2bYn6aSCc2Qr4UQZ8aQVMxD8+nqHAIRZm5CYL9\nAAIyUvViwpKWNVElyswI1cTo0pdQk5pIXmv52JFYTPskVqo3G3iUyl+gfISRmaeO\n5TeDULgxIy27dEhXqlyzXzcVtH1dRURj9cBLg6HeS6IS/ivhXFhqpnS5crc+/4lN\nfBjRtW3qkvNnU62TeHKrn9iJ4WLO11ZXzAv2EhGrF/8udhnE/jbg/nTM+J/uzScA\nIfxheUjlJ3S7veQmjI1Q\n-----END CERTIFICATE-----\n"
}
六 分發證書
將生成的證書和秘鑰文件(后綴名為.pem)拷貝到所有機器的 /etc/kubernetes/ssl 目錄下
for n in {cat host.txt}; do ssh root@$n ; mkdir -p /etc/kubernetes/ssl;exit;done
for n in {cat hosts.txt}; do scp -r *.pem root@/etc/kubernetes/ssl/;done
本文有參考雲原生社區文章
個人理解,如有錯誤麻煩指正^_^
