k8s集群分為三個部分。如下圖所示
1. master節點
2. node節點
3. etcd存儲
部署master.com節點
根據上圖所示，master節點分為三個組件（apiserver、controller-manager、scheduler）
apiserver：k8s集群的總入口，基於hppts通訊。所以要部署證書。
第一步：准備k8s證書：
vim ca-config.json
{ "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } }
vim ca-csr.json
{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing", "O": "k8s", "OU": "System" } ] }
vim kube-proxy-csr.json
{ "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] }
vim server-csr.json 此處記得修改ip
{ "CN": "kubernetes", "hosts": [ "10.0.0.1", "127.0.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local", "192.168.1.46", "192.168.1.47", "192.168.1.46" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] }
若已安裝這三個工具，請忽略。若未安裝，復制即可。且給755的權限。
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
生成證書，待備用。
第二步：下載二進制文件。
下載官網：
wget https://dl.k8s.io/v1.16.13/kubernetes-server-linux-amd64.tar.gz
tar -zxvf kubernetes-server-linux-amd64.tar.gz
所以接下來，我的服務規划還是放在opt下
bin：二進制文件
cfg：配置文件
logs：日志
ssl：證書
mkdir -p /opt/kubernetes/{bin,cfg,logs,ssl}
cp ./kubernetes/server/bin/{kube-apiserver,kube-controller-manager,kubectl,kube-scheduler} /opt/kubernetes/bin/
復制剛才的證書過來
cp *.pem /opt/kubernetes/ssl/ （proxy的是node用的。可以不復制）
第三步：配置三個服務的配置文件
kube-apiserver要用到token，所以我們還需要生成一個token文件。
head -c 16 /dev/urandom | od -An -t x | tr -d ' ' # 命令會隨機生成一個token
vim /opt/kubernetes/cfg/token.csv
# 若需要更換token，請使用： head -c 16 /dev/urandom | od -An -t x | tr -d ' ' # 命令會隨機生成一個token
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
下面正式編寫三個組件的配置文件
vim /opt/kubernetes/cfg/kube-apiserver.conf (記得修改etcd集群的ip和監聽地址的ip)
KUBE_APISERVER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --etcd-servers=https://192.168.31.61:2379,https://192.168.31.62:2379,https://192.168.31.63:2379 \ --bind-address=192.168.31.61 \ --secure-port=6443 \ --advertise-address=192.168.31.61 \ --allow-privileged=true \ --service-cluster-ip-range=10.0.0.0/24 \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \ --authorization-mode=RBAC,Node \ --enable-bootstrap-token-auth=true \ --token-auth-file=/opt/kubernetes/cfg/token.csv \ --service-node-port-range=30000-32767 \ --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \ --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \ --tls-cert-file=/opt/kubernetes/ssl/server.pem \ --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \ --client-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \ --etcd-cafile=/opt/etcd/ssl/ca.pem \ --etcd-certfile=/opt/etcd/ssl/server.pem \ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
更多選項參數，請參考官方文檔。
vim /opt/kubernetes/cfg/kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect=true \ --master=127.0.0.1:8080 \ --address=127.0.0.1 \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ --service-cluster-ip-range=10.0.0.0/24 \ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \ --root-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \ --experimental-cluster-signing-duration=87600h0m0s"
vim /opt/kubernetes/cfg/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect \ --master=127.0.0.1:8080 \ --address=127.0.0.1"
master已經部署完畢。master其實就只有這么點內容。（如下圖）
下面就可以啟動三個組件服務了。為了方便，我們順便編寫一下三個服務的啟動文件吧
第四步：編寫組件的啟動文件，然后啟動服務
vim /usr/lib/systemd/system/kube-apiserver.service
[Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target
vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target
vim /usr/lib/systemd/system/kube-scheduler.service
[Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target
啟動服務，並設置開機自啟動
systemctl start kube-apiserver systemctl start kube-controller-manager systemctl start kube-scheduler systemctl enable kube-apiserver systemctl enable kube-controller-manager systemctl enable kube-scheduler
最后一步：因為加入節點利用token自動頒發證書。所以kubelet-bootstrap授權
kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap
部署node節點
部署前的准備，同樣的關閉防火牆，swap，配置hosts文件。
node節點也是由三個組件組成的。分別是docker、kube-proxy、kubelet
安裝docker-ce。
docker的安裝省略，請參考相關的文檔：https://developer.aliyun.com/mirror/docker-ce?spm=a2c6h.13651102.0.0.3e221b11P6sarI
sudo yum install -y yum-utils device-mapper-persistent-data lvm2 sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo sudo yum makecache fast sudo yum -y install docker-ce sudo systemctl start docker
必須先啟動kubelet，再到kube-proxy
規划目錄
mkdir /opt/kubernetes/{bin,cfg,logs,ssl} -p
第一步：把證書復制到ssl目錄下。(ca.pem kube-proxy-key.pem kube-proxy.pem)
此處的ca證書必須與master上的ca-key.pem是一對的
第二步：把二進制文件拷貝到bin目錄
官網下載。服務端中，有kube-proxy的相關組件的。所以可以直接復制過來使用
第三步：編寫kubelet，kube-proxyt的配置文件。
kubelet和kube-proxy都是由三個組件組成的服務。所以兩個服務共有6個配置文件
kubelet的三個配置文件
1. bootstrap.kubeconfig：自動頒發證書的配置文件
2. kubelet.conf：主配置文件
3. kubelet-config.yml：生成服務的配置文件
kube-proxy的服務主要由三個配置文件組成
1. conf文件：基本的配置文件
2. kubeconfig：連接api-server的配置文件
3. yml文件：kube-proxy的主要配置文件。此文件是從早期版本的conf文件分離出來。
所以我們先編輯kubelet三個配置文件
vim /opt/kubernetes/cfg/bootstrap.kubeconfig
apiVersion: v1 clusters: - cluster: certificate-authority: /opt/kubernetes/ssl/ca.pem server: https://192.168.31.61:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubelet-bootstrap name: default current-context: default kind: Config preferences: {} users: - name: kubelet-bootstrap user: token: c47ffb939f5ca36231d9e3121a252940
溫馨提示：記得修改ip，還有連接服務端的token要一致
vim /opt/kubernetes/cfg/kubelet.conf 節點名字，必須與其他節點不一樣。
KUBELET_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --hostname-override=k8s-node1 \ --network-plugin=cni \ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \ --config=/opt/kubernetes/cfg/kubelet-config.yml \ --cert-dir=/opt/kubernetes/ssl \ --pod-infra-container-image=lizhenliang/pause-amd64:3.0"
溫馨提示：pause-amd64，請參考阿里雲https://developer.aliyun.com/article/680942?spm=a2c6h.14164896.0.0.72ac5d8d3HQauK
vim /opt/kubernetes/cfg/kubelet-config.yml
kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: 0.0.0.0 port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - 10.0.0.2 clusterDomain: cluster.local failSwapOn: false authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% maxOpenFiles: 1000000 maxPods: 110
編寫kubelet.service服務啟動文件，並啟動服務
vim /usr/lib/systemd/system/kubelet.service
[Unit] Description=Kubernetes Kubelet After=docker.service Before=docker.service [Service] EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target
kubelet服務部署完畢。
編寫kube-proxy的配置3個配置文件
vim /opt/kubernetes/cfg/kube-proxy.conf
KUBE_PROXY_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --config=/opt/kubernetes/cfg/kube-proxy-config.yml"
vim /opt/kubernetes/cfg/kube-proxy.kubeconfig 溫馨提示：此處記得修改連接kube-apiserver的ip哦
apiVersion: v1 clusters: - cluster: certificate-authority: /opt/kubernetes/ssl/ca.pem server: https://192.168.1.47:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kube-proxy name: default current-context: default kind: Config preferences: {} users: - name: kube-proxy user: client-certificate: /opt/kubernetes/ssl/kube-proxy.pem client-key: /opt/kubernetes/ssl/kube-proxy-key.pem
vim /opt/kubernetes/cfg/kube-proxy-config.yml # 此處的名字不能與其他節點名字一樣，所以需要修改
kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 address: 0.0.0.0 metricsBindAddress: 0.0.0.0:10249 clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig hostnameOverride: k8s-node1 clusterCIDR: 10.0.0.0/24 mode: ipvs ipvs: scheduler: "rr" iptables: masqueradeAll: true
注意：k8s-node1這個主機名必須與宿主機的hostname一致。否則會出現找不到host的情況。
Error from server: Get https://k8s-node1:10250/containerLogs/kube-system/kube-flannel-ds-amd64-fmmh9/kube-flannel: dial tcp: lookup k8s-node1 on 100.100.2.136:53: no such host
編寫啟動文件kube-proxy.service
vim /usr/lib/systemd/system/kube-proxy.service
[Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target
到此kubelet和kube-proxy全部配置完畢。
啟動docker、kubelet、kube-proxy三個服務。
systemctl daemon-reload
systemctl start kubelet && systemctl enable kubelet
systemctl start kube-proxy && systemctl enable kube-proxy
最后還有一步非常關鍵的一步，就是在master主節點上允許頒發證書。
kubectl get csr kubectl certificate approve node-csr-MYUxbmf_nmPQjmH3LkbZRL2uTO-_FCzDQUoUfTy7YjI
檢查驗證：node是否成功加入到主節點上
node節點添加成功。
【master、node部署完畢】
為了方便，加個簡單的啟動腳本。此步驟可省略，個人喜歡。
vim /usr/local/sbin/master-service
#!/bin/bash case $1 in start) systemctl $1 kube-apiserver systemctl $1 kube-controller-manager systemctl $1 kube-scheduler ;; stop) systemctl $1 kube-apiserver systemctl $1 kube-controller-manager systemctl $1 kube-scheduler ;; restart) systemctl $1 kube-apiserver systemctl $1 kube-controller-manager systemctl $1 kube-scheduler ;; *) echo "'$1'輸入有誤,僅能輸入start、stop、restart" esac
chmod 755/usr/local/sbin/master-service
vim /usr/local/sbin/node-service
#!/bin/bash case $1 in start) systemctl $1 docker systemctl $1 kubelet systemctl $1 kube-proxy ;; stop) systemctl $1 docker systemctl $1 kubelet systemctl $1 kube-proxy ;; restart) systemctl $1 docker systemctl $1 kubelet systemctl $1 kube-proxy ;; *) echo "'$1'輸入有誤，僅能輸入start、stop、restart"
esac
chmod 755 /usr/local/sbin/node-service
集群部署完畢后，但我們還需要一個插件來維持集群的網絡。
安裝cni網絡 ——>> flannel
第一步：在node節點下載cni，並解壓到bin目錄下 （所有node）
規划cni服務：mkdir /opt/cni/{bin,net.d} -p
溫馨提示：kubelet服務會調用cni，所以node節點上都必須有cni插件。
cni下載地址：https://github.com/containernetworking/plugins/releases
wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
tar -zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin/
第二步：在master下安裝flannel工具。官網地址：https://github.com/coreos/flannel#flannel
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml
第三步：檢查結果。flannel其實也是一個pod，安裝的默認命令空間在kube-system
【整個k8s單master集群部署完畢】
授權查看日志等功能
vim apiserver-to-kubelet-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubelet rules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics - pods/log verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kube-apiserver namespace: "" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubernetes
kubectl apply -f apiserver-to-kubelet-rbac.yaml
集群部署coredns
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns/coredns
此官方的文件有兩個要修改的地方。
1. 域名的變量，大概在70行。
2. clusterIP要與客戶端的kubelet的一致。kubelet有指定dns。大概在214行。
