環境說明
三台主機：
k8s-master 10.0.3.225
k8s-node1 10.0.3.226
k8s-node2 10.0.3.227
配置主機名解析
[root@k8s-master kubernetes]# vim /etc/hosts 10.0.3.225 k8s-master 10.0.3.226 k8s-node1 10.0.3.227 k8s-node2
關閉防火牆和SElinux
[root@k8s-master kubernetes]# systemctl stop firewalld [root@k8s-master kubernetes]# getenforce Disabled
設置環境變量
vim /root/.bash_profile PATH=$PATH:$HOME/bin:/opt/kubernetes/bin
source /root/.bash_profile
為了拷貝方便，設置SSH免密
[root@k8s-master src]# ssh-keygen [root@k8s-master src]# ssh-copy-id 10.0.3.225 [root@k8s-master src]# ssh-copy-id 10.0.3.226 [root@k8s-master src]# ssh-copy-id 10.0.3.227
系統環境初始化
1.安裝Docker（三台操作）
第一步：使用國內Docker源
[root@linux-node1 ~]# cd /etc/yum.repos.d/ [root@linux-node1 yum.repos.d]# wget \ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
第二步：Docker安裝：
[root@linux-node1 ~]# yum install -y docker-ce
第三步：啟動后台進程：
[root@linux-node1 ~]# systemctl start docker
2.准備部署目錄（三台操作）
mkdir -p /opt/kubernetes/{cfg,bin,ssl,log}
3.准備軟件包百度網盤下載地址：（在mastrt操作）
[https://pan.baidu.com/s/1zs8sCouDeCQJ9lghH1BPiw](https://pan.baidu.com/s/1zs8sCouDeCQJ9lghH1BPiw)
cd /usr/local/ unzip k8s-v1.10.1-manual.zip cd k8s-v1.10.1-manual/k8s-v1.10.1/ mv * /usr/local/src/
k8s官網地址：https://github.com/kubernetes
4.解壓軟件包（在mastrt操作）
cd /usr/local/src/ tar zxvf kubernetes.tar.gz && \ tar zxvf kubernetes-server-linux-amd64.tar.gz && \ tar zxvf kubernetes-client-linux-amd64.tar.gz && \ tar zxvf kubernetes-node-linux-amd64.tar.gz
所有軟件包都在kubernetes目錄下了
[root@k8s-master src]# ls cfssl-certinfo_linux-amd64 cfssl_linux-amd64 etcd-v3.2.18-linux-amd64.tar.gz kubernetes kubernetes-node-linux-amd64.tar.gz kubernetes.tar.gz cfssljson_linux-amd64 cni-plugins-amd64-v0.7.1.tgz flannel-v0.10.0-linux-amd64.tar.gz kubernetes-client-linux-amd64.tar.gz kubernetes-server-linux-amd64.tar.gz [root@k8s-master src]# cd kubernetes/ [root@k8s-master kubernetes]# ls addons client cluster docs examples hack kubernetes-src.tar.gz LICENSES node README.md server third_party version [root@k8s-master kubernetes]#
手動制作CA證書
1.安裝 CFSSL
上面的安裝包，已經下載好cfssl了。移動/opt/kubernetes/bin/即可。
cd /usr/local/src chmod +x cfssl* && \ mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo && \ mv cfssljson_linux-amd64 /opt/kubernetes/bin/cfssljson && \ mv cfssl_linux-amd64 /opt/kubernetes/bin/cfssl
復制cfssl命令文件到k8s-node1和k8s-node2節點。如果實際中多個節點，就都需要同步復制。
[root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.3.226:/opt/kubernetes/bin/ [root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.3.227:/opt/kubernetes/bin/
CFSSL下載地址：
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
2.初始化cfssl
[root@k8s-master src]# cd /usr/local/src/ [root@k8s-master src]# mkdir ssl && cd ssl [root@k8s-master ssl]# cfssl print-defaults config > config.json [root@k8s-master ssl]# cfssl print-defaults csr > csr.json
3.創建用來生成 CA 文件的 JSON 配置文件
[root@k8s-master ssl]# vim ca-config.json { "signing": { "default": { "expiry": "8760h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" } } } }
4.創建用來生成 CA 證書簽名請求（CSR）的 JSON 配置文件
[root@k8s-master ssl]# vim ca-csr.json { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
5.生成CA證書（ca.pem）和密鑰（ca-key.pem）
[root@k8s-master ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca 2018/11/13 16:12:11 [INFO] generating a new CA key and certificate from CSR 2018/11/13 16:12:11 [INFO] generate received request 2018/11/13 16:12:11 [INFO] received CSR 2018/11/13 16:12:11 [INFO] generating key: rsa-2048 2018/11/13 16:12:12 [INFO] encoded CSR 2018/11/13 16:12:12 [INFO] signed certificate with serial number 672169645465776774912654230107317469678160608506 [root@k8s-master ssl]# ls -l ca* -rw-r--r-- 1 root root 290 Nov 13 16:10 ca-config.json -rw-r--r-- 1 root root 1001 Nov 13 16:12 ca.csr -rw-r--r-- 1 root root 208 Nov 13 16:11 ca-csr.json -rw------- 1 root root 1675 Nov 13 16:12 ca-key.pem -rw-r--r-- 1 root root 1359 Nov 13 16:12 ca.pem
6.分發證書
[root@k8s-master ssl]# cp ca.csr ca.pem ca-key.pem ca-config.json /opt/kubernetes/ssl
#scp證書到k8s-node1和k8s-node2節點 [root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.3.226:/opt/kubernetes/ssl ca.csr 100% 1001 1.0KB/s 00:00 ca.pem 100% 1359 1.3KB/s 00:00 ca-key.pem 100% 1675 1.6KB/s 00:00 ca-config.json 100% 290 0.3KB/s 00:00 [root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.3.227:/opt/kubernetes/ssl ca.csr 100% 1001 1.0KB/s 00:00 ca.pem 100% 1359 1.3KB/s 00:00 ca-key.pem 100% 1675 1.6KB/s 00:00 ca-config.json 100% 290 0.3KB/s 00:00
部署ETCD集群
詳細介紹查看etcd官網： https://github.com/coreos/etcd-operator
0.准備etcd軟件包
[root@k8s-master src]# pwd /usr/local/src [root@k8s-master src]# tar zxvf etcd-v3.2.18-linux-amd64.tar.gz [root@k8s-master src]# cd etcd-v3.2.18-linux-amd64/ [root@k8s-master etcd-v3.2.18-linux-amd64]# ls Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
[root@k8s-master etcd-v3.2.18-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin/ [root@k8s-master etcd-v3.2.18-linux-amd64]# cp etcd etcdctl 10.0.3.226:/opt/kubernetes/bin/ [root@k8s-master etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 10.0.3.227:/opt/kubernetes/bin/
下載地址：wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz
1.創建 etcd 證書簽名請求：
[root@k8s-master ssl]# cd /usr/local/src/ssl/ [root@k8s-master ssl]# vim etcd-csr.json { "CN": "etcd", "hosts": [ "127.0.0.1", "10.0.3.225", #指定etcd節點的IP地址 "10.0.3.226", "10.0.3.227" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
2.生成 etcd 證書和私鑰：
[root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ > -ca-key=/opt/kubernetes/ssl/ca-key.pem \ > -config=/opt/kubernetes/ssl/ca-config.json \ > -profile=kubernetes etcd-csr.json | cfssljson -bare etcd 會生成一下證書文件 [root@k8s-master ssl]# ls -l etcd* -rw-r--r-- 1 root root 1062 Nov 13 16:40 etcd.csr -rw-r--r-- 1 root root 278 Nov 13 16:39 etcd-csr.json -rw------- 1 root root 1675 Nov 13 16:40 etcd-key.pem -rw-r--r-- 1 root root 1436 Nov 13 16:40 etcd.pem
3.將證書移動到/opt/kubernetes/ssl目錄下
[root@k8s-master ssl]# cp etcd*.pem /opt/kubernetes/ssl [root@k8s-master ssl]# scp etcd*.pem 10.0.3.226:/opt/kubernetes/ssl etcd-key.pem 100% 1675 1.6KB/s 00:00 etcd.pem 100% 1436 1.4KB/s 00:00 [root@k8s-master ssl]# scp etcd*.pem 10.0.3.227:/opt/kubernetes/ssl etcd-key.pem 100% 1675 1.6KB/s 00:00 etcd.pem
4.設置ETCD配置文件
[root@k8s-master ssl]# vim /opt/kubernetes/cfg/etcd.conf #[member] ETCD_NAME="etcd-node1" #唯一的，每個節點不能一樣 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #數據存放目錄 #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://10.0.3.225:2380" #集群之間通信使用 ETCD_LISTEN_CLIENT_URLS="https://10.0.3.225:2379,https://127.0.0.1:2379" #是給客戶端使用 #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.3.225:2380" # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." ETCD_INITIAL_CLUSTER="etcd-node1=https://10.0.3.225:2380,etcd-node2=https://10.0.3.226:2380,etcd-node3=https://10.0.3.227:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://10.0.3.225:2379" #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
5.創建ETCD系統服務
[root@k8s-master ssl]# vim /etc/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target [Service] Type=simple WorkingDirectory=/var/lib/etcd EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf # set GOMAXPROCS to number of processors ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd" Type=notify [Install] WantedBy=multi-user.target
6.重新加載系統服務
[root@k8s-master ssl]# systemctl daemon-reload
[root@k8s-master ssl]# systemctl enable etcd.service
etcd.conf和etcd.service拷貝到其他節點
[root@k8s-master ssl]# scp /opt/kubernetes/cfg/etcd.conf 10.0.3.226:/opt/kubernetes/cfg/ etcd.conf 100% 1148 1.1KB/s 00:00 [root@k8s-master ssl]# scp /opt/kubernetes/cfg/etcd.conf 10.0.3.227:/opt/kubernetes/cfg/ etcd.conf 100% 1148 1.1KB/s 00:00 [root@k8s-master ssl]# scp /etc/systemd/system/etcd.service 10.0.3.226:/opt/kubernetes/cfg/ etcd.service 100% 314 0.3KB/s 00:00 [root@k8s-master ssl]# scp /etc/systemd/system/etcd.service 10.0.3.227:/opt/kubernetes/cfg/ etcd.service
其他節點需要修改一下配置。以node1為例，node2一樣要修改。
[root@k8s-node1 yum.repos.d]# vim /opt/kubernetes/cfg/etcd.conf 2 ETCD_NAME="etcd-node2" 7 ETCD_LISTEN_PEER_URLS="https://10.0.3.226:2380" 8 ETCD_LISTEN_CLIENT_URLS="https://10.0.3.226:2379,https://127.0.0.1:2379" 13 ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.3.226:2380" 19 ETCD_ADVERTISE_CLIENT_URLS="https://10.0.3.226:2379"
同樣需要reload一下。systemctl daemon-reload
systemctl enable etcd
在所有節點上創建etcd存儲目錄並啟動etcd
mkdir /var/lib/etcd systemctl start etcd systemctl status etcd
netstat -lntup|grep 23
需要在所有的 etcd 節點重復上面的步驟，直到所有機器的 etcd 服務都已啟動。
7.驗證集群
[root@k8s-node1 ~]# etcdctl --endpoints=https://10.0.3.225:2379 \ > --ca-file=/opt/kubernetes/ssl/ca.pem \ > --cert-file=/opt/kubernetes/ssl/etcd.pem \ > --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health member 981424cb437f053 is healthy: got healthy result from https://10.0.3.226:2379 member 7a30be4a7549b699 is healthy: got healthy result from https://10.0.3.227:2379 member d12a89096ab8db09 is healthy: got healthy result from https://10.0.3.225:2379 cluster is healthy