一、概述
主要介紹搭建流程及使用注意事項，如果線上使用的話，請務必做好相關測試及壓測。
1、基礎環境准備
系統：ubuntu TLS 16.04 5台
docker-ce：17.06.2
kubeadm、kubelet、kubectl：1.12.0
keepalived、haproxy
etcd-3.2.22
2、安裝前准備
1）k8s各節點SSH免密登錄。
2）各Node必須關閉swap：swapoff -a，否則kubelet啟動失敗。
vim /etc/sysctl.conf vm.swappiness = 0 sysctl -p
3）各節點主機名和IP加入/etc/hosts解析
etcd采用單獨部署的集群。
二、部署准備
1、安裝docker：
apt-get update apt-get install \ apt-transport-https \ ca-certificates \ curl \ software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - apt-key fingerprint 0EBFCD88 add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" apt-get update apt policy docker-ce 找到自己需要的版本，然后安裝 apt install -y docker-ce=17.06.2~ce-0~ubuntu
修改docker的一些設置
vim /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --graph=/data/docker --storage-driver=overlay2
配置docker的proxy，否則無法直接下載鏡像。
mkdir /etc/systemd/system/docker.service.d
cat http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://10.42.3.110:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.42.73.110,10.10.25.49,10.2.140.154,10.42.104.113,10.2.177.142,10.2.68.77,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hp.com,charts.gitlab.io,.mirror.ucloud.cn"
cat https-proxy.conf
[Service]
Environment="HTTPS_PROXY=http://10.42.3.110:8118" "NO_PROXY=localhost,127.0.0.1,0.0.0.0,10.42.73.110,10.10.25.49,10.2.140.154,10.42.104.113,10.2.177.142,10.2.68.77,172.11.0.0,172.10.0.0,172.11.0.0/16,172.10.0.0/16,10.,172.,.evo.get.com,.kube.hp.com,charts.gitlab.io,.mirror.ucloud.cn"
systemctl daemon-reload
systemctl restart docker
iptables設置，17.06.2默認的FORWARD是DROP，這樣pod之間的通信就會收到影響，所以需要設置成ACCEPT
iptables -P FORWARD ACCEPT iptables-save
2、安裝CA證書
1）安裝cfssl證書管理工具
mkdir /opt/bin wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64 mv cfssl_linux-amd64 /opt/bin/cfssl wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64 mv cfssljson_linux-amd64 /opt/bin/cfssljson wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl-certinfo_linux-amd64 mv cfssl-certinfo_linux-amd64 /opt/bin/cfssl-certinfo echo "export PATH=/opt/bin:$PATH" > /etc/profile.d/k8s.sh
source /etc/profile.d/k8s.sh
2)創建CA配置文件
mkdir ssl;cd ssl cfssl print-defaults config >config.json cfssl print-defaults csr >csr.json
根據生成的文件創建ca-config.json文件，過期時間設置為87600h
# cat ca-config.json
{ "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } }
創建CA證書簽名請求
# cat ca-csr.json
{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "GD", "L": "SZ", "O": "k8s", "OU": "System" } ] }
生成證書和私鑰
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
copy證書到相應節點
mkdir /etc/kubernetes/ssl -p cp ~/ssl/ca* /etc/kubernetes/ssl scp -r /etc/kubernetes/ km12-02:/etc scp -r /etc/kubernetes/ km12-03:/etc
3、下載etcd，新版本的k8s要求etcd版本3.2.18以上，這里使用3.2.22
wget https://github.com/coreos/etcd/releases/download/v3.2.22/etcd-v3.2.22-linux-amd64.tar.gz tar xf etcd-v3.2.22-linux-amd64.tar.gz cd etcd-v3.2.22-linux-amd64 cp etcd etcdctl /opt/bin
source /etc/profile.d/k8s.sh
制作etcd證書：
創建簽名文件
# cat etcd-csr.json
{ "CN": "etcd", "hosts": [ "127.0.0.1", "10.42.13.230", "10.42.43.147", "10.42.150.212" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "GD", "L": "SZ", "O": "k8s", "OU": "System" } ] }
生成證書和私鑰
cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd mkdir /etc/etcd/ssl -p cp etcd*.pem /etc/etcd/ssl scp -r /etc/etcd/ km12-02:/etc scp -r /etc/etcd/ km12-03:/etc
部署etcd集群：
創建etcd的systemd unit文件
# mkdir -p /var/lib/etcd
# cat /etc/systemd/system/etcd.service
[Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos
[Service] Type=notify WorkingDirectory=/var/lib/etcd/ EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/opt/bin/etcd \ --name=etcd-host0 \ --cert-file=/etc/etcd/ssl/etcd.pem \ --key-file=/etc/etcd/ssl/etcd-key.pem \ --peer-cert-file=/etc/etcd/ssl/etcd.pem \ --peer-key-file=/etc/etcd/ssl/etcd-key.pem \ --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --initial-advertise-peer-urls=https://10.42.13.230:2380 \
--listen-peer-urls=https://10.42.13.230:2380 \
--listen-client-urls=https://10.42.13.230:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://10.42.13.230:2379 \
--initial-cluster-token=etcd-cluster-1 \ --initial-cluster=etcd-host0=https://10.42.13.230:2380,etcd-host1=https://10.42.43.147:2380,etcd-host2=https://10.42.150.212:2380 \
--initial-cluster-state=new \ --data-dir=/var/lib/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target
將創建好的啟動文件和之前解壓的etcd文件都copy到其他master節點上
tar xf etcd-v3.2.22-linux-amd64.tar.gz cd etcd-v3.2.22-linux-amd64 cp etcd etcdctl /opt/bin scp -r /opt/bin km12-02:/opt scp -r /opt/bin km12-03:/opt source /etc/profile.d/k8s.sh copy到其他節點 ，注意修改幾個地方，修改為相應的名字和地址： --name=etcd-host0 --initial-advertise-peer-urls=https://10.42.13.230:2380
--listen-peer-urls=https://10.42.13.230:2380
--listen-client-urls=https://10.42.13.230:2379,http://127.0.0.1:2379
--advertise-client-urls=https://10.42.13.230:2379
啟動etcd
systemctl enable etcd
systemctl start etcd
systemctl status etcd
查看集群狀態
etcdctl --key-file /etc/etcd/ssl/etcd-key.pem --cert-file /etc/etcd/ssl/etcd.pem --ca-file /etc/kubernetes/ssl/ca.pem member list etcdctl --key-file /etc/etcd/ssl/etcd-key.pem --cert-file /etc/etcd/ssl/etcd.pem --ca-file /etc/kubernetes/ssl/ca.pem cluster-health
為了方便使用etcdctl我們可以做一個alias,因為kubernetes默認使用etcdv3，而etcd部署默認使用etcdv2，所以在etcd里面查看不到kubernetes的數據，需要添加一個全局變量export ETCDCTL_API=3,
在所有master節點上添加如下內容：
# vim ~/.bashrc
#etcd
host1='10.42.13.230:2379'
host2='10.42.43.147:2379'
host3='10.42.150.212:2379'
endpoints=$host1,$host2,$host3
alias etcdctl='etcdctl --endpoints=$endpoints --key /etc/etcd/ssl/etcd-key.pem --cert /etc/etcd/ssl/etcd.pem --cacert /etc/kubernetes/ssl/ca.pem'
export ETCDCTL_API=3
是內容生效：
#source ~/.bashrc
4、在另外的兩台服務器上面部署keepalived、haproxy
apt install -y keepalived haproxy
1)創建keepalived配置文件
master:
cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { notification_email { root@localhost #發送郵箱 } notification_email_from keepalived@localhost #郵箱地址 smtp_server 127.0.0.1 #郵件服務器地址 smtp_connect_timeout 30 router_id km12-01 #主機名，每個節點不同即可 } vrrp_instance VI_1 { state MASTER #在另一個節點上為BACKUP interface eth0 #IP地址漂移到的網卡 virtual_router_id 6 #多個節點必須相同 priority 100 #優先級，備用節點的值必須低於主節點的值 advert_int 1 #通告間隔1秒 authentication { auth_type PASS #預共享密鑰認證 auth_pass 571f97b2 #密鑰 } unicast_src_ip 10.42.13.23 unicast_peer { 10.42.43.14 } virtual_ipaddress { 10.42.79.210/16 #VIP地址 } }
slave1: cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost #發送郵箱 } notification_email_from keepalived@localhost #郵箱地址 smtp_server 127.0.0.1 #郵件服務器地址 smtp_connect_timeout 30 router_id km12-02 #主機名，每個節點不同即可 } vrrp_instance VI_1 { state BACKUP #在另一個節點上為MASTER interface eth0 #IP地址漂移到的網卡 virtual_router_id 6 #多個節點必須相同 priority 80 #優先級，備用節點的值必須低於主節點的值 advert_int 1 #通告間隔1秒 authentication { auth_type PASS #預共享密鑰認證 auth_pass 571f97b2 #密鑰 } unicast_src_ip 10.42.43.14 unicast_peer {
10.42.13.23 } virtual_ipaddress { 10.42.79.210/16 #漂移過來的IP地址 } }
啟動keepalived
systemctl enable keepalived
systemctl start keepalived
keepalived采用的是單播。
2)haproxy配置文件
cat /etc/haproxy/haproxy.cfg global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon defaults mode tcp log global retries 3 timeout connect 10s timeout client 1m timeout server 1m frontend kubernetes bind *:6443 mode tcp default_backend kubernetes-master backend kubernetes-master balance roundrobin server master 10.42.13.230:6443 check maxconn 2000 server master2 10.42.43.147:6443 check maxconn 2000
兩台配置文件是一樣的。
啟動haproxy
systemctl enable haproxy
systemctl start haproxy
5、安裝指定版本的kubeadm、kubectl、kubelet
由於服務器不翻牆，下載會有問題，所以我們需要換一下源：
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
EOF
安裝
apt-get update apt policy kubeadm apt-get install kubeadm=1.12.0-00 kubectl=1.12.0-00 kubelet=1.12.0-00
升級軟件
apt-get upgrade kubeadm
回退版本
apt-get install aptitude
比如升到了kubeadm1.13.0，想退回到1.12.0
aptitude install kubeadm=1.12.0-00
卸載軟件
apt-get --purge remove 包名稱 如果不記得包的全部名稱，則可以使用如下命令查找 dpkg --get-selections | grep kube
創建初始化的kubeadm-config.yaml
# cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1alpha3
controlPlaneEndpoint: 10.42.9.210:6443 kind: ClusterConfiguration kubernetesVersion: v1.12.0 networking: podSubnet: 192.244.0.0/16
serviceSubnet: 192.96.0.0/16 apiServerCertSANs: - 10.42.13.230
- 10.42.43.147
- 10.42.150.212
- km12-01
- km12-02
- km12-03
- 127.0.0.1
- localhost - 10.42.9.210 etcd: external: endpoints: - https://10.42.13.230:2379
- https://10.42.43.147:2379
- https://10.42.150.212:2379
caFile: /etc/kubernetes/ssl/ca.pem certFile: /etc/etcd/ssl/etcd.pem keyFile: /etc/etcd/ssl/etcd-key.pem dataDir: /var/lib/etcd
clusterDNS:
- 10.96.0.2
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
controlPlaneEndpoint是haproxy的地址，或vip
檢查是否加載了ipvs：
lsmod | grep ip_vs
如果沒有輸出，則證明沒有加載ipvs，需要手動加載，這里為了方便寫了一個腳本：
# cat ipvs.sh
#!/bin/bash list="ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4" for i in $list do modprobe $i done
初始化
kubeadm init --config kubeadm-config.yaml
如果是已經初始化完成的集群想要修改為ipvs，做如下修改：
kubectl edit configmap kube-proxy -n kube-system ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "" syncPeriod: 30s kind: KubeProxyConfiguration metricsBindAddress: 127.0.0.1:10249 mode: "ipvs"
重新部署kube-proxy:
kubectl delete pod kube-proxy-ppsk90sd -n kube-system
將生成的pki拷貝到其他節點
scp -r /etc/kuberbetes/pki km12-02:/etc/kubernetes scp -r /etc/kubernetes/pki km12-03:/etc/kubernetes 注意，到其他服務器上面刪掉里面的兩個文件： rm -rf apiserver.crt apiserver.key
將初始化的文件以及/etc/profile.d/k8s.conf拷貝到其他節點
scp -r kubeadm-config.yaml km12-02: scp -r kubeadm-config.yaml km12-03: scp -r /etc/profile.d/k8s.sh km12-02:/etc/profile.d scp -r /etc/profile.d/k8s.sh km12-03:/etc/profile.d
有與所有的節點都設置了taints，所以都不能部署，此時會看到coredns一直處於pending的狀態，我們暫時把taints去掉，使其能正常部署
kubectl taint nodes --all node-role.kubernetes.io/master-
6、部署網絡
網絡有幾種選擇flannel+vxlan、flannel+host-gw、calico+BGP,經過測試calico+BGP這種方式網絡性能是比較好的，這里給出兩種方式部署網絡flannel+host-gw和calico+BGP。推薦使用calico+BGP。
第一種calico+BGP：
calico的datastore有兩種方式：kubernetes和etcd，這里建議使用etcd進行數據存儲。calico默認使用ipip模式，如果想關閉ipip模式只需要將CALICO_IPV4POOL_IPIP設置為Off，然后就會使用BGP。
下載文件：
wget https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/calico.yaml
etcd使用前面部署好的etcd集群，所以需要修改calico.yaml文件：
主要修改幾個地方：一個是etcd的相關配置，一個是CALICO_IPV4POOL_CIDR
# vim calico.yaml
kind: ConfigMap apiVersion: v1 metadata: name: calico-config namespace: kube-system data: # Configure this with the location of your etcd cluster. etcd_endpoints: "https://10.42.13.230:2379,https://10.42.43.147:2379,https://10.42.150.212:2379" # If you're using TLS enabled etcd uncomment the following.
# You must also populate the Secret below with these files. etcd_ca: "/calico-secrets/etcd-ca" etcd_cert: "/calico-secrets/etcd-cert" etcd_key: "/calico-secrets/etcd-key"
apiVersion: v1 kind: Secret type: Opaque metadata: name: calico-etcd-secrets namespace: kube-system data: # Populate the following files with etcd TLS configuration if desired, but leave blank if # not using TLS for etcd. # This self-hosted install expects three files with the following names. The values # should be base64 encoded strings of the entire contents of each file. etcd-key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFB0NBUUVBd2F3ZHNKcHB5WTR1UjFtZXJocHl5dWlPU1B3N0lWS1Zxc3dXaFV6ZG1YVWRzN2swCmlMaVNGbDhIUjRaVDcvcEIzOGRuenJINE1pQit4TTRDVHRKVkZjTFJBK2tkNUwzZWdodC9KOHlBS3UyVnE2M2wKTWpBazlIb3NqUU96a2dLSnN1RlhyRFdaZG5FeGlvNkZrQXBncDFySmZJdFh1UkpFdDFRMGplblVTM0xLaFZVWQpkdTdTV0thVkc5T3RmS3BzZUsvc2tYcDJlVmJIdGpiRE9lQURodnhyNk1FcDV4SVJ2bmZlWXppWmpUaUJXMkUrCkdFOCtjTGVCTmVLMVYxWFYvR1pmR0FTN2orWFVLOEIxVHFEQ2ZzU3QrRTEzTGMvQWhnTTF2QTZjaDIzMm16RlUKWWw0L21VaU5ZeUFHZmNkTW9XMkR4ck96M28yQ0JoMm50R1FIbndJREFRQUJBb0lCQVFDdUF2dGFKaU5PbVpVegpQbTY0ZDhNVVZRY2l4SjVNUG11RFhvUU5tUmF5UHV4RkF1OFV1T0ZwZ0ZDR0QwZHpveU4wOFRGd0lhK2pPbGNzCmRQelYzVjNkZzRGUlJpODV5dThWYjZjOEZ4NUJjOTY2dGkvc2ZCMTRIQjNUUmJRZVNIRjRhOGZlVHZwbjFTUnkKSGNRYnUzZEVadW9LSVJqY2pIRjVWd3N4UnhzYi9hSGE3MzUzR2pwTDhsZThQVGtielVyVktGYkZDN1U2cE1IYwo3MGxTeW5uUUVmNC9QdjJFcGNWa1hjQVFvWk44MkJUNWNEbTFheWhaWWhaUk5wdEhmV0w5VWtHaUZma3h5TXRnCjllSFJPNFpoMnZ6RW12QWRGbTZKcjB3aDlOQVVMUDhkNlh5OStnUFJ0Q1I4K2p0WHd3SzgwazZZMlpTWXhSTS8KYlAxYytyc0JBb0dCQU1xdk12cHVjTTVrc04vdkl0NE9Uc09lNE5QMzhSQVdaWktycmt4TjEwTXFrSEM5V0cxNQpwYmcyWGVyN2ZYTUFycmtpZUhiUU5ySzB6OTVVN3VnMzMrZ3VvVzRMNUJrYlNYVHY0eDJqMFBuQ3lhOFlMNkwrCjhuTi95SFBmUW1scnZWazJOd1FPSzgyaHQwL0RuOHFQeENNZVY1RmJUOWNjVVN2ZEI1czZqMHcvQW9HQkFQU2UKQzFTRnhJYThFSzVobGxZdXhQSUFHd29BNEIxV2R3RXRLMUF2WjRaamdCdmJqRjEySi9yMGNoQ08zbnllbThGdQovTU51WnczM1hsTThLTmJJWGpja3Y5RG9zQ2lOUXVtRE1ZUVY0SGFCL3U2SHo0MkNiaW1IMUlOSmtacDg5eS9nClRsQmRzb3puMUZiMG9aSFBsamcxR3BlM1hlUW9IZjJnRFFtSFJleWhBb0dBQjhXYURkSklUeEc3V29yZjZtWGcKRFU0OVRzTjJJZWpKOGtyVS95WUkrQkZjd29Yd0t4cDhDVWxFWnF0SUR5M2FoVXpXRTdBK09MSUlPbjFGUC9CTQpqS21sRlFRdHRleSs3MVgzZm41MmwrMHBNS2FieFFORHByd2lvcGJRQkJ6V2dPSThUOUovU1g1RytpOEZKSTVJCnRoUUd4Wk1ieDZMVlZmbyt2V3dNYko4Q2dZQkVEUk9wMSt6c2JyVlZUZUM4NlJYeEhRWm9xZ1d2STdiSHBRRS8dFhCZmVwN1JJU3JUZFdONTlUY21WQmloSXA3Q0dWWklLQmFUVkJYeG9mTGFqYk5vTTlrSkRUSzBsVmZnRHBkZgozNVlxWWMvQ0hCWXBqL2VGcGp3QXFoN1BrNlJRdFY0VURYejJwaWYwYU9ucDNvNHo1TklaRXZJVDN2VTQ4YVd1CjJPc2pBUUtCZ0c0OXZiNjNyc1p0eHZ3ZkhSaUVPUWdZZE9uWkhET2I0K2MxU2hpQWluN0hpOGRjdFR3cnl4a2UKdzBwZzV0R09xa1d1QUJ5VTFSQWR1c0pnUnZQZDBFc2l2YVBVTXhmd0cyWjBEVFpIZGVjS0tjWkNrdXkxQ3NKdgpkVEJLbnRjNmhwUUUrRnl3eHpBc1VTSCtpYUJ4YzczRlp3SWEwWEw2T1BuOFpnTnl5Qi9lCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg etcd-cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ0akNDQXNxZ0UJBZ0lVWFdsazdGY3JJR005T3Nxc08vOVBqTitjWGlrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1d6RUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUQWtkRU1Rc3dDUVlEVlFRSEV3SlRXakVNTUFvRwpBMVVFQ2hNRGF6aHpNUTh3RFFZRFZRUUxFd1pUZVhOMFpXMHhFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13CkhoY05NVGd4TVRBeE1EVTBNVEF3V2hjTk1qZ3hNREk1TURVME1UQXdXakJWTVFzd0NRWURWUVFHRXdKRFRqRUwKTUFrR0ExVUVDQk1DUjBReEN6QUpCZ05WQkFjVEFsTmFNUXd3Q2dZRFZRUUtFd05yT0hNeER6QU5CZ05WQkFzVApCbE41YzNSbGJURU5NQXNHQTFVRUF4TUVaWFJqWkRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDCkFRb0NnZ0VCQU1Hc0hiQ2FhY21PTGtkWm5xNGFjc3JvamtqOE95RlNsYXJNRm9WTTNabDFIYk81TklpNGtoWmYKQjBlR1UrLzZRZC9IWjg2eCtESWdmc1RPQWs3U1ZSWEMwUVBwSGVTOTNvSWJmeWZNZ0NydGxhdXQ1VEl3SlBSNgpMSTBEczVJQ2liTGhWNncxbVhaeE1ZcU9oWkFLWUtkYXlYeUxWN2tTUkxkVU5JM3AxRXR5eW9WVkdIYnUwbGltCmxSdlRyWHlxYkhpdjdKRjZkbmxXeDdZMnd6bmdBNGI4YStqQktlY1NFYjUzM21NNG1ZMDRnVnRoUGhoUFBuQzMKZ1RYaXRWZFYxZnhtWHhnRXU0L2wxQ3ZBZFU2Z3duN0VyZmhOZHkzUHdJWUROYndPbklkdDlwc3hWR0plUDVsSQpqV01nQm4zSFRLRnRnOGF6czk2TmdnWWRwN1JrQjU4Q0F3RUFBYU9Cb3pDQm9EQU9CZ05WSFE4QkFmOEVCQU1DCkJhQXdIUVlEVlIwbEJCWXdGQVlJS3dZQkJRVUhBd0VHQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXcKSFFZRFZSME9CQllFRkdDa2lWS0RwcEtWUHp3TVkrU1hWY2NoeFB4OE1COEdBMVVkSXdRWU1CYUFGSlFEaTdGNQp2WlNaSFhFVTlqK0FXVE9Xemx6TU1DRUdBMVVkRVFRYU1CaUhCSDhBQUFHSEJBb3FqSnFIQkFvcWFIR0hCQW9xCnNZNHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQmlkNUh2VmNGUTNlUjhBZXNHekhaRHRsWUZsNGVSUVlsZ0kKd0orbC9XV09QamwxL0lwSDJtanBsWHlnNlJBUmhIeENnZXY1NG1BSGhyK0U4TktBOXMyZFdMNWpnd2twL2p3aApzaDhsUEljeVE2ZHdXdjJNYWZaTU1Xdng5OWIrd2wyRHRGQUpiTzM5bWU4QVFJc2FrRGgrV2ZlZk02dW5QVkdHClhqMjVFbjVwQlB4NHFyRTRUcTVmeURaYUF1ay9tN0NianQ2WklaUUthRjVwYmczMVpCNjZRTVdQbjYvU05lWmkKbEhkWHY3c2xweExvWkZFczdXdEpxM0VUcDE4aXRjTER3S3FDWGZaVzhVZkRWaWNHdlRNYmpBb3NXSG1uaFdnZwpEc0w5MVJhUW9qTTzZnd1d0VaZ3lMaEhrQi96NFR1UUFXT1NaRWU3clJ0ckx0UFNNdVE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K etcd-ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURxakNDQXBLZ0F3SUJBZ0lVR0tCU2dDUTdVU2t6b2E4WnRlNTJ6UERGUk1zd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1d6RUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUQWtkRU1Rc3dDUVlEVlFRSEV3SlRXakVNTUFvRwpBMVVFQ2hNRGF6aHpNUTh3RFFZRFZRUUxFd1pUZVhOMFpXMHhFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13CkhoY05NVGd4TVRBeE1EVXpPREF3V2hjTk1qTXhNRE14TURVek9EQXdXakJiTVFzd0NRWURWUVFHRXdKRFRqRUwKTUFrR0ExVUVDQk1DUjBReEN6QUpCZ05WQkFjVEFsTmFNUXd3Q2dZRFZRUUtFd05yT0hNeER6QU5CZ05WQkFzVApCbE41YzNSbGJURVRNQkVHQTFVRUF4TUthM1ZpWlhKdVpYUmxjekNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTjdLd0ZJUlFNL2hTZTdZdTRsSTEyVDRLdzJHYVc4bnA4YU9XZXZzNzhGQU9qYnAKQnBsSEZVQmFrVVcrN3pnU0FHQVMzRnU2Um14ZGk1T2c3a2dZUG9TbitNRDFkWjVIYm1oQjhCY2RtOGVnQnZRcApONUtlS0ovOTJTN09SSm9VeHpmWEhUN3V5bTlZZUhSa0owaTFzSlNJWVB5dXJUbGhOUGs2K3B6aEUzN3Rhd1NkClNHM3l5UlJOY25BOHFyRE5nQlVyUUJBVVRwa0N5SklyVG1oOEpqckl1YTdKMml4YTRidXo3ZWxGRURDYTJpcTcKVmNQZkZ3MVdnM09pMVNmSWVxWVlTUW0zQkpiRnZ2cjBCaUQzYVM5cDNJenNoNEtRYlkyUlhrKzZxTWxYb2ZNaAo4Vmp0MVU2Tmh4RHlkZGwwVDJrRWNwRkx3dWk4ZFRRSEtndU4yQTBDQXdFQUFhTm1NR1F3RGdZRFZSMFBBUUgvCkJBUURBZ0VHTUJJR0ExVWRFd0VCL3dRSU1BWUJBZjhDQVFJd0hRWURWUjBPQkJZRUZKUURpN0Y1dlpTWkhYRVUKOWorQVdUT1d6bHpNTUI4R0ExVWRJd1FZTUJhQUZKUURpN0Y1dlpTWkhYRVU5aitBV1RPV3psek1NQTBHQ1NxRwpTSWIzRFFFQkN3VUFBNElCQVFBamRMakRoMGN2bnNSai9STVhwbWFZRnEwaFowcmZnVmtWcnowK1VDdWRkZnZ3ClVISVEva0p2RFVCdDl5WEtud3ZqQ0JKR3grMWsySTlENVVUcDI4bUpXcUpFbXdWcllqaXo4eUhYamNVbUlNVWkKeHFrZUlwU2lYR0tVZ3NDV1d0U0cwR2Vnbi9telVPMERwVHBVMkdyM0ZjbXJPcGlVNFd0RURZVUZtUm4yMXc0Wgo1S3RnSkpPNWFXaGFMR3c2VU52MFVhRUhTTjZoUk9qWG9pMy8zK09KN1pRajc1YXFnU1k0TURheDZBSFZ4NENSCnk2Q0srNUx0NWhNc0lKVkhETjZkQXYyTlFBU0VxSFVZazJmVFJ1VjJjeExIcE1MQVVTc1JRaThZWmJwYys2TUMKZjkxOHp3MG03NjZ3eTdqYVFJYmJTTXJzbGUvTnhoZnFTQSs0anNFdwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
etcd-key：填寫cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n' 的輸出
etcd-cert：填寫 cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n' 的輸出
etcd-ca：填寫cat /etc/kubernetes/ssl/ca.pem | base64 | tr -d '\n' 的輸出
- name: CALICO_IPV4POOL_CIDR value: "192.244.0.0/16"
CALICO_IPV4POOL_CIDR的value需要與初始化設置的 podSubnet: 192.244.0.0/16相同，不能和serviceSubnet重復。
# Auto-detect the BGP IP address. - name: IP value: "autodetect"
- name: IP_AUTODETECTION_METHOD value: "can-reach=8.8.8.8"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
value: "Off"
部署
kubectl create-f calico.yaml
刪除calico
kubectl delete -f calico.yaml modprobe -r ipip
ifconfig cali開頭的網卡 down
ip link delete cali開頭的網卡
etcdctl del --prefix /calico
QA：部署過程中可能會碰到如下錯誤：
2018-11-19 08:51:49.719 [INFO][8] startup.go 264: Early log level set to info 2018-11-19 08:51:49.719 [INFO][8] startup.go 284: Using HOSTNAME environment (lowercase) for node name 2018-11-19 08:51:49.719 [INFO][8] startup.go 292: Determined node name: ui 2018-11-19 08:51:49.720 [INFO][8] startup.go 105: Skipping datastore connection test 2018-11-19 08:51:49.721 [INFO][8] startup.go 365: Building new node resource Name="ui"
2018-11-19 08:51:49.721 [INFO][8] startup.go 380: Initialize BGP data 2018-11-19 08:51:49.722 [INFO][8] startup.go 582: Using autodetected IPv4 address on interface br-99ef8e85b719: 172.18.0.1/16
2018-11-19 08:51:49.722 [INFO][8] startup.go 450: Node IPv4 changed, will check for conflicts 2018-11-19 08:51:49.722 [WARNING][8] startup.go 879: Calico node 'ku' is already using the IPv4 address 172.18.0.1. 2018-11-19 08:51:49.722 [WARNING][8] startup.go 1104: Terminating Calico node failed to start
導致node節點的calico-node不能啟動，這個主要是ip地址沖突導致，因為calico里面有個參數IP_AUTODETECTION_METHOD默認值是first-found.官方解釋：
The method to use to autodetect the IPv4 address for this host. This is only used when the IPv4 address is being autodetected. See IP Autodetection methods for details of the valid methods. [Default: first-found]
https://docs.projectcalico.org/v3.4/reference/node/configuration
由於我們使用的ipv4，如果是ipv6的花還需要設置IP6_AUTODETECTION_METHOD。
這里的解決方式是在calico.yaml文件里面添加如下修改：
# Auto-detect the BGP IP address. - name: IP value: "autodetect"
- name: IP_AUTODETECTION_METHOD value: "can-reach=8.8.8.8"
重新部署calico即可解決。
QA: 之前使用calico+ipip的模式，升級到1.13.0的時候出現一個請求長度的限制，超過1400就會被拒絕掉，切換到BGP模式就解決了這個問題，所以直接使用BGP模式。
安裝calicoctl：
https://docs.projectcalico.org/v3.4/usage/calicoctl/install
我這里采用binary的方式安裝
curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.3.2/calicoctl
chmod +x calicoctl
我們之前是將數據存儲到etcd里面，所以為calicoctl創建如下配置文件：
# cat /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3 kind: CalicoAPIConfig metadata: spec: etcdEndpoints: https://10.42.140.154:2379,https://10.42.104.113:2379,https://10.42.177.142:2379
etcdKeyFile: /etc/etcd/ssl/etcd-key.pem etcdCertFile: /etc/etcd/ssl/etcd.pem etcdCACertFile: /etc/kubernetes/ssl/ca.pem
第二種flannel+host-gw：
下載文件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
需要作如下修改
vim kube-flannel.yml args: - --ip-masq - --kube-subnet-mgr - --iface=eth0
如果是自己定義的 podSubnet: 10.244.0.0/16 和serviceSubnet: 10.96.0.0/16，則還需要修改一個地方，修改為 podSubnet：
net-conf.json: | { "Network": "192.244.0.0/16", "Backend": { "Type": "host-gw" } }
部署
kubectl apply -f kube-flannel.yaml
刪除flannel
kubectl delete -f flannel.yaml ifconfig cni0 down ip link delete cni0 ifconfig flannel.1 down ip link delete flannel.1
rm -rf /var/lib/cni/
三、添加node節點
默認每個節點使用kubeadm初始化的時候都會創建一個token，可以選擇任意一個master節點的token來添加節點，但是這樣管理起來不是很方便。所一先把節點的所有token都刪掉，然后創建一個token
用來添加節點，token默認ttl是24小時，可以講ttl設置為0，這樣token就永久有效，不建議將ttl設置成0，設置成2天輪換一次。具體步驟如下：
1、刪掉原油默認的token
kubeadm tokne list 找到默認的token，然后都刪掉 kubeadm token delete l36ybs.04f5zkvewgtsd703 kubeadm token delete zvbdfg.9kx9fzozxfz8seko kubeadm token delete zvfoiw.r7p0rxtvgxc95owu 使用kubeadm token create生成token kubeadm token create --ttl 72h
為了方便使用，寫了一個腳本：
# cat uptk.sh #!/bin/bash tk=`kubeadm token list | awk '{print $1}' |grep -v TOKEN` kubeadm token delete $tk kubeadm token create --ttl 72h
crontab -e * * */2 * * /data/scripts/uptk.sh 2>&1 & >>/dev/null
關於kubeadm管理token更多細節參考官方文檔：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-token/
2、添加節點
准備一台新的服務器，做好免密交互，安裝kubeadm、kubelet、docker等，按照上面准備master那樣，只是不執行初始化，而是kubeadm join，注意版本要一致：
在master上操作：
查看集群的token kubeadm token list | grep authentication,signing | awk '{print $1}' 查看discovery-token-ca-cert-hash openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
為了方便使用將上述命令做成了一個生成腳本，需要在master節點上執行，當然可以根據服務器情況進行修改：
cat generate_kubeadm_join.sh #!/bin/bash #delete the invalid token and create new token tk_old=`kubeadm token list | awk '{print $1}' | grep -v TOKEN` kubeadm token delete $tk_old >>/dev/null 2>& 1 kubeadm token create --ttl 72h >>/dev/null 2>& 1 #print option which kubeadm join tk_new=`kubeadm token list | awk '{print $1}' | grep -v TOKEN` hs=`openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'` port=6443 ip=`ip a | grep eth0 | grep inet| awk -F / '{print $1}'| awk '{print $2}'` op="kubeadm join --token $tk_new $ip:$port --discovery-token-ca-cert-hash sha256:$hs"
echo $op
在node節點上操作：
根據上面獲取的token和hash kubeadm join --token c04f89.b781cdb55d83c1ef 10.42.9.210:6443 --discovery-token-ca-cert-hash sha256:986e83a9cb948368ad0552b95232e31d3b76e2476b595bd1d905d5242ace29af
在master上操作：
為node節點打標簽 kubectl label node knode1 node-role.kubernetes.io/node=
刪除節點：
在master節點上： kubectl drain node1 --delete-local-data --force --ignore-daemonsets kubectl delete node node1 在node節點上： kubeadm reset
ps：node1 是節點名稱
如果部署的有問題，可以使用如下的方法進行重置：
#kubeadm reset ifconfig cni0 down ip link delete cni0 ifconfig flannel.1 down ip link delete flannel.1 rm -rf /var/lib/cni/
rm -rf ~/.kube
ipvsadm --clear
如果使用的etcd是external的方式，也就是外部集群，使用kubeadm reset重置集群的時候並不能刪除etcd里面的數據，需要使用etcdctl工具手動刪除。
由於前面已經設置了alias，所以可以直接使用etcdctl：
etcdctl del "" --prefix
四、部署監控
1、部署metrics-server
git clone https://github.com/cuishuaigit/k8s-monitor.git
cd k8s-monitor/metrics-server
我都是把這種服務部署在master節點上面，此時需要修改metrics-server-deployment.yaml
--- apiVersion: v1 kind: ServiceAccount metadata: name: metrics-server namespace: kube-system --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: metrics-server namespace: kube-system labels: k8s-app: metrics-server spec: selector: matchLabels: k8s-app: metrics-server template: metadata: name: metrics-server labels: k8s-app: metrics-server spec: serviceAccountName: metrics-server tolerations: - effect: NoSchedule key: node.kubernetes.io/unschedulable operator: Exists - key: NoSchedule operator: Exists effect: NoSchedule volumes: # mount in tmp so we can safely use from-scratch images and/or read-only containers - name: tmp-dir emptyDir: {} containers: - name: metrics-server image: k8s.gcr.io/metrics-server-amd64:v0.3.1 imagePullPolicy: Always command: - /metrics-server - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP volumeMounts: - name: tmp-dir mountPath: /tmp nodeSelector: metrics: "yes"
為master添加label
kubectl label nodes ku metrics=yes
部署kubectl create -f metrics-server/deploy/1.8+/
如果有報錯參考我的另外一片博客：https://www.cnblogs.com/cuishuai/p/9857120.html
接下來稍等幾十秒就可以查看狀態了
2、部署prometheus
下載相關文件 使用上面clone的k8s-monitor部署
cd k8s-monitor
1.搭建nfs服務動態提供持久化存儲（如果已有可以跳過）
1.安裝nfs
sudo apt-get install -y nfs-kernel-server
sudo apt-get install -y nfs-common
sudo vi /etc/exports
/data/opv *(rw,sync,no_root_squash,no_subtree_check)
注意將*換成自己的ip段，純內網的話也可以用*，代替任意
sudo /etc/init.d/rpcbind restart
sudo /etc/init.d/nfs-kernel-server restart
sudo systemctl enable rpcbind nfs-kernel-server
客戶端掛在使用
sudo apt-get install -y nfs-common
mount -t nfs ku13-1:/data/opv /data/opv -o proto=tcp -o nolock
為了方便使用將上面的mount命令直接放到.bashrc里面
2.創建namesapce
kubectl creaet -f nfs/monitoring-namepsace.yaml
3.為nfs創建rbac
kubectl create -f nfs/rbac.yaml
4.創建deployment
kubectl create -f nfs/nfs-deployment.yaml
5.創建storageclass
kubectl create -f nfs/storageClass.yaml
2.安裝Prometheus
cd k8s-monitor/Promutheus/prometheus
1.創建權限
kubectl create -f rbac.yaml
2.創建 node-exporter
kubectl create -f prometheus-node-exporter-daemonset.yaml
kubectl create -f prometheus-node-exporter-service.yaml
3.創建 kube-state-metrics
kubectl create -f kube-state-metrics-deployment.yaml
kubectl create -f kube-state-metrics-service.yaml
4.創建 node-directory-size-metrics
kubectl create -f node-directory-size-metrics-daemonset.yaml
5.創建 prometheus
kubectl create -f prometheus-pvc.yaml
kubectl create -f prometheus-core-configmap.yaml
kubectl create -f prometheus-core-deployment.yaml
kubectl create -f prometheus-core-service.yaml
kubectl create -f prometheus-rules-configmap.yaml
6.修改core-configmap里的etcd地址
3.安裝Grafana
cd k8s-monitor/Promutheus/grafana
1.安裝grafana service
kubectl create -f grafana-svc.yaml
2.創建configmap
kubectl create -f grafana-configmap.yaml
3.創建pvc
kubectl create -f grafana-pvc.yaml
4.創建gragana deployment
kubectl create -f grafana-deployment.yaml
5.創建dashboard configmap
kubectl create configmap "grafana-import-dashboards" --from-file=dashboards/ --namespace=monitoring
6.創建job，導入dashboard等數據
kubectl create -f grafana-job.yaml
prometheus和grafana都是采用的nodePort方式暴漏的服務，所以可以直接訪問。
grafana默認的用戶名密碼：admin/admin