服務器規划和系統初始化
一、服務器規划
10.255.20.205 Master01 kube-apiserver、kube-controller-manager、kube-scheduler、ETCD 10.255.20.6 Master02 kube-apiserver、kube-controller-manager、kube-scheduler、ETCD 10255.20.242 Master03 kube-apiserver、kube-controller-manager-kube-scheduler、ETCD 10.255.20.117 Node01 kubelet、kube-proxy、docker 10.255.20.176 Node02 kubelet、kube-proxy、docker
二、系統初始化(所有節點全部執行)
關閉防火牆： # systemctl stop firewalld # systemctl disable firewalld 關閉selinux： # setenforce 0 # 臨時 # sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久 關閉swap： # swapoff -a # 臨時 # vim /etc/fstab # 永久 同步系統時間： # ntpdate time.windows.com 添加hosts： # vim /etc/hosts 10.255.20.205 master01 10.255.20.6 master02 10.255.20.242 master03 10.255.20.117 node01 10.255.20.176 node02 修改主機名： hostnamectl set-hostname node-name
2.安裝依賴和升級內核到5.x的最新版本
function Install_depend_environment(){ rpm -qa | grep nfs-utils &> /dev/null && echo -e "已完成依賴環境安裝，退出依賴環境安裝步驟 " && return yum install -y nfs-utils curl yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim ntpdate libseccomp libtool-ltdl telnet echo -e "升級Centos7系統內核到5版本，解決Docker-ce版本兼容問題" rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org && \ rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm && \ yum --disablerepo=\* --enablerepo=elrepo-kernel repolist && \ yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-ml.x86_64 && \ yum remove -y kernel-tools-libs.x86_64 kernel-tools.x86_64 && \ yum --disablerepo=\* --enablerepo=elrepo-kernel install -y kernel-ml-tools.x86_64 && \ grub2-set-default 0 modprobe br_netfilter cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sysctl -p /etc/sysctl.d/k8s.conf ls /proc/sys/net/bridge }
3.重啟機器之后,查看內核加載
[root@master03 ~]# ls /proc/sys/net/bridge bridge-nf-call-arptables bridge-nf-call-ip6tables bridge-nf-call-iptables bridge-nf-filter-pppoe-tagged bridge-nf-filter-vlan-tagged bridge-nf-pass-vlan-input-dev
ETCD集群安裝
一、生成ETCD證書(可以任選一個ETCD的節點)
# cd TLS/etcd/ #ls #默認下面有這幾個文件 ca-config.json ca-csr.json server-csr.json generate_etcd_cert.sh # ca-config.json 和 ca-csr.json 用於生成ca.pem ca-key.pem ca.csr # server-csr.json 用於生成server.pem server-key.pem server.csr # generate_etcd_cert.sh 這是生成ca和server的腳本
1.安裝cfssl工具
[root@master01 ~]# cd TLS [root@master01 TLS]# ls cfssl cfssl-certinfo cfssljson cfssl.sh etcd k8s [root@master01 TLS]# ./cfssl.sh
2.修改請求文件中hosts字段包含所有etcd節點IP
[root@master01 TLS]# cd etcd/ [root@master01 etcd]# vim server-csr.json { "CN": "etcd", "hosts": [ "10.255.20.205", #master1的IP "10.255.20.6", #master2的IP "10.255.20.242" #master3的IP ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] }
3.生成
[root@master01 etcd]# ./generate_etcd_cert.sh #可以執行這個腳本直接生成ca和server,也可以分開執行,下面演示分開執行
4.查看生成的ca和server
[root@master01 etcd]# ls *pem ca-key.pem ca.pem server-key.pem server.pem
二、部署三個ETCD節點
1.在一個節點部署好ETCD
[root@master01]# tar zxvf etcd.tar.gz [root@master01]# cd etcd [root@master01]# cp TLS/etcd/{ca,server,server-key}.pem ssl #證書 [root@master01]# cp -r etcd /opt
[root@master01] cp etcd.service /usr/lib/systemd/system
2.將master1上部署好的etcd分發到剩下兩個節點
[root@master01 ~]# scp –r etcd root@10.255.20.242:/opt [root@master01 ~]# scp etcd.service root@10.255.20.242:/usr/lib/systemd/system [root@master01 ~]# scp –r etcd root@10.255.20.6:/opt [root@master01 ~]# scp etcd.service root@10.255.20.6:/usr/lib/systemd/system
3.分別在3個節點修改etcd配置文件
主要是節點名稱和IP
vi /opt/etcd/cfg/etcd.conf #[Member] ETCD_NAME="master01" #每個節點都修改為自己的節點名稱 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://10.255.20.205:2380" #改為自己的IP ETCD_LISTEN_CLIENT_URLS="https://10.255.20.205:2379" #改為自己的IP #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.255.20.205:2380" #改為自己的IP ETCD_ADVERTISE_CLIENT_URLS="https://10.255.20.205:2379" #改為自己的IP ETCD_INITIAL_CLUSTER="master01=https://10.255.20.205:2380,master02=https://10.255.20.6:2380,master03=https://10.255.20.242:2380" #三個IP和端口 ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
4.啟動
# systemctl start etcd # systemctl enable etcd
5.查看集群健康狀態
# /opt/etcd/bin/etcdctl \ --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://10.255.20.205:2379,https://10.255.20.117:2379,https://10.255.20.176:2379" cluster-health member 37f20611ff3d9209 is healthy: got healthy result from https://10.255.20.205:2379 member b10f0bac3883a232 is healthy: got healthy result from https://10.255.20.6:2379 member b46624837acedac9 is healthy: got healthy result from https://10.255.20.242:2379 cluster is healthy
Master節點部署
一、生成apiserver證書(隨便再一個master節點操作,然后分發到另外兩個節點)
1.修改server-csr.json
需要提前把LB和所有master節點都寫上,如果后期需要加新master節點需要重新生成證書,在分發證書
[root@master01 k8s]# cd TLS/k8s [root@master01 k8s]# vi server-csr.json { "CN": "kubernetes", "hosts": [ "10.0.0.1", "127.0.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local", "10.255.20.205", #master1 ip "10.255.20.6", #master2 ip "10.255.20.242", #master3 ip "10.255.20.165" #滴滴雲內網LB的ip ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] }
2.生成證書
[root@master01 k8s]# ./generate_k8s_cert.sh [root@master01 k8s]# ls *pem ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem server-key.pem server.pem
二、部署kube-apiserver、kube-controller-manager、kube-scheduler
所有master組件的bin和cfg都在k8s-master里
1.拷貝證書和拷貝三個master組件的啟動文件
[root@master01]# tar zxvf k8s-master.tar.gz [root@master01]# cd k8s-master/kubernetes/ [root@master01 k8s-master]# tree kubernetes/ kubernetes/ ├── bin │ ├── kube-apiserver │ ├── kube-controller-manager │ ├── kubectl │ └── kube-scheduler ├── cfg │ ├── kube-apiserver.conf │ ├── kube-controller-manager.conf │ ├── kube-scheduler.conf │ └── token.csv ├── logs └── ssl [root@master01 kubernetes]# cp /root/TLS/k8s/*.pem ssl [root@master01 kubernetes]# cd .. [root@master01 k8s-master]# cp kubernetes /opt [root@master01 k8s-master]# cp kube-apiserver.service kube-controller-manager.service kube-scheduler.service /usr/lib/systemd/system
2.修改apiserver相關配置文件
[root@master01 k8s-master]# cd kubernetes/cfg/ [root@master01 cfg] vim kube-apiserver.conf KUBE_APISERVER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --etcd-servers=https://10.255.20.205:2379,https://10.255.20.117:2379,https://10.255.20.176:2379 \ #ETCD集群鏈接 --bind-address=10.255.20.205 \ #本機IP --secure-port=6443 \ --advertise-address=10.255.20.205 \ #本機IP --allow-privileged=true \ --service-cluster-ip-range=10.0.0.0/24 \ #service的clusterIP地址段, 跟kube-controller-manager.conf和kube-proxy-config.yml對應 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \ --authorization-mode=RBAC,Node \ --enable-bootstrap-token-auth=true \ --token-auth-file=/opt/kubernetes/cfg/token.csv \ --service-node-port-range=30000-32767 \ --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \ --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \ --tls-cert-file=/opt/kubernetes/ssl/server.pem \ --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \ --client-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \ --etcd-cafile=/opt/etcd/ssl/ca.pem \ --etcd-certfile=/opt/etcd/ssl/server.pem \ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
3.修改kube-controller-manager相關配置文件
[root@master01 cfg]vim kube-controller-manager.conf KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect=true \ --master=127.0.0.1:8080 \ #鏈接api-server,為本地不安全地址 --address=127.0.0.1 \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ #Pod的IP地址段 --service-cluster-ip-range=10.0.0.0/24 \ #service clusterIP段,跟kube-apiserver.conf和kube-proxy-config.yaml對應 --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \ --root-ca-file=/opt/kubernetes/ssl/ca.pem \ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \ --experimental-cluster-signing-duration=87600h0m0s"
4.修改kube-scheduler相關配置文件
[root@master01 cfg] vim kube-scheduler.conf KUBE_SCHEDULER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --leader-elect \ --master=127.0.0.1:8080 \ #鏈接apiserver --address=127.0.0.1"
5.啟動和開機自啟master組件
# systemctl start kube-apiserver # systemctl start kube-controller-manager # systemctl start kube-scheduler # systemctl enable kube-apiserver # systemctl enable kube-controller-manager # systemctl enable kube-scheduler
6.分發master相關組件到其他兩個節點(kubernetes目錄和啟動文件)
#在准備好master的master01上操作 [root@master01] scp –r /opt/kubernetes root@10.255.20.6:/opt [root@master01] scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@10.255.20.6:/usr/lib/systemd/system
7.在其他兩個節點修改apiserver的配置文件為本機IP,並啟動和開機自啟相關master組件
#####修改剩下兩個節點apiserver的配置文件為本地IP##### #在剩下兩個節點操作 vim /opt/kubernetes/cfg/kube-apiserver.conf KUBE_APISERVER_OPTS="--logtostderr=false \ --v=2 \ --log-dir=/opt/kubernetes/logs \ --etcd-servers=https://10.255.20.205:2379,https://10.255.20.6:2379,https://10.255.20.242:2379 \ --bind-address=10.255.20.6 \ --secure-port=6443 \ --advertise-address=10.255.20.6 \
# systemctl start kube-apiserver # systemctl start kube-controller-manager # systemctl start kube-scheduler # systemctl enable kube-apiserver # systemctl enable kube-controller-manager # systemctl enable kube-scheduler
三、啟用TLS bootstrapping
1.為kubelet TLB bootstrapping授權
[root@master01]cat /opt/kubernetes/cfg/token.csv c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
2.給kubelet-bootstrap授權(一個master節點執行就可以)
[root@master01]kubectl create clusterrolebinding kubelet-bootstrap \ --clusterrole=system:node-bootstrapper \ --user=kubelet-bootstrap
注意:token可以自己生成替換
head -c 16 /dev/urandom | od -An -t x | tr -d ' ' #但apiserver配置的token必須要與node節點bootstrap.kubeconfig配置里一致。
master的kube-apiserver.conf中的配置 --token-auth-file=/opt/kubernetes/cfg/token.csv
node的 bootstrap.kubeconfig里 token: c47ffb939f5ca36231d9e3121a252940就是apiserver.conf里token.csv的內容
部署Node節點相關組件
一、安裝docker(兩個node節點)
二進制包下載地址：https://download.docker.com/linux/static/stable/x86_64/
[root@node01 ]# tar zxvf k8s-node.tar.gz [root@node01 ]# tar zxvf docker-18.09.6.tgz [root@node01 ]# mv docker/* /usr/bin [root@node01 ]# mkdir /etc/docker [root@node01 ]# mv daemon.json /etc/docker [root@node01 ]# mv docker.service /usr/lib/systemd/system [root@node01 ]# systemctl start docker [root@node01 ]# systemctl enable docker
二、部署kubelet和kube-proxy
1.拷貝工作目錄和啟動配置文件(兩個node節點)
[root@node01]# tree kubernetes kubernetes/ ├── bin │ ├── kubelet │ └── kube-proxy ├── cfg │ ├── bootstrap.kubeconfig │ ├── kubelet.conf │ ├── kubelet-config.yml │ ├── kube-proxy.conf │ ├── kube-proxy-config.yml │ └── kube-proxy.kubeconfig ├── logs └── ssl [root@node01] # mv kubernetes /opt [root@node01] # cp kubelet.service kube-proxy.service /usr/lib/systemd/system
2.從master上拷貝2node節點需要的證書
[root@master01]# cd TLS/k8s [root@master01 k8s]# scp ca.pem kube-proxy*.pem root@10.255.20.117:/opt/kubernetes/ssl/
[root@master01 k8s]# scp ca.pem kube-proxy*.pem root@10.255.20.176:/opt/kubernetes/ssl/
3.在兩個node節點修改配置文件中apiserver的IP
為啥3個master節點只寫了一個節點的IP呢,因為6443是apiserver的IP,用了滴滴雲的負載均衡,給master做了高可用,滴滴雲LB的6443轉發到3台master節點的6443,生成kubernetes證書的時候也把滴滴雲的負載均衡IP加進去了 10.255.20.165
bootstrap.kubeconfig: server: https://10.255.20.165:6443
kube-proxy.kubeconfig: server: https://10.255.20.165:6443
4.在兩個node節點修改配置文件中自己的主機名
kubelet.conf:--hostname-override=node01 kube-proxy-config.yml:hostnameOverride: node01 kubelet.conf:--hostname-override=node02 kube-proxy-config.yml:hostnameOverride: node02 #上面的主機名,是注冊到master顯示的名稱
5.啟動kubelet和kube-proxy
# systemctl start kubelet # systemctl start kube-proxy # systemctl enable kubelet # systemctl enable kube-proxy
注意:node節點組件啟動的時候,就向master申請證書了,需要到master去頒發下證書
6.在master上給兩個node頒發證書
[root@master01 ]# kubectl get csr [root@master01 ]# kubectl certificate approve node-csr-MYUxbmf_nmPQjmH3LkbZRL2uTO-_FCzDQUoUfTy7YjI [root@master01 ]# kubectl get node NAME STATUS ROLES AGE VERSION node01 NotReady <none> 18h v1.16.0 node02 NotReady <none> 18h v1.16.0 ###為啥是notReady呢，因為還沒有cni網絡插件
7.部署CNI網絡插件
二進制包下載地址：https://github.com/containernetworking/plugins/releases
7.1.1每個node上部署CNI插件包和創建CNI插件目錄
# mkdir /opt/cni/bin /etc/cni/net.d -p # tar zxvf cni-plugins-linux-amd64-v0.8.2.tgz -C /opt/cni/bin
7.1.2確保kubelet啟動CNI
# cat /opt/kubernetes/cfg/kubelet.conf --network-plugin=cni
7.1.3在master上部署flannel
[root@master ] # kubectl apply -f kube-flannel.yaml [root@master ] # kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE kube-flannel-ds-amd64-5xmhh 1/1 Running 6 171m kube-flannel-ds-amd64-ps5fx 1/1 Running 0 150m 注意:kube-flannel.yaml 里 net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } } ##上面這段配置中10.244.0.0/16必須得跟kube-controller-manager.conf 里--cluster-cidr=10.244.0.0/16 這段配置的網段一樣
注意:flannel在每個node上都會啟動一個容器
8.授權apiserver訪問kubelet
為提供安全性，kubelet禁止匿名訪問，必須授權才可以,授權之后才能查看pod日志之類的。
[root@master]# cat /opt/kubernetes/cfg/kubelet-config.yml …… authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /opt/kubernetes/ssl/ca.pem …… [root@master]# kubectl apply –f apiserver-to-kubelet-rbac.yaml
部署WebUI和DNS
一、部署WebUI
https://kubernetes.io/docs/tasks/access-application-clu ster/web-ui-dashboard/
[root@master]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml # vi recommended.yaml … kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30001 selector: k8s-app: kubernetes-dashboard … [root@master]# kubectl apply -f recommended.yaml
1.創建service account並綁定默認cluster-admin管理員集群角色
[root@master]# cat dashboard-adminuser.yaml apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-user namespace: kubernetes-dashboard
2.獲取token
[root@master]# kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
訪問地址：http://NodeIP:30001
使用輸出的token登錄Dashboard
二、部署DNS
[root@master]# kubectl apply –f coredns.yaml [root@master]# kubectl get pods -n kube-system