一、集群環境准備
本文系搭建kubernetes v1.18.6 集群筆記,使用三台虛擬機作為 CentOS 測試機,安裝kubeadm、kubelet、kubectl均使用yum安裝,網絡組件選用的是 flannel行文中難免出現錯誤,如果讀者有高見,請評論與我交流、如需轉載請注明原始出處:https://www.cnblogs.com/luoahong/p/13432410.html
部署集群沒有特殊說明均使用root用戶執行命令
1、硬件信息
root@master ~]# lscpu ...... CPU(s): 4 CPU MHz: 2397.220 Hypervisor vendor: KVM [root@master ~]# free -h total used free shared buff/cache available Mem: 17G 1.0G 13G 17M 2.6G 16G [root@master ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom vda 253:0 0 100G 0 disk
2、軟件信息
[root@master ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) [root@master ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6" [root@master ~]# docker version Client: Docker Engine - Community Version: 19.03.12
3、保證環境正確性
purpose | commands |
---|---|
保證集群各節點互通 | ping -c 3 <ip> |
保證MAC地址唯一 | ip link 或 ifconfig -a |
保證集群內主機名唯一 | 查詢 hostnamectl status ,修改 hostnamectl set-hostname <hostname> |
保證系統產品uuid唯一 | dmidecode -s system-uuid 或 sudo cat /sys/class/dmi/id/product_uuid |
4、確保端口開放正常
kube-master節點端口檢查:
[root@master ~]# netstat -lntup Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 :::10250 :::* LISTEN 14154/kubelet tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 14494/kube-proxy tcp6 0 0 :::10256 :::* LISTEN 14494/kube-proxy tcp 0 0 192.168.118.4:2379 0.0.0.0:* LISTEN 13805/etcd tcp 0 0 192.168.118.4:2380 0.0.0.0:* LISTEN 13805/etcd tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN 13817/kube-controll tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN 13877/kube-schedule tcp6 0 0 :::6443 :::* LISTEN 13755/kube-apiserve
kube-node*節點端口檢查:
[root@node1 ~]# netstat -lntup Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name xy tcp 0 0 0.0.0.0:30443 0.0.0.0:* LISTEN 13294/kube-proxy tcp 0 0 0.0.0.0:30964 0.0.0.0:* LISTEN 13294/kube-proxy tcp6 0 0 :::10250 :::* LISTEN 12951/kubelet tcp6 0 0 :::10256 :::* LISTEN 13294/kube-proxy
二、環境初始化(所有節點執行)
1、配置主機互信
1、分別在各節點配置hosts映射:
cat >> /etc/hosts <<EOF 192.168.118.4 master 192.168.118.19 node1 192.168.118.20 node2 EOF
2、master生成ssh密鑰,分發公鑰到各節點:
#生成ssh密鑰,直接一路回車 ssh-keygen -t rsa #復制剛剛生成的密鑰到各節點可信列表中,需分別輸入各主機密碼 ssh-copy-id /root/.ssh/id_rsa.pub master ssh-copy-id /root/.ssh/id_rsa.pub node1 ssh-copy-id /root/.ssh/id_rsa.pub node2
2、禁用swap
swapoff -a sed -i 's/.*swap.*/#&/' /etc/fstab三、部署docker
三、部署docker(所有節點執行)
1、添加docker yum源
#安裝必要依賴 yum install -y yum-utils device-mapper-persistent-data lvm2 #添加aliyun docker-ce yum源 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo #重建yum緩存 yum makecache fast
2、安裝docker
#查看可用docker版本 yum list docker-ce.x86_64 --showduplicates | sort -r #安裝指定版本docker yum install -y docker-ce-19.03.12-3.el7
3、確保網絡模塊開機自動加載
lsmod | grep overlay lsmod | grep br_netfilter
若上面命令無返回值輸出或提示文件不存在,需執行以下命令:
cat > /etc/modules-load.d/docker.conf <<EOF overlay br_netfilter EOF modprobe overlay modprobe br_netfilter
4、使橋接流量對iptables可見
cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system
驗證是否生效,均返回 1
即正確
sysctl -n net.bridge.bridge-nf-call-iptables sysctl -n net.bridge.bridge-nf-call-ip6tables
5、配置docker
mkdir /etc/docker #修改cgroup驅動為systemd[k8s官方推薦]、限制容器日志量、修改存儲類型,最后的docker家目錄可修改 cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ], "registry-mirrors": ["https://7uuu3esz.mirror.aliyuncs.com"], "data-root": "/data/docker" } EOF #添加開機自啟,立即啟動 systemctl enable --now docker
6、驗證docker是否正常
#查看docker信息,判斷是否與配置一致 docker info #hello-docker測試 docker run --rm hello-world #刪除測試image docker rmi hello-world
四、部署kubernetes集群
未特殊說明,各節點均需執行如下步驟
1、添加kubernetes源
cat > /etc/yum.repos.d/kubernetes.repo <<EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF #重建yum緩存,輸入y添加證書認證 yum makecache fast
2、安裝kubeadm、kubelet、kubectl
各節點均需安裝kubeadm、kubelet,kubectl僅kube-master節點需安裝(作為worker節點,kubectl無法使用,可以不裝)
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes systemctl enable --now kubelet
3、配置自動補全命令
#安裝bash自動補全插件 yum install bash-completion -y #設置kubectl與kubeadm命令補全,下次login生效 kubectl completion bash >/etc/bash_completion.d/kubectl kubeadm completion bash > /etc/bash_completion.d/kubeadm
4、預拉取kubernetes鏡像
#!/bin/bash KUBE_VERSION=v1.18.6 PAUSE_VERSION=3.2 CORE_DNS_VERSION=1.6.7 ETCD_VERSION=3.4.3-0 # pull kubernetes images from hub.docker.com docker pull kubeimage/kube-proxy-amd64:$KUBE_VERSION docker pull kubeimage/kube-controller-manager-amd64:$KUBE_VERSION docker pull kubeimage/kube-apiserver-amd64:$KUBE_VERSION docker pull kubeimage/kube-scheduler-amd64:$KUBE_VERSION # pull aliyuncs mirror docker images docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$CORE_DNS_VERSION docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:$ETCD_VERSION # retag to k8s.gcr.io prefix docker tag kubeimage/kube-proxy-amd64:$KUBE_VERSION k8s.gcr.io/kube-proxy:$KUBE_VERSION docker tag kubeimage/kube-controller-manager-amd64:$KUBE_VERSION k8s.gcr.io/kube-controller-manager:$KUBE_VERSION docker tag kubeimage/kube-apiserver-amd64:$KUBE_VERSION k8s.gcr.io/kube-apiserver:$KUBE_VERSION docker tag kubeimage/kube-scheduler-amd64:$KUBE_VERSION k8s.gcr.io/kube-scheduler:$KUBE_VERSION docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION k8s.gcr.io/pause:$PAUSE_VERSION docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$CORE_DNS_VERSION k8s.gcr.io/coredns:$CORE_DNS_VERSION docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:$ETCD_VERSION k8s.gcr.io/etcd:$ETCD_VERSION # untag origin tag, the images won't be delete. docker rmi kubeimage/kube-proxy-amd64:$KUBE_VERSION docker rmi kubeimage/kube-controller-manager-amd64:$KUBE_VERSION docker rmi kubeimage/kube-apiserver-amd64:$KUBE_VERSION docker rmi kubeimage/kube-scheduler-amd64:$KUBE_VERSION docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$CORE_DNS_VERSION docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:$ETCD_VERSION
腳本添加可執行權限,執行腳本拉取鏡像:
由於網絡原因建議在早上7點前執行速度更佳、其他時段速度很慢甚至連接超時
chmod +x get-k8s-images.sh ./get-k8s-images.sh
拉取完成,執行 docker images
查看鏡像
[root@master ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.18.6 c3d62d6fe412 2 weeks ago 117MB k8s.gcr.io/kube-controller-manager v1.18.6 ffce5e64d915 2 weeks ago 162MB k8s.gcr.io/kube-apiserver v1.18.6 56acd67ea15a 2 weeks ago 173MB k8s.gcr.io/kube-scheduler v1.18.6 0e0972b2b5d1 2 weeks ago 95.3MB k8s.gcr.io/pause 3.2 80d28bedfe5d 5 months ago 683kB k8s.gcr.io/coredns 1.6.7 67da37a9a360 6 months ago 43.8MB k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 9 months ago 288MB
五、初始化master(僅 master 節點需要執行此步驟)
1、修改kubelet配置默認cgroup driver
cat > /var/lib/kubelet/config.yaml <<EOF apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd EOF systemctl restart kubelet
2、初始化master 10.244.0.0/16是flannel固定使用的IP段,設置取決於網絡組件要求
[root@master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.18.6 W0803 23:20:21.320111 12805 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.6 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.118.4] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [192.168.118.4 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [192.168.118.4 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" W0803 23:20:28.237080 12805 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-scheduler" W0803 23:20:28.238090 12805 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 20.503032 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: aw0koc.6d40t5a2ydm299c9 [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.118.4:6443 --token aw0koc.6d40t5a2ydm299c9 \ --discovery-token-ca-cert-hash sha256:38343d02ddd645b2f74ddf886925c93115604ea72a01b0b03088ca1d2ac14c6f
3、為日常使用集群的用戶添加kubectl使用權限
[root@master ~]#mkdir -p $HOME/.kube [root@master ~]#sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@master ~]#sudo chown $(id -u):$(id -g) $HOME/.kube/config [root@master ~]# echo "export KUBECONFIG=$HOME/.kube/admin.conf" >> ~/.bashrc
4、配置master認證
[root@master ~]# echo 'export KUBECONFIG=/etc/kubernetes/admin.conf' >> /etc/profile [root@master ~]# . /etc/profile
如果不配置這個,會提示如下輸出:
The connection to the server localhost:8080 was refused - did you specify the right host or port?
此時master節點已經初始化成功,但是還未完裝網絡組件,還無法與其他節點通訊
[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master NotReady master 81s v1.18.6
5、安裝網絡組件,以flannel為例
由於網絡原因建議在早上7點前執行速度更佳、其他時段速度很慢甚至連接超時
創建運行
[root@master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml [root@master ~]# kubectl apply -f kube-flannel.yml
查看是否都正常運行
[root@master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-66bff467f8-d9xjc 1/1 Running 0 2m22s coredns-66bff467f8-lvldb 1/1 Running 0 2m22s etcd-master 1/1 Running 0 2m34s kube-apiserver-master 1/1 Running 0 2m34s kube-controller-manager-master 1/1 Running 0 2m34s kube-proxy-lg58q 1/1 Running 0 2m22s kube-scheduler-master 1/1 Running 0 2m33s
查看kube-master節點狀態
[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready master 12h v1.18.6
如果STATUS提示NotReady
,可以通過 kubectl describe node master
查看具體的描述信息,性能差的服務器到達Ready狀態時間會長些
六、初始化node*節點並加入集群
1、備份鏡像供其他節點使用
便於后續傳輸給其他node節點,當然有鏡像倉庫更好
docker save k8s.gcr.io/kube-proxy:v1.18.6 \ k8s.gcr.io/kube-apiserver:v1.18.6 \ k8s.gcr.io/kube-controller-manager:v1.18.6 \ k8s.gcr.io/kube-scheduler:v1.18.6 \ k8s.gcr.io/pause:3.2 \ k8s.gcr.io/coredns:1.6.7 \ k8s.gcr.io/etcd:3.4.3-0 > k8s-imagesV1.18.6.tar
拷貝鏡像到node節點
[root@master ~]#scp k8s-imagesV1.18.6.tar node1 [root@master ~]#scp k8s-imagesV1.18.6.tar node2
2、node節點載入鏡像
docker load < k8s-imagesV1.18.6.tar [root@node1 ~]#docker load < k8s-imagesV1.18.6.tar 225df95e717c: Loading layer [==================================================>] 336.4kB/336.4kB c965b38a6629: Loading layer [==================================================>] 43.58MB/43.58MB Loaded image: k8s.gcr.io/coredns:1.6.7 fe9a8b4f1dcc: Loading layer [==================================================>] 43.87MB/43.87MB ce04b89b7def: Loading layer [==================================================>] 224.9MB/224.9MB 1b2bc745b46f: Loading layer [==================================================>] 21.22MB/21.22MB Loaded image: k8s.gcr.io/etcd:3.4.3-0 82a5cde9d9a9: Loading layer [==================================================>] 53.87MB/53.87MB a2b38eae1b39: Loading layer [==================================================>] 21.62MB/21.62MB f378e9487360: Loading layer [==================================================>] 5.168MB/5.168MB a35a0b8b55f5: Loading layer [==================================================>] 4.608kB/4.608kB dea351e760ec: Loading layer [==================================================>] 8.192kB/8.192kB d57a645c2b0c: Loading layer [==================================================>] 8.704kB/8.704kB 602805206b58: Loading layer [==================================================>] 38.39MB/38.39MB Loaded image: k8s.gcr.io/kube-proxy:v1.18.6 2d99d0f31eb7: Loading layer [==================================================>] 120.7MB/120.7MB Loaded image: k8s.gcr.io/kube-apiserver:v1.18.6 82d47bbb60b8: Loading layer [==================================================>] 110.1MB/110.1MB Loaded image: k8s.gcr.io/kube-controller-manager:v1.18.6 80eec301f276: Loading layer [==================================================>] 42.96MB/42.96MB Loaded image: k8s.gcr.io/kube-scheduler:v1.18.6 ba0dae6243cc: Loading layer [==================================================>] 684.5kB/684.5kB Loaded image: k8s.gcr.io/pause:3.2
3、獲取加入kubernetes命令
剛才在初始化master節點時,有在最后輸出其加入集群的命令,假如我沒記下來,那怎么辦呢?
訪問kube-master輸入創建新token命令,同時輸出加入集群的命令:
[root@master ~]# kubeadm token create --print-join-command W0804 11:51:54.344223 11517 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] kubeadm join 192.168.118.4:6443 --token eynx2u.ph1ohakkqx1utkl8 --discovery-token-ca-cert-hash sha256:38343d02ddd645b2f74ddf886925c93115604ea72a01b0b03088ca1d2ac14c6f
和初始化的集群命令對比
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
#建議記錄下來便於以后添加新的node使用
kubeadm join 192.168.118.4:6443 --token aw0koc.6d40t5a2ydm299c9 \
--discovery-token-ca-cert-hash sha256:38343d02ddd645b2f74ddf886925c93115604ea72a01b0b03088ca1d2ac14c6f
#以下是通過kubeadm token create --print-join-command命令獲實時取的
kubeadm join 192.168.118.4:6443 --token eynx2u.ph1ohakkqx1utkl8 --discovery-token-ca-cert-hash sha256:38343d02ddd645b2f74ddf886925c93115604ea72a01b0b03088ca1d2ac14c6f
4、在node*節點上執行加入集群命令
[root@node1 ~]# kubeadm join 192.168.118.4:6443 --token aw0koc.6d40t5a2ydm299c9 \ > --discovery-token-ca-cert-hash sha256:38343d02ddd645b2f74ddf886925c93115604ea72a01b0b03088ca1d2ac14c6f W0803 23:41:00.380582 12778 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
5、查看集群節點狀態
[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready master 12h v1.18.6 node1 Ready <none> 11h v1.18.6 node2 Ready <none> 11h v1.18.6