kubeadm快速部署一個k8s集群
1. 前置知識點
1.1 生產環境可部署kubernetes集群的兩種方式
-
目前生產部署Kubernetes集群主要有兩種方式
-
kuberadm
Kubeadm是一個K8s部署工具，提供kubeadm init和kubeadm join，用於快速部署Kubernetes集群。
-
二進制包
從github下載發行版的二進制包，手動部署每個組件，組成Kubernetes集群。
Kubeadm降低部署門檻，但屏蔽了很多細節，遇到問題很難排查。如果想更容易可控，推薦使用二進制包部署Kubernetes集群，雖然手動部署麻煩點，期間可以學習很多工作原理，也利於后期維護。
-
1.2 安裝要求
- 在開始之前，部署Kubernetes集群機器需要滿足以下幾個條件：
- 一台或多台機器，操作系統 CentOS7.x-86_x64
- 硬件配置：2GB或更多RAM，2個CPU或更多CPU，硬盤30GB或更多
- 集群中所有機器之間網絡互通
- 可以訪問外網，需要拉取鏡像，如果服務器不能上網，需要提前下載鏡像並導入節點
- 禁止swap分區
1.3 環境准備
1.3.1 軟件環境
| 軟件 | 版本 |
|---|---|
| 操作系統 | CentOS7.8_x64 （mini） |
| Docker | 19-ce |
| Kubernetes | 1.18 |
1.3.2 單master架構圖
1.3.3 服務器整體規划
| 角色 | IP | 組件 |
|---|---|---|
| k8s-master | 192.168.0.201 | kube-apiserver，kube-controller-manager，kube-scheduler，etcd |
| k8s-node1 | 192.168.0.202 | kubelet，kube-proxy，docker etcd |
| k8s-node2 | 192.168.0.203 | kubelet，kube-proxy，docker，etcd |
| 鏡像倉庫 | 192.168.0.229 | jenkins，harbor，docker，git，maven， |
1.4 操作系統初始化配置
# 關閉防火牆
systemctl stop firewalld
systemctl disable firewalld
# 關閉selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
setenforce 0 # 臨時
# 關閉swap
swapoff -a # 臨時
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
# 根據規划設置主機名
hostnamectl set-hostname <hostname>
# 在master添加hosts
cat >> /etc/hosts << EOF
192.168.0.201 k8s-master
192.168.0.202 k8s-node1
192.168.0.203 k8s-node2
192.168.0.229 harbor.scajy.org
EOF
# 將橋接的IPv4流量傳遞到iptables的鏈
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system # 生效
# 時間同步
yum install ntpdate -y
ntpdate time.windows.com
2. 部署Etcd集群
Etcd 是一個分布式鍵值存儲系統，Kubernetes使用Etcd進行數據存儲，所以先准備一個Etcd數據庫，為解決Etcd單點故障，應采用集群方式部署，這里使用3台組建集群，可容忍1台機器故障，當然，你也可以使用5台組建集群，可容忍2台機器故障。
| 節點名稱 | IP |
|---|---|
| etcd-1 | 192.168.0.201 |
| etcd-2 | 192.168.0.202 |
| etcd-3 | 192.168.0.203 |
注：為了節省機器，這里與K8s節點機器復用。也可以獨立於k8s集群之外部署，只要apiserver能連接到就行。
3. 安裝docker【所有節點】
Kubernetes默認CRI（容器運行時）為Docker，因此先安裝Docker。
3.1 安裝docker
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum -y install docker-ce
systemctl enable docker && systemctl start docker
3.2 配置docker鏡像下載加速器
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
3.3 安裝docker的tab補全功能
yum install -y bash-completion
3.4 重啟docker
systemctl daemon-reload && systemctl restart docker
4. 安裝kubeadm/kubelet 【所有節點】
4.1 配置添加阿里雲YUM軟件源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
4.2 安裝kubeadm，kubelet和kubectl
-
由於版本更新頻繁，這里指定版本號安裝
yum install -y kubelet-1.19.0 kubeadm-1.19.0 kubectl-1.19.0 systemctl enable kubelet
5. 部署kubernetes Master
5.1 在master上執行
kubeadm init \
--apiserver-advertise-address=192.168.0.201 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.19.0 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=all
- --apiserver-advertise-address 集群通告地址
- --image-repository 由於默認拉取鏡像地址k8s.gcr.io國內無法訪問，這里指定阿里雲鏡像倉庫地址
- --kubernetes-version K8s版本，與上面安裝的一致
- --service-cidr 集群內部虛擬網絡，Pod統一訪問入口
- --pod-network-cidr Pod網絡，，與下面部署的CNI網絡組件yaml中保持一致
5.2 執行配置
[root@k8s-master ~]# kubeadm init \
> --apiserver-advertise-address=192.168.0.201 \
> --image-repository registry.aliyuncs.com/google_containers \
> --kubernetes-version v1.19.0 \
> --service-cidr=10.96.0.0/12 \
> --pod-network-cidr=10.244.0.0/16 \
> --ignore-preflight-errors=all
W1125 15:27:14.705112 21592 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.0
[preflight] Running pre-flight checks
[WARNING NumCPU]: the number of available CPUs 1 is less than the required 2
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.201]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.201 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.201 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.504596 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: f71sjf.y17zsxdsnm7pft5l
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.201:6443 --token f71sjf.y17zsxdsnm7pft5l \
--discovery-token-ca-cert-hash sha256:2a1dcf2140a80f526be1858aebb140037565c5a1fdf52e190fb4e3074d9c7290
5.3 拷貝kubectl使用的連接k8s認證文件到默認路徑：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
6. 加入kubernetes node節點
在所有node節點上執行，向集群添加新節點，執行在kubeadm init輸出的kubeadm join命令：
kubeadm join 192.168.0.201:6443 --token f71sjf.y17zsxdsnm7pft5l \
--discovery-token-ca-cert-hash sha256:2a1dcf2140a80f526be1858aebb140037565c5a1fdf52e190fb4e3074d9c7290
7. 部署容器網絡（CNI）
注意：只需要部署下面其中一個，推薦Calico。
Calico是一個純三層的數據中心網絡方案，Calico支持廣泛的平台，包括Kubernetes、OpenStack等。
Calico 在每一個計算節點利用 Linux Kernel 實現了一個高效的虛擬路由器（ vRouter） 來負責數據轉發，而每個 vRouter 通過 BGP 協議負責把自己上運行的 workload 的路由信息向整個 Calico 網絡內傳播。
此外，Calico 項目還實現了 Kubernetes 網絡策略，提供ACL功能。
https://docs.projectcalico.org/getting-started/kubernetes/quickstart
7.1 下載容器網絡配置
wget https://docs.projectcalico.org/manifests/calico.yaml
下載完后還需要修改里面定義
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16" # 這里的IP為前面的kubeadm初始化里面的pod IP
7.2 部署網絡
[root@k8s-master ~]# kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created
7.3 查看部署
[root@k8s-master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-5dc87d545c-nscfb 1/1 Running 0 2m12s
calico-node-j6rhw 0/1 Running 0 2m12s
calico-node-n7d6s 1/1 Running 0 2m11s
calico-node-x86s2 0/1 Running 0 2m11s
coredns-6d56c8448f-hkgnk 1/1 Running 0 156m
coredns-6d56c8448f-jfbjs 1/1 Running 0 156m
etcd-k8s-master 1/1 Running 0 157m
kube-apiserver-k8s-master 1/1 Running 0 157m
kube-controller-manager-k8s-master 1/1 Running 0 157m
kube-proxy-fhgbd 1/1 Running 0 99m
kube-proxy-l7q4r 1/1 Running 0 156m
kube-proxy-qwpjp 1/1 Running 0 99m
kube-scheduler-k8s-master 1/1 Running 0 157m
8. 部署dashboard
8.1 下載dashboard配置文件
-
下載配置文件
[root@k8s-master ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.3/aio/deploy/recommended.yaml
8.2 修改dashboard配置文件，暴露對外端口
-
修改默認dashboard只能集群內部訪問，修改service為nodePort類型，暴露到外部：
vi recommended.yaml ... #... 代表隱藏 kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: ports: - port: 443 targetPort: 8443 nodePort: 30001 selector: k8s-app: kubernetes-dashboard type: NodePort ...
8.3 啟動dashboard服務
[root@k8s-master ~]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
The Service "kubernetes-dashboard" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
8.4 查看服務
[root@k8s-master ~]# kubectl get pods -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-7b59f7d4df-fbkg6 1/1 Running 0 85s
kubernetes-dashboard-5dbf55bd9d-spd9q 1/1 Running 0 87s
8.5 瀏覽器訪問
- 訪問地址：https://NodeIP:30001
8.6 配置用戶角色
-
創建service account並綁定默認cluster-admin管理員集群角色：
# 創建用戶 kubectl create serviceaccount dashboard-admin -n kube-system # 用戶授權 kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin # 獲取用戶Token kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}') -
執行角色權限，
[root@k8s-master ~]# kubectl create serviceaccount dashboard-admin -n kube-system serviceaccount/dashboard-admin created [root@k8s-master ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created [root@k8s-master ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}') Name: dashboard-admin-token-ckqml Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name: dashboard-admin kubernetes.io/service-account.uid: 6c7b07d3-d3e3-4cd6-ad37-4b60de7c1a3d Type: kubernetes.io/service-account-token Data ==== token: eyJhbGciOiJSUzI1NiIsImtpZCI6InhnSERNcGRHaDV1TXBmay1SQ2NIenlLVHRlSi1uNTV0TnBBMHhHSFN6YmcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tY2txbWwiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNmM3YjA3ZDMtZDNlMy00Y2Q2LWFkMzctNGI2MGRlN2MxYTNkIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.czvP9MCh-uki7xWfrGfCt7IMNVYr1oNd2r10phUS8JFu4pMUmjGSq2M5mff_J-i5CmlC7tIzZRC1ZsA4qi9E00hedcl2zJ523Kiup_EcEUt-X-B5iCOgeNY38cc_f5u61_4y2jSmqayalTHy2ICX7_2aN8gQcC1r1iUf2Gy-Snm3Z9zsuZZxYD9VnUe0CLwiBCqYETn43vYvQx-1D6dKNqFWxNbDEcJWlLYbYTqKx0fNsTCucgXdH3pJzJ5Qx_3FgKPZO9Tjub-pkkFqsMD0S0WbFUxAwkyjBj1PK4DgoAwGcE2VS1wNu_bhbM7M4izIv6jz8wcn1cwWJfvAkfWGbg ca.crt: 1066 bytes namespace: 11 bytes
9. 集群基本驗證
- 驗證pod工作
- 驗證pod網絡通信
- 驗證DNS解析
在kubernetes集群中創建一個pod，驗證是否正常運行：
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc
- 瀏覽器訪問測試
