Ubuntu 18 Kubernetes集群的安裝和部署以及Helm的安裝

本文轉載自查看原文 2019-08-08 13:02 2997 docker

首先說一下我的環境，我是在windows 10 上面建了一個ubuntu18的虛擬機，同時由於某些原因不受網絡限制，所以安裝比較順利。

Install

1.安裝並啟用 Docker

sudo apt install docker.io
sudo systemctl enable docker
docker --version

2.添加 Kubernetes signing key 和Repository

sudo apt install curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"

3.安裝Kubeadm

sudo apt install kubeadm
kubeadm version

#常用命令
重啟kubelet服務：
systemctl daemon-reload
systemctl restart kubelet
sudo systemctl restart kubelet.service
 
sudo systemctl daemon-reload
sudo systemctl stop kubelet
sudo systemctl enable kubelet
sudo systemctl start kubelet

4.禁用 swapoff

sudo swapoff -a

sudo sed -i '/ swap / s/^/#/' /etc/fstab

以上的指令我只在一台Ubuntu上執行的（如果你有多台計算機，需要在所有的計算機上執行以上指令，我這里是通過拷貝虛擬機來實現的）

5.准備2台虛擬機k8s-master和k8s-node(我這里把上面的計算機命名為 k8s_master ，copy它並命名為k8s_node)

sudo hostnamectl set-hostname k8s-master #在k8s-master 上執行 IP：192.168.255.229
sudo hostnamectl set-hostname k8s-node #k8s-node 上執行 IP：192.168.255.230

Deploy

1.在master上初始化 Kubernetes

使用kubeadm config print init-defaults可以打印集群初始化默認的使用的配置,使用kubeadm默認配置初始化的集群，會在master節點打上node-role.kubernetes.io/master:NoSchedule的污點，阻止master節點接受調度運行工作負載。這里測試環境只有兩個節點，所以將這個taint的effect從NoSchedule改為PreferNoSchedule 還有就是修訂kubernet版本1.15.2

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.100.7
  bindPort: 6443
nodeRegistration:
  taints:
  - effect: PreferNoSchedule
    key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.15.2
networking:
  podSubnet: 10.244.0.0/16

#sudo kubeadm init --pod-network-cidr=192.168.255.229/2
kubeadm init --config kubeadm.yaml #--ignore-preflight-errors=Swap

#sudo kubeadm init --pod-network-cidr=192.168.100.0/2 這里不應該用現有計算機的ip 這里有解決方案 Kubernetes-dashboard pod is crashing again and again

如果執行中遇到detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd"

請參考Container runtimes執行

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "insecure-registries":["192.168.100.20:80"] 
}
EOF
 
mkdir -p /etc/systemd/system/docker.service.d
#我順便吧docker的私有倉庫也加在里面
# Restart docker.
systemctl daemon-reload
systemctl restart docker

如果遇到port 10251 and 10252 are in use 錯誤請執行 netstat -lnp | grep 1025 然后kill 進程ID

2在master節點上執行如下：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
#檢查 master 
kubectl get nodes

3.Deploy a Pod Network & view the status

sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl get pods --all-namespaces
sudo kubectl get nodes

4.add slave node

在k8s-node上執行

kubeadm join 192.168.254.229:6443 --token ewlb93.v0ohocpvncaxgl16 --discovery-token-ca-cert-hash sha256:2522834081168fbe4b5b05854b964e76f1ea8bac6f8fc5e2be21c93c6a27c427

在k8s-master上檢查節點信息：

5.安裝 Dashboard 插件 &檢查狀態

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml 
kubectl get deployment kubernetes-dashboard -n kube-system
kubectl get svc kubernetes-dashboard -n kube-system
#kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

為了簡單，我這里直接為 Dashboard 賦予 Admin 的權限(否者會禁止訪問)，參考https://github.com/kubernetes/dashboard/wiki/Access-control#admin-privileges

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

啟動代理

kubectl proxy 
#或者 如果需要暴露給其他機器，需要指定address:
kubectl proxy --address='0.0.0.0' --port=8001 --accept-hosts='^*$'

訪問http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

不過我在訪問這個地址遇到錯誤如下：

root@k8s-master:~# kubectl get pods --all-namespaces 
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5c98db65d4-97xrf 1/1 Running 0 83m
kube-system coredns-5c98db65d4-vvtfc 1/1 Running 0 83m
kube-system etcd-k8s-master 1/1 Running 0 83m
kube-system kube-apiserver-k8s-master 1/1 Running 0 83m
kube-system kube-controller-manager-k8s-master 1/1 Running 0 83m
kube-system kube-flannel-ds-amd64-gbg49 1/1 Running 1 80m
kube-system kube-flannel-ds-amd64-hmrcp 1/1 Running 0 82m
kube-system kube-proxy-lbp5k 1/1 Running 0 80m
kube-system kube-proxy-szkb8 1/1 Running 0 83m
kube-system kube-scheduler-k8s-master 1/1 Running 0 83m
kube-system kubernetes-dashboard-7d75c474bb-p5nz5 0/1 CrashLoopBackOff 6 10m
root@k8s-master:~# kubectl logs kubernetes-dashboard-7d75c474bb-p5nz5 -n kube-system
2019/08/07 11:34:33 Using in-cluster config to connect to apiserver
2019/08/07 11:34:33 Using service account token for csrf signing
2019/08/07 11:34:33 Starting overwatch
2019/08/07 11:35:03 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ
root@k8s-master:~#

后來在https://stackoverflow.com/questions/52273029/kubernetes-dashboard-pod-is-crashing-again-and-again 找到解決方案，

Make sure you understand the difference between Host network, Pod network and Service network. These 3 networks can not overlap. For example --pod-network-cidr=192.168.0.0/16 must notinclude the IP address of your host, change it to 10.0.0.0/16 or something smaller if necessary.

所以初始化該為 sudo kubeadm init --pod-network-cidr=192.168.100.0/2 就okay 了

創建賬號

kubectl create serviceaccount dashboard -n default
kubectl create clusterrolebinding dashboard-admin -n default --clusterrole=cluster-admin --serviceaccount=default:dashboard
kubectl get secret $(kubectl get serviceaccount dashboard -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode

Copy the decoded password and login to dashboard.

在Kubernetes的安裝過程中會默認安裝很多Service賬戶，分別有不同的訪問權限，要找到對應的token，你可以使用下面的方式：

kubectl -n kube-system get secret
kubectl -n kube-system describe secret certificate-controller-token-4xr9x

上面我們使用token來登錄，這里我們還是用上面的dashboard賬號來生成kubeconfig文件，以后用生成kubeconfig文件來登錄

#配置集群信息
kubectl config set-cluster kubernetes --server=192.168.100.11:6443 --kubeconfig=/root/dashbord.conf

#使用token寫入集群驗證
#kubectl get secret $(kubectl get serviceaccount dashboard -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode #查看token
kubectl config set-credentials dashboard --token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRhc2hib2FyZC10b2tlbi12dDl4OSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkYXNoYm9hcmQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxZmM1ZDc1ZS0xMjE2LTQwMDgtYThhOS03ZjEwZGQ1NWJjNWEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkYXNoYm9hcmQifQ.Neb5_blRSig6IU5oPtRIQlQhcELaeI8uu7jeiVEdiR3CLiCZYyiI7X6uNsrpGKAkR-OkGM1gOp09-pmxFI6m4lKHYu9S7R1MNigmQrxfZB4RJ-iYZCNp3Rra7mFrluwY_yMbzuZ__XeYShSOiO1VAS2ezWFGk9adgtbiWZkef_NxmYdwEmTAGkmazhatK9SGDWBea-1seoJx-SGFyA9j0gNcWqNrX93ozFmuNtYrPZSwhYkul-q4NHOz4Dp4Ux1C7gZzTIgBySaYZd5tiJIAmZ-6CV-ukmPtFn7tVlNaDkK4K5N6jzyDttlvHZJtWqBR7iWTyamAKAbycm_BmaQR4Q --kubeconfig=/root/dashboard.conf 
配置上下文和當前上下文
kubectl config set-context dashboard@kubernetes --cluster=kubernetes --user=dashboard --kubeconfig=/root/dashboard.conf 
kubectl config use-context dashboard@kubernetes --kubeconfig=/root/dashboard.conf
kubectl config view --kubeconfig=/root/dashboard.conf

將/root/dashboard.conf 文件發送到宿主機，瀏覽器訪問時選擇Kubeconfig認證，載入該配置文件，點擊登陸，即可實現訪問，如圖：

卸載集群

想要撤銷kubeadm做的事，首先要排除節點，並確保在關閉節點之前要清空節點。

在主節點上運行：

kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>
然后在需要移除的節點上，重置kubeadm的安裝狀態：

kubeadm reset

重啟master節點如果遇到如下錯誤

The connection to the server 10.2.0.165:6443 was refused - did you specify the right host or port? 解決方案The connection to the server <host>:6443 was refused - did you specify the right host or port?

sudo -i
swapoff -a
exit
strace -eopenat kubectl version

#如果以上方法沒有搞定可以執行以下命令
systemctl status docker #查看docker狀態
systemctl status kubelet #查看kubelet的狀態
netstat -pnlt | grep 6443 #查看6443的端口應該啟動起來了
journalctl -xeu kubelet #查看kubelet的日志解決問題

systemctl restart kubelet #最后重啟

-------------------------2019-08-20--------------------------

Kubernetes常用組件部署

Helm的安裝

Helm由客戶端命helm令行工具和服務端tiller組成，Helm的安裝十分簡單。下載helm命令行工具到master節點node1的/usr/local/bin下，這里下載的2.14.1版本：

curl -O https://get.helm.sh/helm-v2.14.1-linux-amd64.tar.gz
tar -zxvf helm-v2.14.1-linux-amd64.tar.gz
cd linux-amd64/ cp helm /usr/local/bin/

為了安裝服務端tiller，還需要在這台機器上配置好kubectl工具和kubeconfig文件，確保kubectl工具可以在這台機器上訪問apiserver且正常使用。這里的node1節點已經配置好了kubectl。

因為Kubernetes APIServer開啟了RBAC訪問控制，所以需要創建tiller使用的service account: tiller並分配合適的角色給它。詳細內容可以查看helm文檔中的Role-based Access Control。這里簡單起見直接分配cluster-admin這個集群內置的ClusterRole給它。創建helm-rbac.yaml文件

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system

接下來使用helm部署tiller，tiller默認被部署在k8s集群中的kube-system這個namespace下

kubectl create -f helm-rbac.yaml
helm init --service-account tiller --skip-refresh kubectl get pod -n kube-system -l app=helm helm version

最后在k8s-master上修改helm chart倉庫的地址為azure提供的鏡像地址：

helm repo add stable http://mirror.azure.cn/kubernetes/charts

使用Helm部署Nginx Ingress

為了便於將集群中的服務暴露到集群外部，需要使用Ingress。接下來使用Helm將Nginx Ingress部署到Kubernetes上。 Nginx Ingress Controller被部署在Kubernetes的邊緣節點上，關於Kubernetes邊緣節點的高可用相關的內容可以查看之前整理的Bare metal環境下Kubernetes Ingress邊緣節點的高可用,Ingress Controller使用hostNetwork。

我們將node1(192.168.100.7)做為邊緣節點，打上Label：

root@k8s-master:~/linux-amd64# kubectl label node k8s-master node-role.kubernetes.io/edge= 
node/k8s-master labeled root@k8s-master:~/linux-amd64# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready edge,master 34m v1.15.2 k8s-node Ready <none> 28m v1.15.2 root@k8s-master:~/linux-amd64#

stable/nginx-ingress chart的值文件ingress-nginx.yaml如下：

controller:
  replicaCount: 1 hostNetwork: true nodeSelector: node-role.kubernetes.io/edge: '' affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-ingress - key: component operator: In values: - controller topologyKey: kubernetes.io/hostname tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule defaultBackend: nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule

nginx ingress controller的副本數replicaCount為1，將被調度到k8s-master這個邊緣節點上。這里並沒有指定nginx ingress controller service的externalIPs，而是通過hostNetwork: true設置nginx ingress controller使用宿主機網絡。

helm repo update
helm install stable/nginx-ingress -n nginx-ingress --namespace ingress-nginx  -f ingress-nginx.yaml helm repo add stable http://mirror.azure.cn/kubernetes/charts

如果訪問http://192.168.100.7返回default backend，則部署完成。

kubectl get pod -n ingress-nginx -o wide
kubectl -n ingress-nginx exec nginx-ingress-controller-xxxxx -- cat /etc/nginx/nginx.conf

使用Helm部署dashboard

kubernetes-dashboard.yaml

image:
  repository: k8s.gcr.io/kubernetes-dashboard-amd64 tag: v1.10.1 ingress: enabled: true hosts: - k8s.frognew.com annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" tls: - secretName: frognew-com-tls-secret hosts: - k8s.frognew.com nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule rbac: clusterAdminRole: true

安裝：

helm install stable/kubernetes-dashboard -n kubernetes-dashboard --namespace kube-system -f kubernetes-dashboard.yaml
kubectl -n kube-system get secret | grep kubernetes-dashboard-token
kubectl describe -n kube-system secret/kubernetes-dashboard-token-xxx
helm search kubernetes-dashboard
kubectl get pods -n kube-system -o wide