deepin 安裝 microk8s 1.17 踩坑記錄

microk8s 1.17

環境: Debian 系 Deepin 15.11 桌面系統, ubuntu 理論上可參照

安裝參考鏈接: http://www.imooc.com/article/291860

安裝

sudo apt update
sudo apt install snapd snap
export $PATH=PATH:/snap/bin >> ~/.zshrc && source ~/.zshrc
sudo snap install microk8s --classic
sudo microk8s.status --wait-ready

## status 輸出
microk8s is running
addons:
cilium: disabled
dashboard: enabled
dns: enabled
fluentd: disabled
gpu: disabled
helm: disabled
ingress: disabled
istio: disabled
jaeger: disabled
juju: disabled
knative: disabled
kubeflow: disabled
linkerd: disabled
metallb: disabled
metrics-server: disabled
prometheus: disabled
rbac: disabled
registry: disabled
storage: disabled

監控 pods 狀態

watch microk8s.kubectl get all --all-namespaces

這是問題解決后的狀態, STATUS 都是 Running

NAMESPACE     NAME                                                  READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-9b8997588-hlqxz                           1/1     Running   54         4h38m
kube-system   pod/dashboard-metrics-scraper-687667bb6c-7f79n        0/1     Pending   0          6m50s
kube-system   pod/dashboard-metrics-scraper-687667bb6c-r8tgq        0/1     Evicted   0          37m
kube-system   pod/heapster-v1.5.2-5c58f64f8b-lj2nf                  4/4     Running   0          37m
kube-system   pod/kubernetes-dashboard-5c848cc544-47fqk             0/1     Evicted   0          6m53s
kube-system   pod/kubernetes-dashboard-5c848cc544-4zdgs             0/1     Evicted   0          6m52s
kube-system   pod/kubernetes-dashboard-5c848cc544-7mhmj             0/1     Evicted   0          6m52s
kube-system   pod/kubernetes-dashboard-5c848cc544-7xwfw             0/1     Pending   0          6m50s
kube-system   pod/kubernetes-dashboard-5c848cc544-c7t4v             0/1     Evicted   0          6m51s
kube-system   pod/kubernetes-dashboard-5c848cc544-kfnds             0/1     Evicted   0          6m53s
kube-system   pod/kubernetes-dashboard-5c848cc544-l8r6s             0/1     Evicted   0          6m54s
kube-system   pod/kubernetes-dashboard-5c848cc544-ms8gg             0/1     Evicted   0          6m54s
kube-system   pod/kubernetes-dashboard-5c848cc544-ngvlc             0/1     Evicted   0          6m54s
kube-system   pod/kubernetes-dashboard-5c848cc544-p7xqc             0/1     Evicted   0          6m54s
kube-system   pod/kubernetes-dashboard-5c848cc544-wlw5m             0/1     Evicted   0          37m
kube-system   pod/monitoring-influxdb-grafana-v4-6d599df6bf-nvr62   2/2     Running   0          37m

NAMESPACE     NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes                  ClusterIP   10.152.183.1     <none>        443/TCP                  5h34m
kube-system   service/dashboard-metrics-scraper   ClusterIP   10.152.183.61    <none>        8000/TCP                 37m
kube-system   service/heapster                    ClusterIP   10.152.183.168   <none>        80/TCP                   37m
kube-system   service/kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   4h38m
kube-system   service/kubernetes-dashboard        ClusterIP   10.152.183.29    <none>        443/TCP                  37m
kube-system   service/monitoring-grafana          ClusterIP   10.152.183.195   <none>        80/TCP                   37m
kube-system   service/monitoring-influxdb         ClusterIP   10.152.183.212   <none>        8083/TCP,8086/TCP        37m

NAMESPACE     NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns                          1/1     1            1           4h38m
kube-system   deployment.apps/dashboard-metrics-scraper        0/1     1            0           37m
kube-system   deployment.apps/heapster-v1.5.2                  1/1     1            1           37m
kube-system   deployment.apps/kubernetes-dashboard             0/1     1            0           37m
kube-system   deployment.apps/monitoring-influxdb-grafana-v4   1/1     1            1           37m

NAMESPACE     NAME                                                        DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-9b8997588                           1         1         1       4h38m
kube-system   replicaset.apps/dashboard-metrics-scraper-687667bb6c        1         1         0       37m
kube-system   replicaset.apps/heapster-v1.5.2-5c58f64f8b                  1         1         1       37m
kube-system   replicaset.apps/kubernetes-dashboard-5c848cc544             1         1         0       37m
kube-system   replicaset.apps/monitoring-influxdb-grafana-v4-6d599df6bf   1         1         1       37m

修改 .zshrc

.bashrc 教程一大把

如果本地沒有安裝 kubectl 可以使用 alias; 否則請不要全部復制粘貼;

如果已經安裝了 kubectl，可以用下面的命令覆蓋配置文件：

microk8s.kubectl config view --raw > $HOME/.kube/config

-- 來自開篇的安裝教程 http://www.imooc.com/article/291860

以下僅限於 zsh 用戶; bash 用戶百度下即可

vim ~/.zshrc

export PATH=$PATH:/usr/local/go/bin:/snap/bin
alias kubectl='microk8s.kubectl'
# 命令補全
if [ $commands[microk8s.kubectl] ]; then 
	source <(microk8s.kubectl completion zsh | 
	sed "s/complete -o default -F __start_kubectl kubectl/complete -o default -F __start_kubectl microk8s.kubectl/g" | 
	sed "s/complete -o default -o nospace -F __start_kubectl kubectl/complete -o default -o nospace -F __start_kubectl microk8s.kubectl/g"); 
fi

添加ctr proxy

microk8s.docker 命令在 1.17 版本被移除; 由 containerd 代替;

之前版本的, 需要修改 dockerd-env 加代理

sudo vim /var/snap/microk8s/current/args/containerd-env

HTTPS_PROXY=http://127.0.0.1:1082

重啟 containerd 服務

sudo systemctl restart snap.microk8s.daemon-containerd.service

沒有proxy 的也可以參照開篇鏈接教程, 條條大路通羅馬, 不是非要proxy才能完成這個事情

修改內存/硬盤空間限制

sudo vim /var/snap/microk8s/current/args/kubelet

# 酌情復制
--eviction-hard="memory.available<1024Mi,nodefs.available<1Gi,imagefs.available<1Gi"
## 意思是: 當本node宿主機的 內存小於 1024Mi / 硬盤存儲 小於 1Gi 時, 會將 pod 強制驅逐

這里之前有一些理解上的錯誤, 原本以為是 允許多大 內存/硬盤 使用, 后來發現並不是這樣子的; 鑒於這篇文章現在有 7 個閱讀, 我對這 7 位讀者表示抱歉...
后面針對這樣不明確的地方一定查證后再上傳

防火牆 ufw

關於 CrashLoopBackOff 問題

sudo iptables -P FORWARD ACCEPT
# 1.17版本是 cni0; 之前版本是 cnr0, 參照官網 TroubleShooting
sudo ufw allow in on cni0 && sudo ufw allow out on cni0
sudo ufw default allow routed

重啟 microk8s

microk8s.stop && microk8s.start

開啟 add-on

sudo microk8s.enable dns dashboard
# 安裝輸出備忘部分
# enable dashbord后的輸出部分, RBAC 未開啟狀態下需要依賴 token開頭的兩個命令 獲取 token
If RBAC is not enabled access the dashboard using the default token retrieved with:

token=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d " " -f1);microk8s.kubectl -n kube-system describe secret $token

In an RBAC enabled setup (microk8s.enable RBAC) you need to create a user with restricted
permissions as shown in:
https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md

開啟dashboard

開啟proxy

kubectl proxy --address='0.0.0.0' --accept-hosts='^*$'

新開命令行, 使用 kubectl get service -n kube-system 查看dashboard的 ip

kubectl get service -n kube-system 
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
dashboard-metrics-scraper   ClusterIP   10.152.183.169   <none>        8000/TCP                 17h
heapster                    ClusterIP   10.152.183.76    <none>        80/TCP                   17h
kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   22h
kubernetes-dashboard        ClusterIP   10.152.183.237   <none>        443/TCP                  17h
monitoring-grafana          ClusterIP   10.152.183.197   <none>        80/TCP                   17h
monitoring-influxdb         ClusterIP   10.152.183.82    <none>        8083/TCP,8086/TCP        17h

瀏覽器訪問 https://10.152.183.237 可以到達 k8s-dashboard 界面 *注意 https 一定要加上, 不能去訪問 443 端口(10.152.183.237:443 是行不通的)

或者: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#/login

dashboard 用戶

csdn 大佬路子 https://blog.csdn.net/wucong60/article/details/81911859

#### 
### dashboard addon 啟動之后, microk8s 會自動啟動一個 token 認證服務 default-token-b96pr 在 default namespace 里面
### 可以通過命令:
### $ token=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d " " -f1);microk8s.kubectl -n kube-system describe secret $token 
### 直接獲取到 token
####

---

# 下面是手動創建secret
# 創建服務
kubectl create serviceaccount cluster-admin-dashboard-sa
# 啟動
kubectl create clusterrolebinding cluster-admin-dashboard-sa --clusterrole=cluster-admin --serviceaccount=default:cluster-admin-dashboard-sa

# 獲取 pod 名稱
kubectl get secret | grep cluster-admin-dashboard-sa

# 獲取 token
kubectl describe secrets/cluster-admin-dashboard-sa-token-82dwx

# 查看 token 服務
kubectl get serviceaccount

# 刪除手動創建的服務
kubectl delete serviceaccount cluster-admin-dashboard-sa

https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md

操作流程:

開啟 RBAC 認證后的 官方推薦 操作流程

Create Service Account

mkdir ~/microk8s && cd ~/microk8s

vim dashboard-adminuser.yaml
# 寫入文件
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
# EOF
# 應用更改
kubectl apply -f ./dashboard-adminuser.yaml
# 輸出 serviceaccount/admin-user created

Create ClusterRoleBinding

touch role-bind.yaml
echo 'apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard' > role-bind.yaml


kubectl apply -f ./role-bind.yaml          
# 輸出 clusterrolebinding.rbac.authorization.k8s.io/admin-user created

獲取token

kubectl -n kube-system describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

coredns 不啟動 - deepin

snap 的安裝 觸發了 apparmor 的啟動;

pod錯誤: CrashLoopBackOff

coredns日志:

kubectl logs -f coredns-xxxxxxx-xxxxx -n kube-system

:: socket permission denied; listen tcp port failed

暫行解決辦法:

option#1. 關閉 apparmor https://wiki.debian.org/AppArmor/HowToUse#Disable_AppArmor

$ sudo mkdir -p /etc/default/grub.d
$ echo 'GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT apparmor=0"' \
  | sudo tee /etc/default/grub.d/apparmor.cfg
$ sudo update-grub
$ sudo reboot

option#2. 參考鏈接 https://blog.csdn.net/u014062332/article/details/100099196

刪除 Evicted pod

microk8s 運行一天后, 發現有很多處於 Evicted(被k8s放棄) 的 pod, 原因不明,刪了去求,原因是 kubectl 配置文件有問題->k8s檢測到系統資源達到了閥值, 放棄了pod以釋放資源

kubectl get pods -n kube-system | grep Evicted | awk '{print $1}' | xargs microk8s.kubectl delete pod -n kube-system

參考鏈接 https://serverfault.com/questions/972120/microk8s-keeps-evicting-pods

參考鏈接 https://blog.csdn.net/zzq900503/article/details/83788152