部署一個簡單的 demo 發現一個很奇妙的錯誤,yaml 文件內容在下方,鏡像倉庫是內網自建的 harbor-1.9.1 單節點倉庫,測試就沒有整高可用了。
用戶名:admin
密碼:admin123
[root@k8s-m1 ~]# cat /opt/work/jenkins/jenkins_home/workspace/k8s-web-demo/web.yaml
#deploy apiVersion: apps/v1 kind: Deployment metadata: name: k8s-web-demo spec: selector: matchLabels: app: k8s-web-demo replicas: 1 template: metadata: labels: app: k8s-web-demo spec: containers: - name: k8s-web-demo image: harbor.unchch.com/kubernetes/k8s-web-demo:20191031_105142_3551 ports: - containerPort: 8080 --- #service apiVersion: v1 kind: Service metadata: name: k8s-web-demo spec: ports: - port: 80 protocol: TCP targetPort: 8080 selector: app: k8s-web-demo type: ClusterIP --- #ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: k8s-web-demo spec: rules: - host: k8s-web.unchch.com http: paths: - path: / backend: serviceName: k8s-web-demo servicePort: 80
檢查 pods,發現報錯:
繼續查看錯誤信息
# kubectl describe po k8s-web-demo-675df649c7-s2scg
報錯內容是:
pulling image "harbor.unchch.com/kubernetes/k8s-web-demo:20191031_105142_3551" Warning Failed 36s (x3 over 71s) kubelet, k8s-n1 Failed to pull image "harbor.unchch.com/kubernetes/k8s-web-demo:20191031_105142_3551": rpc error: code = Unknown desc = Error response from daemon: pull access denied for harbor.unchch.com/kubernetes/k8s-web-demo, repository does not exist or may require 'docker login': denied: requested access to the resource is denied Warning Failed 36s (x3 over 71s) kubelet, k8s-n1 Error: ErrImagePull Normal BackOff 10s (x4 over 70s) kubelet, k8s-n1 Back-off pulling image "harbor.unchch.com/kubernetes/k8s-web-demo:20191031_105142_3551" Warning Failed 10s (x4 over 70s) kubelet, k8s-n1 Error: ImagePullBackOff
錯誤提示 “repository does not exist or may require 'docker login': denied: requested access to the resource is denied”,理解為“鏡像不存在或者需要 docker login 拿到 pull 權限”。
但是在對應的 node 上執行 docker pull harbor.unchch.com/kubernetes/k8s-web-demo:20191031_105142_3551 是可以獲取到鏡像的,這個就尷尬了
排查過程:
1、去對應的 worker 節點手動 pull 該鏡像,是 ok 的,說明之前的 docker login 的用戶名和密碼是正確的,排除鏡像不存在和用戶名密碼輸錯的可能;
2、檢查 /root/.docker/config 文件中的用戶名密碼是不是有問題,解密之后也排除了:
3、測試了 docker push 等操作,排除了 harbor 倉庫的功能出錯等問題;
4、最后想到是不是 kubectl apply -f web.yaml 時,不能正確的獲取 docker login 的認證信息,基本就這一種解釋了,於是,手工創建了一個secret,使用 secret 來指定 pull 權限:
(1)創建 secret,此處在 harbor 上新建了一個用戶 manager:Harbor123
kubectl create secret docker-registry registry-harbor --namespace=default \ --docker-server=harbor.unchch.com --docker-username=manager \ --docker-password=Harbor123 --docker-email=unchch.xt@gmail.com
(2)查看創建的 secret
[root@k8s-m1 script]# kubectl get secrets NAME TYPE DATA AGE default-token-rqf8z kubernetes.io/service-account-token 3 9d registry-harbor kubernetes.io/dockerconfigjson 1 3h24m [root@k8s-m1 script]# kubectl describe secrets registry-harbor Name: registry-harbor Namespace: default Labels: <none> Annotations: <none> Type: kubernetes.io/dockerconfigjson Data ==== .dockerconfigjson: 141 bytes
(3)在 yaml 文件對應的部分指定secret:
再次 部署,一切正常。