kubernetes 無法刪除 pod 問題的解決

[摘要] kubernetes 可能會產生垃圾或者僵屍pod，在刪除rc的時候，相應的pod沒有被刪除，手動刪除pod后會自動重新創建，這時一般需要先刪除掉相關聯的resources，實際中還要具體情況具體分析。

在使用Helm部署gitlab的時候發現有垃圾的pod存在，一直刪除不掉

 

# kubectl get pods -n kube-system
NAME                            READY    STATUS    RESTARTS   AGE
gitlab-migrations.1-2km-4rwnm      1/1   Running       158    21h
gitlab-migrations.1-30d-gx9nz      1/1   Running       164    21h
gitlab-migrations.1-dsw-m5qbc      1/1   Running       167    21h

　　

一般來說是要刪除pod關聯的rc，rs以及deployment等等，可以執行如下命令查詢

 

kubectl get ClusterRole -n kube-system
kubectl get ClusterRoleBinding -n kube-system
kubectl get Deployments -n kube-system
kubectl get Pods -n kube-system
kubectl get ServiceAccounts -n kube-system
kubectl get CustomResourceDefinition -n kube-system
kubectl get replicasets -n kube-system
kubectl get replicationcontrollers -n kube-system
kubectl get services -n kube-system

　　

找到對應的標簽進行刪除，對應的pod一般就會被刪除掉，如果pod還在的話再手動刪除pod。
實際中我這三個pod還是無法刪除掉，刪除之后仍然會重新啟動，所以仔細看了下pod的具體描述內容。

 

# kubectl describe pod -n kube-system gitlab-migrations.1-30d-gx9nz
Name:           gitlab-migrations.1-30d-gx9nz
Namespace:      kube-system
Node:           172.30.0.169/172.30.0.169
Start Time:     Mon, 26 Mar 2018 11:53:45 +0800
Labels:         app=migrations
                controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209
                job-name=gitlab-migrations.1-30d
                release=gitlab
Annotations:    <none>
Status:         Running
IP:             172.100.14.7
Controlled By:  Job/gitlab-migrations.1-30d
Init Containers:
  configure:
    Container ID:  docker://a3051d24e65267489aa97ae368bcbe5819e4ae4f3b5ddc4d277b0b25ee585716
    Image:         busybox
    Image ID:      docker-pullable://docker.io/busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21
    Port:          <none>
    Command:
      sh
      /config/configure
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 26 Mar 2018 11:54:24 +0800
      Finished:     Mon, 26 Mar 2018 11:54:24 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /config from migrations-config (ro)
      /init-config from init-migrations-secrets (ro)
      /init-secrets from migrations-secrets (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro)
Containers:
  migrations:
    Container ID:  docker://c33e809de76ba44da330063d9e3c6f58631b30e3c805ea561dc68af38ffe3926
    Image:         registry.gitlab.com/gitlab-org/build/cng/gitlab-rails:latest
    Image ID:      docker-pullable://registry.gitlab.com/gitlab-org/build/cng/gitlab-rails@sha256:7dca5eb3b686342a9cff798f8349ad07459a1ef1e9d47a23b787b284d874cc17
    Port:          <none>
    Args:
      /scripts/wait-for-deps
      /scripts/db-migrate
    State:          Running
      Started:      Tue, 27 Mar 2018 09:50:09 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 27 Mar 2018 09:41:52 +0800
      Finished:     Tue, 27 Mar 2018 09:45:02 +0800
    Ready:          True
    Restart Count:  164
    Environment:
      CONFIG_TEMPLATE_DIRECTORY:  /var/opt/gitlab/templates
      CONFIG_DIRECTORY:           /var/opt/gitlab/config/gitlab/
      GITLAB_ROOT_PASSWORD:       Vi4ree2R
    Mounts:
      /etc/gitlab from migrations-secrets (ro)
      /home/git/gitlab/config/secrets.yml from migrations-secrets (rw)
      /var/opt/gitlab/templates from migrations-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  migrations-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gitlab-migrations
    Optional:  false
  init-migrations-secrets:
  <unknown>
  migrations-secrets:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  Memory
  default-token-6rdp2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6rdp2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason       Age                  From                   Message
  ----     ------       ----                 ----                   -------
  Warning  FailedMount  6m (x655 over 21h)   kubelet, 172.30.0.169  MountVolume.SetUp failed for volume "migrations-config" : configmaps "gitlab-migrations" not found
  Warning  BackOff      3m (x3516 over 21h)  kubelet, 172.30.0.169  Back-off restarting failed container

　　

需要關注的地方是Labels標簽

 

Labels:         app=migrations
                controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209
                job-name=gitlab-migrations.1-30d
                release=gitlab

　　

發現有個job-name=gitlab-migrations.xxx
於是繼續查看kubernetes是否有job存在

 

# kubectl get jobs -n kube-system
NAME                      DESIRED   SUCCESSFUL   AGE
gitlab-migrations.1-2km   1         0            23h
gitlab-migrations.1-30d   1         0            23h
gitlab-migrations.1-dsw   1         0            23h

　　

先執行刪除job看下：

 

# kubectl delete job gitlab-migrations.1-2km -n kube-system
job "gitlab-migrations.1-2km" deleted
# kubectl delete job gitlab-migrations.1-30d -n kube-system
job "gitlab-migrations.1-30d" deleted
# kubectl delete job gitlab-migrations.1-dsw -n kube-system
job "gitlab-migrations.1-dsw" deleted

　　

再看pod情況：

 

# kubectl get pods -n kube-system
NAME                               READY STATUS     RESTARTS  AGE
gitlab-migrations.1-30d-gx9nz      1/1   Terminating   173    23h
gitlab-migrations.1-dsw-m5qbc      0/1   Terminating   176    23h

　　

附圖為kubernets 的一些對象資源：

  

再看pod已經被刪除了，還有處於Terminating狀態的，一會就會消失掉。至此pod刪除成功，所以刪除與pods相關聯的依賴資源是解決問題的最終方法！