Kubernetes namespace刪除失敗


刪除命名空間失敗

 

今天在作測試的時候,清理集群。就把沒用的都清理掉包括命名空間。但是發現失敗了,一直卡在終止狀態。

 

導致刪除失敗的原因一般有兩種:

 

1、命名空間下還有資源在用,如果有刪除,命名空間自動消失(原因:API服務器強制要求,只有在名稱空間為空的情況下,並且只有在名稱空間的情況下才能從存儲中刪除名稱空間。)

 

2、就是名稱空間下沒有資源

 

 

開始分析原因

 

正常可以有三種方式

 

1、查看控制器日志  kube-controller-manager

2、查看api-resources會有報錯信息在下面

  • 獲取所有注冊ns的資源,提取能delete的部分(kubectl api-resources --namespaced=true --verbs=delete)
  • 接下來查看注冊服務kubectl get apiservice

3、失敗查看當前yaml文件,通過get獲取。其中有status字段中的message會有相關信息。

 

 

kubectl get ns t1 -o yaml

 

 

刪除了一些留了主要的部分

 

 

[root@ECS1 ~]# kubectl get ns t1 -o yaml apiVersion: v1 kind: Namespace metadata: creationTimestamp: "2021-06-12T08:43:19Z" deletionTimestamp: "2021-06-12T08:43:24Z" name: t1 resourceVersion: "546044" uid: 62f4616c-ea45-441f-8ccf-4079d35c2796 spec: finalizers: - kubernetes status: conditions: - lastTransitionTime: "2021-06-12T08:43:29Z" message: 'Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource'
 reason: DiscoveryFailed status: "True" type: NamespaceDeletionDiscoveryFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All legacy kube types successfully parsed reason: ParsedGroupVersions status: "False" type: NamespaceDeletionGroupVersionParsingFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content successfully deleted, may be waiting on finalization reason: ContentDeleted status: "False" type: NamespaceDeletionContentFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content successfully removed reason: ContentRemoved status: "False" type: NamespaceContentRemaining - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content-preserving finalizers finished reason: ContentHasNoFinalizers status: "False" type: NamespaceFinalizersRemaining phase: Terminating

 

問題直接定位到,這個資源找不到。並且在查看資源與查看注冊資源時,輸出最下面會有error。不處理上文故障的處理辦法如下:

 

正文:

 

處理方法

 

[root@ECS1 ~]# kubectl get ns NAME STATUS AGE app-team1  Terminating 3d7h default Active 3d19h internal Active 2d23h kube-node-lease Active 3d19h kube-public Active 3d19h kube-system Active 3d19h [root@ECS1 ~]# kubectl delete ns/app-team1 namespace "app-team1" deleted ^C [root@ECS1 ~]# 

 

沒辦法只能手動停止,不然卡到你天荒地老。

 

找到一個神奇的地方找到這么一段話

 

There's one situation that may require forcing finalization for a namespace. If you've deleted a namespace and you've cleaned out all of the objects under it, but the namespace still exists, deletion can be forced by updating the namespace subresource, finalize. This informs the namespace controller that it needs to remove the finalizer from the namespace and perform any cleanup:

 

大體意思就是如果刪除了命名空間,在已經清除空間下所有對象后。空間還在,那么需要通過更新名稱空間子資源來強制刪除。這種方式通知名稱空間控制器,我要從命名空間中刪除終結器並且執行清理所有操作。

 

這東西用的是restful請求方式,但是我這不安全端口都封掉了,開個代理吧(可以選擇用證書)

 

[root@ECS1 ~]# kubectl proxy --port=8081 Starting to serve on 127.0.0.1:8081

 

開始刪除

 

cat <<EOF | curl -X PUT \ localhost:8081/api/v1/namespaces/app-team1/finalize \ -H "Content-Type: application/json" \ --data-binary @- { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "app-team1" }, "spec": { "finalizers": null } } EOF

 

查看結果

 

[root@ECS1 ~]# cat <<EOF | curl -X PUT \ > localhost:8081/api/v1/namespaces/app-team1/finalize \ > -H "Content-Type: application/json" \ > --data-binary @- > { > "kind": "Namespace", > "apiVersion": "v1", > "metadata": { > "name": "app-team1" > }, > "spec": { > "finalizers": null > } > } > EOF { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "app-team1", "uid": "108e6665-9b70-422c-8f94-783347101836", "resourceVersion": "533794", "creationTimestamp": "2021-06-08T23:46:24Z", "deletionTimestamp": "2021-06-12T06:27:33Z", "managedFields": [ { "manager": "curl", "operation": "Update", "apiVersion": "v1", "time": "2021-06-12T06:58:32Z", "fieldsType": "FieldsV1", "fieldsV1": {"f:status":{"f:phase":{}}} } ] }, "spec": { }, "status": { "phase": "Terminating", "conditions": [ { "type": "NamespaceDeletionDiscoveryFailure", "status": "True", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "DiscoveryFailed", "message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource" }, { "type": "NamespaceDeletionGroupVersionParsingFailure", "status": "False", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "ParsedGroupVersions", "message": "All legacy kube types successfully parsed" }, { "type": "NamespaceDeletionContentFailure", "status": "False", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "ContentDeleted", "message": "All content successfully deleted, may be waiting on finalization" }, { "type": "NamespaceContentRemaining", "status": "False", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "ContentRemoved", "message": "All content successfully removed" }, { "type": "NamespaceFinalizersRemaining", "status": "False", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "ContentHasNoFinalizers", "message": "All content-preserving finalizers finished" } ] } } [root@ECS1 ~]# kubectl get ns NAME STATUS AGE default Active 3d19h internal Active 2d23h kube-node-lease Active 3d19h kube-public Active 3d19h kube-system Active 3d19h [root@ECS1 ~]# 

 

刪除成功

 

 

 

理解不了上面開始刪除那部分的代碼可以參考下面這部分。

 

kubectl get namespace t1 -o json > ns.json

 

開始刪除(先作錯誤示范)

 

 

[root@ECS1 ~]# curl -X PUT localhost:8081/api/v1/namespaces/t1/finalize -H "Content-Type: application/json" --data-binary @ns.json { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "t1", "uid": "62f4616c-ea45-441f-8ccf-4079d35c2796", "resourceVersion": "546044", "creationTimestamp": "2021-06-12T08:43:19Z", "deletionTimestamp": "2021-06-12T08:43:24Z", "managedFields": [ { "manager": "kubectl-create", "operation": "Update", "apiVersion": "v1", "time": "2021-06-12T08:43:19Z", "fieldsType": "FieldsV1", "fieldsV1": {"f:status":{"f:phase":{}}} }, { "manager": "kube-controller-manager", "operation": "Update", "apiVersion": "v1", "time": "2021-06-12T08:43:29Z", "fieldsType": "FieldsV1", "fieldsV1": {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"NamespaceContentRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} } ] }, "spec": { "finalizers": [ "kubernetes" ] }, "status": { "phase": "Terminating", "conditions": [ { "type": "NamespaceDeletionDiscoveryFailure", "status": "True", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "DiscoveryFailed", "message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource" }, { "type": "NamespaceDeletionGroupVersionParsingFailure", "status": "False", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "ParsedGroupVersions", "message": "All legacy kube types successfully parsed" }, { "type": "NamespaceDeletionContentFailure", "status": "False", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "ContentDeleted", "message": "All content successfully deleted, may be waiting on finalization" }, { "type": "NamespaceContentRemaining", "status": "False", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "ContentRemoved", "message": "All content successfully removed" }, { "type": "NamespaceFinalizersRemaining", "status": "False", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "ContentHasNoFinalizers", "message": "All content-preserving finalizers finished" } ] } }

 

失敗了

 

[root@ECS1 ~]# kubectl get ns NAME STATUS AGE default Active 3d22h kube-node-lease Active 3d22h kube-public Active 3d22h kube-system Active 3d22h t1 Terminating 59m

 

 

編輯ns.json文件刪除spec部分保存文件,重新執行命令即可。

 

展示一下執行結果

 

 

[root@ECS1 ~]# curl -X PUT localhost:8081/api/v1/namespaces/t1/finalize -H "Content-Type: application/json" --data-binary @ns.json { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "t1", "uid": "62f4616c-ea45-441f-8ccf-4079d35c2796", "resourceVersion": "546044", "creationTimestamp": "2021-06-12T08:43:19Z", "deletionTimestamp": "2021-06-12T08:43:24Z", "managedFields": [ { "manager": "kubectl-create", "operation": "Update", "apiVersion": "v1", "time": "2021-06-12T08:43:19Z", "fieldsType": "FieldsV1", "fieldsV1": {"f:status":{"f:phase":{}}} }, { "manager": "kube-controller-manager", "operation": "Update", "apiVersion": "v1", "time": "2021-06-12T08:43:29Z", "fieldsType": "FieldsV1", "fieldsV1": {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"NamespaceContentRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} } ] }, "spec": { }, "status": { "phase": "Terminating", "conditions": [ { "type": "NamespaceDeletionDiscoveryFailure", "status": "True", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "DiscoveryFailed", "message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource" }, { "type": "NamespaceDeletionGroupVersionParsingFailure", "status": "False", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "ParsedGroupVersions", "message": "All legacy kube types successfully parsed" }, { "type": "NamespaceDeletionContentFailure", "status": "False", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "ContentDeleted", "message": "All content successfully deleted, may be waiting on finalization" }, { "type": "NamespaceContentRemaining", "status": "False", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "ContentRemoved", "message": "All content successfully removed" }, { "type": "NamespaceFinalizersRemaining", "status": "False", "lastTransitionTime": "2021-06-12T08:43:29Z", "reason": "ContentHasNoFinalizers", "message": "All content-preserving finalizers finished" } ] } } [root@ECS1 ~]# kubectl get ns NAME STATUS AGE default Active 3d22h kube-node-lease Active 3d22h kube-public Active 3d22h kube-system Active 3d22h test Terminating 61m

 

 

 

這么作雖然能刪除,但是有風險(盡量保證下面什么資源都沒有后在刪除空間)

 

This should be done with caution as it may delete the namespace only and leave orphan objects within the, now non-exiting, namespace - a confusing state for Kubernetes. If this happens, the namespace can be re-created manually and sometimes the orphaned objects will re-appear under the just-created namespace which will allow manual cleanup and recovery

 

譯文:這樣做時應該謹慎,因為它可能只刪除名稱空間,而將孤立對象留在現在不存在的名稱空間中——這對Kubernetes來說是一種令人困惑的狀態。 如果發生這種情況,可以手動重新創建名稱空間,有時孤立的對象將重新出現在剛剛創建的名稱空間下,這將需要手動清理和恢復

 

原因:

官方的一段話:

當執行刪除后 Kubernetes 報告該對象已被刪除,但是,它尚未在傳統意義上被刪除。相反,它處於刪除過程中。當我們再次嘗試該對象時,我們發現該對象已修改,以包括刪除時間戳。

 

 

[root@ECS1 ~]# curl -X PUT localhost:8081/api/v1/namespaces/internal/finalize -H "Content-Type: application/json" --data-binary @ns.json { "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "Operation cannot be fulfilled on namespaces \"internal\": the object has been modified; please apply your changes to the latest version and try again", "reason": "Conflict", "details": { "name": "internal", "kind": "namespaces" }, "code": 409 }

 

 

 

所發生的是對象已更新,而不是刪除。這是因為 Kubernetes 看到對象包含終結器,並將其置於僅讀取狀態。刪除時間戳表示對象只能讀取,但刪除終結者密鑰更新除外。換句話說,刪除將不完整,直到我們編輯對象並刪除終結者。

 

 

 

 

 

作者K&


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM