刪除命名空間失敗
今天在作測試的時候,清理集群。就把沒用的都清理掉包括命名空間。但是發現失敗了,一直卡在終止狀態。
導致刪除失敗的原因一般有兩種:
1、命名空間下還有資源在用,如果有刪除,命名空間自動消失(原因:API服務器強制要求,只有在名稱空間為空的情況下,並且只有在名稱空間的情況下才能從存儲中刪除名稱空間。)
2、就是名稱空間下沒有資源
開始分析原因
正常可以有三種方式
1、查看控制器日志 kube-controller-manager
2、查看api-resources會有報錯信息在下面
獲取所有注冊ns的資源,提取能delete的部分(kubectl api-resources --namespaced=true --verbs=delete)
接下來查看注冊服務kubectl get apiservice
3、失敗查看當前yaml文件,通過get獲取。其中有status字段中的message會有相關信息。
kubectl get ns t1 -o yaml
刪除了一些留了主要的部分
[root@ECS1 ~]# kubectl get ns t1 -o yaml apiVersion: v1 kind: Namespace metadata: creationTimestamp: "2021-06-12T08:43:19Z" deletionTimestamp: "2021-06-12T08:43:24Z" name: t1 resourceVersion: "546044" uid: 62f4616c-ea45-441f-8ccf-4079d35c2796 spec: finalizers: - kubernetes status: conditions: - lastTransitionTime: "2021-06-12T08:43:29Z" message: 'Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource'
reason: DiscoveryFailed status: "True" type: NamespaceDeletionDiscoveryFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All legacy kube types successfully parsed reason: ParsedGroupVersions status: "False" type: NamespaceDeletionGroupVersionParsingFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content successfully deleted, may be waiting on finalization reason: ContentDeleted status: "False" type: NamespaceDeletionContentFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content successfully removed reason: ContentRemoved status: "False" type: NamespaceContentRemaining - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content-preserving finalizers finished reason: ContentHasNoFinalizers status: "False" type: NamespaceFinalizersRemaining phase: Terminating
問題直接定位到,這個資源找不到。並且在查看資源與查看注冊資源時,輸出最下面會有error。不處理上文故障的處理辦法如下:
正文:
處理方法
[root@ECS1 ~]# kubectl get ns NAME STATUS AGE app-team1 Terminating 3d7h default Active 3d19h internal Active 2d23h kube-node-lease Active 3d19h kube-public Active 3d19h kube-system Active 3d19h [root@ECS1 ~]# kubectl delete ns/app-team1 namespace "app-team1" deleted ^C [root@ECS1 ~]#
沒辦法只能手動停止,不然卡到你天荒地老。
找到一個神奇的地方找到這么一段話
There's one situation that may require forcing finalization for a namespace. If you've deleted a namespace and you've cleaned out all of the objects under it, but the namespace still exists, deletion can be forced by updating the namespace subresource, finalize
. This informs the namespace controller that it needs to remove the finalizer from the namespace and perform any cleanup:
大體意思就是如果刪除了命名空間,在已經清除空間下所有對象后。空間還在,那么需要通過更新名稱空間子資源來強制刪除。這種方式通知名稱空間控制器,我要從命名空間中刪除終結器並且執行清理所有操作。
這東西用的是restful請求方式,但是我這不安全端口都封掉了,開個代理吧(可以選擇用證書)
[root@ECS1 ~]# kubectl proxy --port=8081 Starting to serve on 127.0.0.1:8081
cat <<EOF | curl -X PUT \ localhost:8081/api/v1/namespaces/app-team1/finalize \ -H "Content-Type: application/json" \ --data-binary @- { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "app-team1" }, "spec": { "finalizers": null } } EOF
查看結果
[root@ECS1 ~]# cat <<EOF | curl -X PUT \ > localhost:8081/api/v1/namespaces/app-team1/finalize \ > -H "Content-Type: application/json" \ > --data-binary @- > { > "kind": "Namespace", > "apiVersion": "v1", > "metadata": { > "name": "app-team1" > }, > "spec": { > "finalizers": null > } > } > EOF { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "app-team1", "uid": "108e6665-9b70-422c-8f94-783347101836", "resourceVersion": "533794", "creationTimestamp": "2021-06-08T23:46:24Z", "deletionTimestamp": "2021-06-12T06:27:33Z", "managedFields": [ { "manager": "curl", "operation": "Update", "apiVersion": "v1", "time": "2021-06-12T06:58:32Z", "fieldsType": "FieldsV1", "fieldsV1": {"f:status":{"f:phase":{}}} } ] }, "spec": { }, "status": { "phase": "Terminating", "conditions": [ { "type": "NamespaceDeletionDiscoveryFailure", "status": "True", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "DiscoveryFailed", "message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource" }, { "type": "NamespaceDeletionGroupVersionParsingFailure", "status": "False", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "ParsedGroupVersions", "message": "All legacy kube types successfully parsed" }, { "type": "NamespaceDeletionContentFailure", "status": "False", "