刪除命名空間失敗
今天在作測試的時候,清理集群。就把沒用的都清理掉包括命名空間。但是發現失敗了,一直卡在終止狀態。
導致刪除失敗的原因一般有兩種:
1、命名空間下還有資源在用,如果有刪除,命名空間自動消失(原因:API服務器強制要求,只有在名稱空間為空的情況下,並且只有在名稱空間的情況下才能從存儲中刪除名稱空間。)
2、就是名稱空間下沒有資源
開始分析原因
正常可以有三種方式
1、查看控制器日志 kube-controller-manager
2、查看api-resources會有報錯信息在下面
獲取所有注冊ns的資源,提取能delete的部分(kubectl api-resources --namespaced=true --verbs=delete)接下來查看注冊服務kubectl get apiservice
3、失敗查看當前yaml文件,通過get獲取。其中有status字段中的message會有相關信息。
kubectl get ns t1 -o yaml
刪除了一些留了主要的部分
[root@ECS1 ~]# kubectl get ns t1 -o yaml apiVersion: v1 kind: Namespace metadata: creationTimestamp: "2021-06-12T08:43:19Z" deletionTimestamp: "2021-06-12T08:43:24Z" name: t1 resourceVersion: "546044" uid: 62f4616c-ea45-441f-8ccf-4079d35c2796 spec: finalizers: - kubernetes status: conditions: - lastTransitionTime: "2021-06-12T08:43:29Z" message: 'Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource'
reason: DiscoveryFailed status: "True" type: NamespaceDeletionDiscoveryFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All legacy kube types successfully parsed reason: ParsedGroupVersions status: "False" type: NamespaceDeletionGroupVersionParsingFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content successfully deleted, may be waiting on finalization reason: ContentDeleted status: "False" type: NamespaceDeletionContentFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content successfully removed reason: ContentRemoved status: "False" type: NamespaceContentRemaining - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content-preserving finalizers finished reason: ContentHasNoFinalizers status: "False" type: NamespaceFinalizersRemaining phase: Terminating
問題直接定位到,這個資源找不到。並且在查看資源與查看注冊資源時,輸出最下面會有error。不處理上文故障的處理辦法如下:
正文:
處理方法
[root@ECS1 ~]# kubectl get ns NAME STATUS AGE app-team1 Terminating 3d7h default Active 3d19h internal Active 2d23h kube-node-lease Active 3d19h kube-public Active 3d19h kube-system Active 3d19h [root@ECS1 ~]# kubectl delete ns/app-team1 namespace "app-team1" deleted ^C [root@ECS1 ~]#
沒辦法只能手動停止,不然卡到你天荒地老。
找到一個神奇的地方找到這么一段話
There's one situation that may require forcing finalization for a namespace. If you've deleted a namespace and you've cleaned out all of the objects under it, but the namespace still exists, deletion can be forced by updating the namespace subresource, finalize. This informs the namespace controller that it needs to remove the finalizer from the namespace and perform any cleanup:
大體意思就是如果刪除了命名空間,在已經清除空間下所有對象后。空間還在,那么需要通過更新名稱空間子資源來強制刪除。這種方式通知名稱空間控制器,我要從命名空間中刪除終結器並且執行清理所有操作。
這東西用的是restful請求方式,但是我這不安全端口都封掉了,開個代理吧(可以選擇用證書)
[root@ECS1 ~]# kubectl proxy --port=8081 Starting to serve on 127.0.0.1:8081
cat <<EOF | curl -X PUT \ localhost:8081/api/v1/namespaces/app-team1/finalize \ -H "Content-Type: application/json" \ --data-binary @- { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "app-team1" }, "spec": { "finalizers": null } } EOF
查看結果
[root@ECS1 ~]# cat <<EOF | curl -X PUT \ > localhost:8081/api/v1/namespaces/app-team1/finalize \ > -H "Content-Type: application/json" \ > --data-binary @- > { > "kind": "Namespace", > "apiVersion": "v1", > "metadata": { > "name": "app-team1" > }, > "spec": { > "finalizers": null > } > } > EOF { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "app-team1", "uid": "108e6665-9b70-422c-8f94-783347101836", "resourceVersion": "533794", "creationTimestamp": "2021-06-08T23:46:24Z", "deletionTimestamp": "2021-06-12T06:27:33Z", "managedFields": [ { "manager": "curl", "operation": "Update", "apiVersion": "v1", "time": "2021-06-12T06:58:32Z", "fieldsType": "FieldsV1", "fieldsV1": {"f:status":{"f:phase":{}}} } ] }, "spec": { }, "status": { "phase": "Terminating", "conditions": [ { "type": "NamespaceDeletionDiscoveryFailure", "status": "True", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "DiscoveryFailed", "message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource" }, { "type": "NamespaceDeletionGroupVersionParsingFailure", "status": "False", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "ParsedGroupVersions", "message": "All legacy kube types successfully parsed" }, { "type": "NamespaceDeletionContentFailure", "status": "False", "