删除命名空间失败
今天在作测试的时候,清理集群。就把没用的都清理掉包括命名空间。但是发现失败了,一直卡在终止状态。
导致删除失败的原因一般有两种:
1、命名空间下还有资源在用,如果有删除,命名空间自动消失(原因:API服务器强制要求,只有在名称空间为空的情况下,并且只有在名称空间的情况下才能从存储中删除名称空间。)
2、就是名称空间下没有资源
开始分析原因
正常可以有三种方式
1、查看控制器日志 kube-controller-manager
2、查看api-resources会有报错信息在下面
获取所有注册ns的资源,提取能delete的部分(kubectl api-resources --namespaced=true --verbs=delete)
接下来查看注册服务kubectl get apiservice
3、失败查看当前yaml文件,通过get获取。其中有status字段中的message会有相关信息。
kubectl get ns t1 -o yaml
删除了一些留了主要的部分
[root@ECS1 ~]# kubectl get ns t1 -o yaml apiVersion: v1 kind: Namespace metadata: creationTimestamp: "2021-06-12T08:43:19Z" deletionTimestamp: "2021-06-12T08:43:24Z" name: t1 resourceVersion: "546044" uid: 62f4616c-ea45-441f-8ccf-4079d35c2796 spec: finalizers: - kubernetes status: conditions: - lastTransitionTime: "2021-06-12T08:43:29Z" message: 'Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource'
reason: DiscoveryFailed status: "True" type: NamespaceDeletionDiscoveryFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All legacy kube types successfully parsed reason: ParsedGroupVersions status: "False" type: NamespaceDeletionGroupVersionParsingFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content successfully deleted, may be waiting on finalization reason: ContentDeleted status: "False" type: NamespaceDeletionContentFailure - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content successfully removed reason: ContentRemoved status: "False" type: NamespaceContentRemaining - lastTransitionTime: "2021-06-12T08:43:29Z" message: All content-preserving finalizers finished reason: ContentHasNoFinalizers status: "False" type: NamespaceFinalizersRemaining phase: Terminating
问题直接定位到,这个资源找不到。并且在查看资源与查看注册资源时,输出最下面会有error。不处理上文故障的处理办法如下:
正文:
处理方法
[root@ECS1 ~]# kubectl get ns NAME STATUS AGE app-team1 Terminating 3d7h default Active 3d19h internal Active 2d23h kube-node-lease Active 3d19h kube-public Active 3d19h kube-system Active 3d19h [root@ECS1 ~]# kubectl delete ns/app-team1 namespace "app-team1" deleted ^C [root@ECS1 ~]#
没办法只能手动停止,不然卡到你天荒地老。
找到一个神奇的地方找到这么一段话
There's one situation that may require forcing finalization for a namespace. If you've deleted a namespace and you've cleaned out all of the objects under it, but the namespace still exists, deletion can be forced by updating the namespace subresource, finalize
. This informs the namespace controller that it needs to remove the finalizer from the namespace and perform any cleanup:
大体意思就是如果删除了命名空间,在已经清除空间下所有对象后。空间还在,那么需要通过更新名称空间子资源来强制删除。这种方式通知名称空间控制器,我要从命名空间中删除终结器并且执行清理所有操作。
这东西用的是restful请求方式,但是我这不安全端口都封掉了,开个代理吧(可以选择用证书)
[root@ECS1 ~]# kubectl proxy --port=8081 Starting to serve on 127.0.0.1:8081
cat <<EOF | curl -X PUT \ localhost:8081/api/v1/namespaces/app-team1/finalize \ -H "Content-Type: application/json" \ --data-binary @- { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "app-team1" }, "spec": { "finalizers": null } } EOF
查看结果
[root@ECS1 ~]# cat <<EOF | curl -X PUT \ > localhost:8081/api/v1/namespaces/app-team1/finalize \ > -H "Content-Type: application/json" \ > --data-binary @- > { > "kind": "Namespace", > "apiVersion": "v1", > "metadata": { > "name": "app-team1" > }, > "spec": { > "finalizers": null > } > } > EOF { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "app-team1", "uid": "108e6665-9b70-422c-8f94-783347101836", "resourceVersion": "533794", "creationTimestamp": "2021-06-08T23:46:24Z", "deletionTimestamp": "2021-06-12T06:27:33Z", "managedFields": [ { "manager": "curl", "operation": "Update", "apiVersion": "v1", "time": "2021-06-12T06:58:32Z", "fieldsType": "FieldsV1", "fieldsV1": {"f:status":{"f:phase":{}}} } ] }, "spec": { }, "status": { "phase": "Terminating", "conditions": [ { "type": "NamespaceDeletionDiscoveryFailure", "status": "True", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "DiscoveryFailed", "message": "Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: discovery.k8s.io/v1: the server could not find the requested resource, policy/v1: the server could not find the requested resource" }, { "type": "NamespaceDeletionGroupVersionParsingFailure", "status": "False", "lastTransitionTime": "2021-06-12T06:27:38Z", "reason": "ParsedGroupVersions", "message": "All legacy kube types successfully parsed" }, { "type": "NamespaceDeletionContentFailure", "status": "False", "