Kubernetes系統安全-准入控制(admission control)
作者：尹正傑
版權聲明：原創作品，謝絕轉載！否則將追究法律責任。
一.准入控制(admission control)概述
1>.常見的准入控制器(Admission Controllers)
AlwaysAdmit(DEPRECATED):
該准入控制器已被廢棄,總是允許，所有的請求(包括不符合定義)的規則都允許，如果有不符合的規則請求會日志記錄而已，和臨時關閉selinux的功能有點像。
AlwaysPullImages:
無論你拉取鏡像的規則定義成什么類型，總是會去拉取鏡像，也就是強制避免使用本地鏡像，即無視本地鏡像定義的拉取規則。
AlwaysDeny(DEPRECATED):
該准入控制器已被廢棄,和AlwaysAdmit相反，所有請求(包括符合定義)的規則拒絕。
DefaultStorageClass:
指定默認的存儲類。
DefaultTolerationSeconds
DenyEscalatingExec
EventRateLimit(alpha)
ExtendedResourceToleration
ImagePolicyWebhook
Initializers(alpha)
LimitPodHardAnitiAffinityTopology
LimitRanger:
允許在名稱空間中創建一個LimitRange資源，任何Pod都需要指定一個資源上限和下限的范圍，若未指定會使用默認值。
MutaingAdmission Webhook(beta in 1.9)
NamespaceAutoProvision
NamespaceExists:
檢查名稱空間是否存在的控制器。
NamespaceLifecycle
NodeRestriction
OwnerReferencesPermissionEnforcement
PodNodeSelector
PersistentVolumeClaimResize
PodPreset
PodSecurityPolicy:
為Pod定義安全策略。
PodTolerationRestriction
Priority
ResourceQuota:
允許在名稱空間中創建一個ResourceQuota資源,可以明確指定Pod使用資源的配額，比如內存不允許超過20G，允許使用的總的PVC數量等K8S集群資源。
SecurityContextDeny
ServiceAccount:
比如創建的Pod內部默認有一個存儲卷，每個存儲卷對應了一個secret用於ApiServer認證,這就是ServiceAccount來定義的。
Storage Object in Use Protection
ValidatingAdmission Webhook(alpha in 1.8;beta in 1.9)
2>.打開或者關閉控制器
Kubernetes API server標志enable adminimission plugins接受在修改集群中的對象之前要調用的許可控制插件的逗號分隔列表。使用"--enable-admission-plugins="指定，如下圖所示。 Kubernetes API服務器標志disable admission plugins接受一個逗號分隔的要禁用的許可控制插件列表，即使它們在默認啟用的插件列表中。使用"--disable-admission-plugins="。
二.LimitRange and LimitRanger
Pod 對象雖然支持使用requests和limits進行可用計算資源配置，但它們卻非強制選項; LimitRange的主要目的是確保請求和/限制根據其規范自動關聯到容器; 如果容器是在具有默認內存限制的命名空間中創建的，並且該容器未指定其自己的內存限制，則會為該容器分配默認內存限制; 由limit range對象定義的限制范圍枚舉pod和container級別的命名空間中的計算資源約束，並指定pod或container可以消耗的資源量。 對項目中的每個LimitRange對象評估所有資源創建和修改請求，如果資源未設置顯式值，並且約束支持默認值，則將默認值應用於資源。 LimitRanger將觀察傳入的請求，並確保它不違反命名空間中LimitRange對象中枚舉的任何約束: 所有資源創建和修改請求都是根據命名空間中的每個LimitRange對象計算的; 如果資源違反任何枚舉約束，則資源將被拒絕。 如果資源未設置顯式值，並且約束支持默認值，則將默認值應用於資源。 LimitRanger還可用於將默認資源請求應用於未指定任何的pod；當前，默認LimitRanger將0.1 CPU需求應用於默認命名空間中的所有pod。
1>.創建LimitRange資源
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/limitrange-demo.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/limitrange-demo.yaml apiVersion: v1 kind: LimitRange metadata: name: cpu-limit-range namespace: yinzhengjie-admission-control spec: limits: - default: cpu: 1000m defaultRequest: cpu: 1000m min: cpu: 500m max: cpu: 2000m maxLimitRequestRatio: cpu: 4 type: Container [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl create ns yinzhengjie-admission-control namespace/yinzhengjie-admission-control created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get ns NAME STATUS AGE default Active 15d develop Active 13h kube-node-lease Active 15d kube-public Active 15d kube-system Active 15d kubernetes-dashboard Active 14h yinzhengjie-admission-control Active 7s [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/limitrange-demo.yaml limitrange/cpu-limit-range created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl describe limitrange cpu-limit-range -n yinzhengjie-admission-control Name: cpu-limit-range Namespace: yinzhengjie-admission-control Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Container cpu 500m 2 1 1 4 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
2>.使用默認的LimitRange
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml apiVersion: v1 kind: Pod metadata: name: pod-demo namespace: yinzhengjie-admission-control spec: containers: - image: ikubernetes/myapp:v1 imagePullPolicy: IfNotPresent name: myapp [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml pod/pod-demo created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-admission-control NAME READY STATUS RESTARTS AGE pod-demo 1/1 Running 0 18s [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe pods -n yinzhengjie-admission-control Name: pod-demo Namespace: yinzhengjie-admission-control Priority: 0 Node: node203.yinzhengjie.org.cn/172.200.1.203 Start Time: Thu, 20 Feb 2020 03:37:13 +0800 Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"pod-demo","namespace":"yinzhengjie-admission-control"},"spec":{"conta... kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container myapp; cpu limit for container myapp Status: Running IP: 10.244.3.4 IPs: IP: 10.244.3.4 Containers: myapp: Container ID: docker://503619b6cd5b1f6e7a0f0394d9710df9d6fe6621c16fdd06466520e3df2989e7 Image: ikubernetes/myapp:v1 Image ID: docker-pullable://ikubernetes/myapp@sha256:9c3dc30b5219788b2b8a4b065f548b922a34479577befb54b03330999d30d513 Port: <none> Host Port: <none> State: Running Started: Thu, 20 Feb 2020 03:37:22 +0800 Ready: True Restart Count: 0 Limits: cpu: 1 Requests: cpu: 1 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-r5wq6 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-r5wq6: Type: Secret (a volume populated by a Secret) SecretName: default-token-r5wq6 Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 25s default-scheduler Successfully assigned yinzhengjie-admission-control/pod-demo to node203.yinzhengjie.org.cn Normal Pulling 24s kubelet, node203.yinzhengjie.org.cn Pulling image "ikubernetes/myapp:v1" Normal Pulled 16s kubelet, node203.yinzhengjie.org.cn Successfully pulled image "ikubernetes/myapp:v1" Normal Created 16s kubelet, node203.yinzhengjie.org.cn Created container myapp Normal Started 16s kubelet, node203.yinzhengjie.org.cn Started container myapp [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-admission-control NAME READY STATUS RESTARTS AGE pod-demo 1/1 Running 0 2m21s [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl delete -f /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml pod "pod-demo" deleted [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-admission-control No resources found in yinzhengjie-admission-control namespace. [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
3>.使用自定義的LimitRange(如果指定的資源范圍不在范圍內則會無法創建Pod)
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml apiVersion: v1 kind: Pod metadata: name: pod-demo namespace: yinzhengjie-admission-control spec: containers: - image: ikubernetes/myapp:v1 imagePullPolicy: IfNotPresent name: myapp resources: requests: cpu: 500m limits: cpu: 1500m [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe limitrange cpu-limit-range -n yinzhengjie-admission-control Name: cpu-limit-range Namespace: yinzhengjie-admission-control Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Container cpu 500m 2 1 1 4 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/pod/test-limit-range.yaml pod/pod-demo created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-admission-control pod-demo -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-demo 1/1 Running 0 59s 10.244.3.5 node203.yinzhengjie.org.cn <none> <none> [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe pods -n yinzhengjie-admission-control pod-demo Name: pod-demo Namespace: yinzhengjie-admission-control Priority: 0 Node: node203.yinzhengjie.org.cn/172.200.1.203 Start Time: Thu, 20 Feb 2020 03:49:42 +0800 Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"pod-demo","namespace":"yinzhengjie-admission-control"},"spec":{"conta... Status: Running IP: 10.244.3.5 IPs: IP: 10.244.3.5 Containers: myapp: Container ID: docker://a991046c9cff1008ae93458dfd90c729794ac7a794df9e9507c92b8aa84646c3 Image: ikubernetes/myapp:v1 Image ID: docker-pullable://ikubernetes/myapp@sha256:9c3dc30b5219788b2b8a4b065f548b922a34479577befb54b03330999d30d513 Port: <none> Host Port: <none> State: Running Started: Thu, 20 Feb 2020 03:49:43 +0800 Ready: True Restart Count: 0 Limits: cpu: 1500m Requests: cpu: 500m Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-r5wq6 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-r5wq6: Type: Secret (a volume populated by a Secret) SecretName: default-token-r5wq6 Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 23s default-scheduler Successfully assigned yinzhengjie-admission-control/pod-demo to node203.yinzhengjie.org.cn Normal Pulled 22s kubelet, node203.yinzhengjie.org.cn Container image "ikubernetes/myapp:v1" already present on machine Normal Created 22s kubelet, node203.yinzhengjie.org.cn Created container myapp Normal Started 22s kubelet, node203.yinzhengjie.org.cn Started container myapp [root@master200.yinzhengjie.org.cn ~]#
三.ResourceQuota
由ResourceQuota對象定義的資源配額提供限制每個命名空間聚合資源消耗的約束。
它可以按類型限制可以在命名空間中創建的對象的數量，以及該命名空間中的資源可能消耗的計算資源和存儲的總量。
LimitRange用於定義單個Pod對象上計算資源的requests及limits，而ResourceQuota則負責為整個namespace設定資源配額。
當特定命名空間中存在資源配額時，將在該命名空間中強制資源配額。
ResourceQuota對象的生效依賴於ResourceQuota Admission Controller.
1>.創建resourcequota資源
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/resoucequota-demo.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/resoucequota-demo.yaml apiVersion: v1 kind: ResourceQuota metadata: name: quota-example namespace: yinzhengjie-admission-control spec: hard: pods: "5" requests.cpu: "1" requests.memory: 1Gi limits.cpu: "2" limits.memory: 2Gi count/deployments.apps: "2" count/deployments.extensions: "2" persistentvolumeclaims: "2" [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/resoucequota-demo.yaml resourcequota/quota-example created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get resourcequota -n yinzhengjie-admission-control NAME CREATED AT quota-example 2020-02-19T20:38:34Z [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe resourcequota -n yinzhengjie-admission-control Name: quota-example Namespace: yinzhengjie-admission-control Resource Used Hard -------- ---- ---- count/deployments.apps 0 2 count/deployments.extensions 0 2 limits.cpu 1500m 2 limits.memory 0 2Gi persistentvolumeclaims 0 2 pods 1 5 requests.cpu 500m 1 requests.memory 0 1Gi [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
2>.使用自定義的resourcequota的資源限制
[root@master200.yinzhengjie.org.cn ~]# kubectl describe resourcequota -n yinzhengjie-admission-control Name: quota-example Namespace: yinzhengjie-admission-control Resource Used Hard -------- ---- ---- count/deployments.apps 0 2 count/deployments.extensions 0 2 limits.cpu 1500m 2 limits.memory 0 2Gi persistentvolumeclaims 0 2 pods 1 5 requests.cpu 500m 1 requests.memory 0 1Gi [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/pod-demo.yaml [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/pod-demo.yaml apiVersion: v1 kind: Pod metadata: name: mynginx namespace: yinzhengjie-admission-control labels: app: pod-demo rel: stable spec: containers: - name: mynginx image: nginx:1.14-alpine resources: requests: cpu: 500m memory: 500Mi limits: cpu: 500m memory: 1Gi [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/pod-demo.yaml pod/mynginx created [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubectl describe resourcequota -n yinzhengjie-admission-control Name: quota-example Namespace: yinzhengjie-admission-control Resource Used Hard -------- ---- ---- count/deployments.apps 0 2 count/deployments.extensions 0 2 limits.cpu 2 2 limits.memory 1Gi 2Gi persistentvolumeclaims 0 2 pods 2 5 requests.cpu 1 1 requests.memory 500Mi 1Gi [root@master200.yinzhengjie.org.cn ~]#
四.PodSecurityPolicy(定義Pod的安全策略)
1>.定義受限制的pod的安全策略參考配置清單
[root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# cat psp-restricted.yaml apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default' seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' spec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - 'configMap' - 'emptyDir' - 'projected' - 'secret' - 'downwardAPI' - 'persistentVolumeClaim' hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: 'MustRunAsNonRoot' seLinux: rule: 'RunAsAny' supplementalGroups: rule: 'MustRunAs' ranges: - min: 1 max: 65535 fsGroup: rule: 'MustRunAs' ranges: - min: 1 max: 65535 readOnlyRootFilesystem: false [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]#
2>.定義有特權的pod的安全策略參考配置清單
[root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# cat psp-privileged.yaml apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: privileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' spec: privileged: true allowPrivilegeEscalation: true allowedCapabilities: - '*' volumes: - '*' hostNetwork: true hostPorts: - min: 0 max: 65535 hostIPC: true hostPID: true runAsUser: rule: 'RunAsAny' seLinux: rule: 'RunAsAny' supplementalGroups: rule: 'RunAsAny' fsGroup: rule: 'RunAsAny' [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]#
3>.安全策略調用參考配置文件
[root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# cat clusterrole-with-psp.yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: psp:restricted rules: - apiGroups: ['policy'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: - restricted --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: psp:privileged rules: - apiGroups: ['policy'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: - privileged [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]#
[root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# cat clusterrolebinding-with-psp.yaml kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: restricted-psp-user roleRef: kind: ClusterRole name: psp:restricted apiGroup: rbac.authorization.k8s.io subjects: - kind: Group apiGroup: rbac.authorization.k8s.io name: system:authenticated --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: privileged-psp-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: psp:privileged subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:masters - apiGroup: rbac.authorization.k8s.io kind: Group name: system:node - apiGroup: rbac.authorization.k8s.io kind: Group name: system:serviceaccounts:kube-system [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]# [root@master200.yinzhengjie.org.cn ~/Kubernetes_Advanced_Practical/chapter10]#
五.准入控制器擴展
Admission Controllers代碼必須要編譯進kube-apiserver,且只能在程序啟動時進行配置，於是，Kubernetes又特地引入了Admission Webhooks(beta in 1.9)和Initializers(alpha)來嘗試突破此限制，以允許用戶單獨開發主奴人控制器並運行時進行配置。 允許webhook是接收請求並對其執行操作的HTTP回調。 您可以定義兩種類型的許可Webhook，驗證許可Webhook和變異許可Webhook。 通過驗證許可webhook，您可以拒絕執行自定義許可策略的請求 使用變異的允許webhook，您可以更改請求以強制使用自定義默認值。