【原創】Pod和容器的LimitRange原理和實踐總結

本文轉載自查看原文 2021-01-29 21:57 328 K8s

一、背景介紹

通常情況下，Pod中的容器可以無限制的使用節點上的CPU和內存資源，在共享資源和資源有限的情況下，若不加以限制，某個集群或命名空間的資源可能會消耗殆盡，導致其節點上優先級低的Pod發生驅逐、重新調度、重啟等問題，因此針對資源限額需要有針對命名空間級別的使用限制，以及針對Pod和容器級別的使用限制，LimitRange就是在后者的背景下誕生的。

二、應用場景

1、適用對集群的某個命名空間中的容器設置默認的統一的request和limit值，配合Resource Quotas更好的實現命名空間級別的資源限額。

2、適用對單個Pod的CPU和內存資源限額控制，如實現Pod中所有容器CPUlimit之和不得超過某值。

3、適用對單個容器的CPU和內存資源限額控制，如實現容器的CPUlimit值不得超過某值。

三、實踐

一、LimitRange設置默認request和limit

適用場景：適用對集群的某個命名空間中的容器設置默認的統一的request和limit值

【新建limitrange】

LimitRange針對某個namespace的Pod或容器的CPU和內存的默認值、request和limit值實現管控，因此限制的的范圍是在namespace下，在創建LimitRange之前，先創建1個用於測試的namespace

 kubectl create namespace limitrange-test-ns

[root@k8s-master limitrange]# kubectl get ns |grep limitrange-test-ns
limitrange-test-ns   Active   78m

namespace創建成功后，在此命名空間下創建limitrange

[root@k8s-master limitrange]# ls
limitrange.yaml
[root@k8s-master limitrange]# cat limitrange.yaml 
apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range-test
  namespace: limitrange-test-ns   #指定命名空間
spec:
  limits:
  - default:
      memory: 256Mi               
    defaultRequest:
      memory: 128Mi
    type: Container

kubectl apply -f limitrange.yaml

[root@k8s-master limitrange]# kubectl describe limitrange mem-limit-range-test -n limitrange-test-ns 
Name:       mem-limit-range-test
Namespace:  limitrange-test-ns
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   memory    -    -    128Mi            256Mi          -

[root@k8s-master limitrange]# kubectl describe ns limitrange-test-ns
Name:         limitrange-test-ns
Labels:       <none>
Annotations:  <none>
Status:       Active

No resource quota.

Resource Limits
 Type       Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
 ----       --------  ---  ---  ---------------  -------------  -----------------------
 Container  memory    -    -    128Mi            256Mi          -

可以看到已為limitrange-test-ns的命名空間下所有的容器設置了默認的request值和limit值，接下來看下在容器中是如何生效的

【容器limitrange生效驗證】

1、容器未設置limit，且未設置request

[root@k8s-master limitrange]# cat container-limitrange.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-container-men-limit-test
spec:
  containers:
  - name: container-limlitrange-test
    image: nginx:latest
    imagePullPolicy: IfNotPresent

[root@k8s-master limitrange]# kubectl get pod -o wide -n limitrange-test-ns 
NAME                           READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
pod-container-men-limit-test   1/1     Running   0          28s   10.122.235.216   k8s-master   <none>           <none>
[root@k8s-master limitrange]# curl 10.122.235.216:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

可以看到此Pod已經running，且業務正常，在創建此Pod容器部分並未設置默認值，但是Pod所在的命名空間有配置過默認的limitrange策略，驗證下此策略是否生效到Pod的容器部分

spec:
  containers:
  - image: nginx:latest
    imagePullPolicy: IfNotPresent
    name: container-limlitrange-test
    resources:
      limits:
        memory: 256Mi
      requests:
        memory: 128Mi

總結：容器未設置limit，且未設置request，則容器運行起來后的request和limit值默認與limitrange type為container設置的request和limit的保持一致。

2、容器中設置limit，但是未設置request

[root@k8s-master limitrange]# cat container-limitrange.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-container-men-limit-test
  namespace: limitrange-test-ns
spec:
  containers:
  - name: container-limlitrange-test
    image: nginx:latest
    imagePullPolicy: IfNotPresent
    resources:
      limits:
        memory: "500Mi"

只設置內存的limit值且大於limitrange設置的limit值

[root@k8s-master limitrange]# kubectl apply -f container-limitrange.yaml 
pod/pod-container-men-limit-test created
[root@k8s-master limitrange]# kubectl get pod -n limitrange-test-ns 
NAME                           READY   STATUS    RESTARTS   AGE
pod-container-men-limit-test   1/1     Running   0          9s

[root@k8s-master limitrange]# kubectl get pod pod-container-men-limit-test -o json -n limitrange-test-ns 
"spec": {
        "containers": [
            {
                "image": "nginx:latest",
                "imagePullPolicy": "IfNotPresent",
                "name": "container-limlitrange-test",
                "resources": {
                    "limits": {
                        "memory": "500Mi"
                    },
                    "requests": {
                        "memory": "500Mi"
                    }
                },
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File"

總結：容器中設置實際limit，但是未設置request，容器運行成功后request=實際limit

3、容器中未設置limit，但是設置request

[root@k8s-master limitrange]# cat container-limitrange.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-container-men-limit-test
  namespace: limitrange-test-ns
spec:
  containers:
  - name: container-limlitrange-test
    image: nginx:latest
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        memory: "150Mi"

[root@k8s-master limitrange]# kubectl apply -f container-limitrange.yaml
pod/pod-container-men-limit-test created

[root@k8s-master limitrange]# kubectl get pod pod-container-men-limit-test -o json -n limitrange-test-ns
"spec": {
        "containers": [
            {
                "image": "nginx:latest",
                "imagePullPolicy": "IfNotPresent",
                "name": "container-limlitrange-test",
                "resources": {
                    "limits": {
                        "memory": "256Mi"
                    },
                    "requests": {
                        "memory": "150Mi"
                    }
                },
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File",

總結：容器中未設置limit，但是設置實際request，則容器運行起來后request=實際request，limit=limitrange.limit

二、Pod和容器的max、min、maxLimitRequestRatio

適用場景：控制容器或Pod的CPU或內存資源使用，如容器的CPU的request不得小於多少，limit不得大於多少，不滿足條件則創建失敗，先介紹下參數：

容器部分：
1、defaultRequest：容器默認的request值，若不指定request值，則默認為此值

2、max：容器的實際設置的limit值應小於等於此值

3、min：容器實際設置的request值應大於等於此值

4、maxLimitRequestRatio：Max Limit/Request Ratio，為容器CPU或內存的Limit/Request值應小於等於此值

Pod部分：

1、max：Pod中所有容器的實際設置的limit之和值應小於等於此值

2、min：Pod中所有容器實際設置的request值之和應大於等於此值

3、maxLimitRequestRatio：Max Limit/Request Ratio，為Pod中所有容器CPU或內存的Limit之和/Request之和值應小於等於此值

演示如下：

 kubectl create namespace limitrange-02-test-ns

[root@k8s-master limitrange]# cat limitrange.yaml 
apiVersion: v1
kind: LimitRange
metadata:
  name: limitrange-test
  namespace: limitrange-02-test-ns 
spec:
 limits:
 - max:
    cpu: 2000m
    memory: 1000Mi
   min:
    cpu: 200m
    memory: 6Mi
   maxLimitRequestRatio:
    cpu: 3
    memory: 2
   type: Pod
 - default:                      #容器的默認limit值
    cpu: 300m
    memory: 200Mi
   defaultRequest:               #容器的默認request值
    cpu: 200m
    memory: 100Mi
   max:                          #容器的limit值不得大於max
    cpu: 1000m
    memory: 500Mi
   min:                          #容器的request的值不得小於min
    cpu: 100m
    memory: 3Mi
   maxLimitRequestRatio:         #容器的limit/request不得大於，如容器的CPU的limit/request不得大於4
    cpu: 5
    memory: 4
   type: Container

 kubectl apply -f container-02-limitrange.yaml

[root@k8s-master limitrange]# kubectl describe ns limitrange-02-test-ns 
Name:         limitrange-02-test-ns
Labels:       <none>
Annotations:  <none>
Status:       Active

No resource quota.

Resource Limits
 Type       Resource  Min   Max     Default Request  Default Limit  Max Limit/Request Ratio
 ----       --------  ---   ---     ---------------  -------------  -----------------------
 Pod        cpu       200m  2       -                -              3
 Pod        memory    6Mi   1000Mi  -                -              2
 Container  cpu       100m  1       200m             300m           5
 Container  memory    3Mi   500Mi   100Mi            200Mi          4

可以看到設置的Pod和容器設置的max和min已經生效了，接下來驗證下設置的策略是否生效，創建1個Pod的容器的內存limit值大於500Mi

[root@k8s-master limitrange]# cat container-02-limitrange.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-limitrange-test
namespace: limitrange-02-test-ns
spec:
containers:
- name: container-limlitrange-test
image: nginx:latest
imagePullPolicy: IfNotPresent
resources:
requests:
memory: "100Mi"
limits:
memory: "600Mi"

[root@k8s-master limitrange]# kubectl apply -f container-02-limitrange.yaml
Error from server (Forbidden): error when creating "container-02-limitrange.yaml": pods "pod-limitrange-test" is forbidden: [memory max limit to request ratio per Pod is 2, but provided ratio is 6.000000, maximum memory usage per Container is 500Mi, but limit is 600Mi, memory max limit to request ratio per Container is 4, but provided ratio is 6.000000]

提示容器的limit/request值為6大於4，且limit為600Mi大於設置最大max值500Mi，因此創建失敗，在Pod只有1個容器的情況下podlimit/request值為6大於2，同樣做下request值小於min值的測試

[root@k8s-master limitrange]# cat container-02-limitrange.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-limitrange-test
  namespace: limitrange-02-test-ns
spec:
  containers:
  - name: container-limlitrange-test
    image: nginx:latest
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        memory: "1Mi"    #小於min值
      limits:
        memory: "200Mi

[root@k8s-master limitrange]# kubectl apply -f container-02-limitrange.yaml 
Error from server (Forbidden): error when creating "container-02-limitrange.yaml": pods "pod-limitrange-test" is forbidden: [minimum memory usage per Pod is 6Mi, but request is 1048576, memory max limit to request ratio per Pod is 2, but provided ratio is 200.000000, minimum memory usage per Container is 3Mi, but request is 1Mi, memory max limit to request ratio per Container is 4, but provided ratio is 200.000000]

提示容器request的值為1Mi小於min值3Mi，且limit/request=200大於4，同樣不滿足Pod的min為6和ratio，因此創建失敗。

四、總結

本文描述了LimitRange的誕生背景、適用場景、以及具體如何實現和控制Pod、容器的資源限額，產品設計上主要考慮如下幾點：

1、產品設計上需要方便用戶低門檻快速設置默認request、limit值，配合Resource Quotas提供強大而穩定的K8S原生資源限額能力。

2、max、min、Limit/Request Ratio等功能如果開放給對K8S不了解的用戶，門檻較高，若無意配置完成且生效，可能會導致容器的創建異常，且不知如何解決，是否直接開放需要認真考慮。

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 ResourceQuota和LimitRange實踐指南 Kubernetes實踐之深入掌握Pod——在容器內獲取Pod信息容器與 Pod 【原創】Kubernetes-PV和PVC的原理和實踐 kafka原理和實踐（六）總結升華【原創】Kubernetes-Statefulset（有狀態負載）原理和實踐 pod 和容器的關系 Kubernetes基石-pod容器 Pod容器共享Volume 在容器內獲取Pod的信息