Kubernetes資源管理

目錄貼：Kubernetes學習系列

1、資源模型

　　虛擬化技術是雲計算平台的基礎，其目標是對計算資源進行整合或划分，這是雲計算管理平台中的關鍵技術。虛擬化技術為雲計算管理乎台的資源管理提供了資源調配上的靈活性，從而使得雲計算管理平台可以通過虛擬化層整合或划分計算資源。

　　相比於虛擬機，新出現的容器技術使用了一系列的系統級別的機制，諸如利用Linux Namespace進行空間隔離，通過文件系統的掛載點決定容器可以訪問哪些文件，通過Cgroup確定每個容器可以利用多少資源。此外，容器之間共享同一個系統內核，這樣當同一個內核被多個容器使用時，內存的使用效率會得到提升。

　　容器和虛擬機兩大虛擬化技術，雖然實現方式完全不同，但是它們的資源需求和模型其實是類似的。容器像虛擬機一樣需要內存、CPU、硬盤空間和網絡帶寬，宿主機系統可以將虛擬機和容器都視作一個整體，為這個整體分配其所需的資源，並進行管理。當然， 虛擬機提供了專用操作系統的安全性和更牢固的邏輯邊界，而容器在資源邊界上比較松散，這帶來了靈活性以及不確定性。

　　Kubernetes是一個容器集群管理平台，Kubernetes需要統計整體平台的資源使用情況，合理地將資源分配給容器使用，並且要保證容器生命周期內有足夠的資源來保證其運行。 更進一步，如果資源發放是獨占的，即資源已發放給了個容器，同樣的資源不會發放給另外一個容器，對於空閑的容器來說占用着沒有使用的資源比如CPU是非常浪費的，Kubernetes需要考慮如何在優先度和公平性的前提下提高資源的利用率。

2、資源請求與限制

　　創建Pod的時候，可以指定計算資源（目前支持的資源類型有CPU和內存），即指定每個容器的資源請求（Request）和資源限制（Limit），資源請求是容器所需的最小資源需求，資源限制則是容器不能超過的資源上限。它們的大小關系是：

0<=request<=limit<=infinity

　　Pod的資源請求就是Pod中容器資源請求之和。Kubernetes在調度Pod時，會根據Node中的資源總量（通過cAdvisor接口獲得），以及該Node上已使用的計算資源，來判斷該Node是否滿足需求。

　　資源請求能夠保證Pod有足夠的資源來運行，而資源限制則是防止某個Pod無限制地使用資源，導致其他Pod崩潰。特別是在公有雲場景，往往會有惡意軟件通過搶占內存來攻擊平台。

　　原理：Docker 通過使用Linux Cgroup來實現對容器資源的控制，具體到啟動參數上是--memory和--cpu-shares。Kubernetes中是通過控制這兩個參數來實現對容器資源的控制。

以下給出某個Pod申請內存及CPU的示例：

[root@k8s-master demon2]# cat test-limit.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: test-limit
    role: master
  name: test-limit
spec:
  containers:
    - name: test-limit
      image: registry:5000/back_demon:1.0
      resources:
        requests:
          memory: "256Mi"
          cpu: "500m"
        limits:
          memory: "512Mi"
          cpu: "1000m"
      command:
      - /run.sh 

待Pod調度到具體某個機器上之后，在該機器上查詢對應容器的詳情，如下：

[root@k8s-node-4 home]# docker inspect 1fdbd6f1b39b
[
    {
        "Id": "1fdbd6f1b39b561d09084adafb382b721959e5edd0ee9538472313ed0a39162a",
        "Created": "2017-03-20T05:40:30.756006226Z",
        "Path": "/run.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 18603,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-03-20T05:40:31.113657323Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:9369911131d30b12759074e5b72356345446996bf6044950c2def787471e9b4c",
        "ResolvConfPath": "/var/lib/docker/containers/8fdb38a4d0074b07ff2f07e21fd8602fdbf4267eafc76179e931d4f5d9265940/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/8fdb38a4d0074b07ff2f07e21fd8602fdbf4267eafc76179e931d4f5d9265940/hostname",
        "HostsPath": "/var/lib/kubelet/pods/ba75e7a9-0d2f-11e7-b3d5-fa163ebba51b/etc-hosts",
        "LogPath": "/var/lib/docker/containers/1fdbd6f1b39b561d09084adafb382b721959e5edd0ee9538472313ed0a39162a/1fdbd6f1b39b561d09084adafb382b721959e5edd0ee9538472313ed0a39162a-json.log",
        "Name": "/k8s_test-limit.79cbd53f_test-limit_default_ba75e7a9-0d2f-11e7-b3d5-fa163ebba51b_efc94078",
        "RestartCount": 0,
        "Driver": "devicemapper",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/var/lib/kubelet/pods/ba75e7a9-0d2f-11e7-b3d5-fa163ebba51b/etc-hosts:/etc/hosts:Z",
                "/var/lib/kubelet/pods/ba75e7a9-0d2f-11e7-b3d5-fa163ebba51b/containers/test-limit/efc94078:/dev/termination-log:Z"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "container:8fdb38a4d0074b07ff2f07e21fd8602fdbf4267eafc76179e931d4f5d9265940",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "container:8fdb38a4d0074b07ff2f07e21fd8602fdbf4267eafc76179e931d4f5d9265940",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 984,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "seccomp=unconfined"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "docker-runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 1024,
            "Memory": 536870912,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 100000,
            "CpuQuota": 200000,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": -1,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "devicemapper",
            "Data": {
                "DeviceId": "210",
                "DeviceName": "docker-253:0-100693626-cb08877222f483f043fc45c5c4b024de8da9c393c3c06c6252d3c59d330dd4d4",
                "DeviceSize": "10737418240"
            }
        },
        "Mounts": [
            {
                "Source": "/var/lib/kubelet/pods/ba75e7a9-0d2f-11e7-b3d5-fa163ebba51b/etc-hosts",
                "Destination": "/etc/hosts",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/kubelet/pods/ba75e7a9-0d2f-11e7-b3d5-fa163ebba51b/containers/test-limit/efc94078",
                "Destination": "/dev/termination-log",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "test-limit",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "222/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "KUBERNETES_PORT_443_TCP_PROTO=tcp",
                "KUBERNETES_PORT_443_TCP_PORT=443",
                "FRONTEND_SERVICE_PORT=tcp://10.254.232.119:8080",
                "REDIS_SERVICE_SERVICE_PORT=6379",
                "REDIS_SERVICE_PORT_6379_TCP_ADDR=10.254.71.136",
                "KUBERNETES_SERVICE_PORT_HTTPS=443",
                "KUBERNETES_SERVICE_HOST=10.254.0.1",
                "KUBERNETES_PORT_443_TCP=tcp://10.254.0.1:443",
                "BACK_SERVICE_PORT_8080_TCP_PORT=8080",
                "FRONTEND_SERVICE_PORT_8080_TCP=tcp://10.254.232.119:8080",
                "FRONTEND_SERVICE_PORT_8080_TCP_ADDR=10.254.232.119",
                "REDIS_SERVICE_PORT_6379_TCP=tcp://10.254.71.136:6379",
                "REDIS_MASTER_PORT_6379_TCP_PORT=6379",
                "FRONTEND_PORT_80_TCP_PORT=80",
                "REDIS_MASTER_PORT_6379_TCP_ADDR=10.254.132.210",
                "REDIS_SLAVE_PORT_6379_TCP=tcp://10.254.104.23:6379",
                "REDIS_SLAVE_PORT_6379_TCP_PORT=6379",
                "BACK_SERVICE_SERVICE_HOST=10.254.246.51",
                "BACK_SERVICE_PORT=tcp://10.254.246.51:8080",
                "BACK_SERVICE_PORT_8080_TCP_PROTO=tcp",
                "FRONTEND_PORT=tcp://10.254.93.91:80",
                "REDIS_MASTER_SERVICE_HOST=10.254.132.210",
                "REDIS_MASTER_PORT_6379_TCP_PROTO=tcp",
                "KUBERNETES_SERVICE_PORT=443",
                "FRONTEND_SERVICE_PORT_8080_TCP_PROTO=tcp",
                "REDIS_MASTER_SERVICE_PORT=6379",
                "REDIS_SLAVE_SERVICE_HOST=10.254.104.23",
                "REDIS_SLAVE_PORT=tcp://10.254.104.23:6379",
                "REDIS_SERVICE_PORT_6379_TCP_PROTO=tcp",
                "REDIS_MASTER_PORT=tcp://10.254.132.210:6379",
                "KUBERNETES_PORT_443_TCP_ADDR=10.254.0.1",
                "BACK_SERVICE_SERVICE_PORT=8080",
                "FRONTEND_SERVICE_HOST=10.254.93.91",
                "FRONTEND_SERVICE_SERVICE_PORT=8080",
                "REDIS_SERVICE_SERVICE_HOST=10.254.71.136",
                "REDIS_SERVICE_PORT=tcp://10.254.71.136:6379",
                "BACK_SERVICE_PORT_8080_TCP=tcp://10.254.246.51:8080",
                "REDIS_SLAVE_SERVICE_PORT=6379",
                "REDIS_SLAVE_PORT_6379_TCP_PROTO=tcp",
                "REDIS_SLAVE_PORT_6379_TCP_ADDR=10.254.104.23",
                "KUBERNETES_PORT=tcp://10.254.0.1:443",
                "BACK_SERVICE_PORT_8080_TCP_ADDR=10.254.246.51",
                "FRONTEND_PORT_80_TCP=tcp://10.254.93.91:80",
                "FRONTEND_PORT_80_TCP_ADDR=10.254.93.91",
                "FRONTEND_SERVICE_PORT_8080_TCP_PORT=8080",
                "REDIS_MASTER_PORT_6379_TCP=tcp://10.254.132.210:6379",
                "FRONTEND_PORT_80_TCP_PROTO=tcp",
                "FRONTEND_SERVICE_SERVICE_HOST=10.254.232.119",
                "REDIS_SERVICE_PORT_6379_TCP_PORT=6379"
            ],
            "Cmd": null,
            "Image": "registry:5000/back_demon:1.0",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/run.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "io.kubernetes.container.hash": "79cbd53f",
                "io.kubernetes.container.name": "test-limit",
                "io.kubernetes.container.restartCount": "0",
                "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
                "io.kubernetes.pod.name": "test-limit",
                "io.kubernetes.pod.namespace": "default",
                "io.kubernetes.pod.terminationGracePeriod": "30",
                "io.kubernetes.pod.uid": "ba75e7a9-0d2f-11e7-b3d5-fa163ebba51b"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": null,
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": null
        }
    }
]

3、LimitRange

　　LimitRange設計的初衷是為了滿足以下場景：

　　　　能夠約束租戶的資源需求。

　　　　能夠約束容器的資源請求范圍。

　　　　能夠約束Pod的資源請求范圍。

　　　　能夠指定容器的默認資源限制。

　　　　能夠指定Pod的默認資源限制。

　　　　能夠約束資源請求和限制之間的比例。

4、Resource Quota

　　Kubernetes是一個多租戶架構，當多用戶或者團隊共享一個Kubernetes系統的時候，系統管理員需要防止租戶的資源搶占，定義好資源分配策略。比如Kubernetes系統共有20 核CPU和32GB內存，分配給A租戶5核CPU和16GB，分配給B租戶5核CPU 和8GB，預留10核CPU和8GB內存。這樣，租戶中所使用的CPU和內存的總和不能超過指定的資源配額，促使其更合理地使用資源。

　　Kubernetes提供API對象Resource Quota（資源配額）來實現資源配額，Resource Quota不僅可以作用於CPU和內存，另外還可以限制比如創建Pod的總數目、Service總數目、RC總數目等。

　　默認情況下，Namespace是沒有Resource Quota的，需要另外創建Resource Quota。一旦Namespace中有了Resource Quota，那么創建Pod的時候就必須制定資源請求，否則Pod就會創建失敗。