簡介： 混合雲K8s容器化應用彈性伸縮實戰
1. 前提條件
本最佳實踐的軟件環境要求如下：
應用環境：
①容器服務ACK基於專有雲V3.10.0版本。
②公共雲雲企業網服務CEN。
③公共雲彈性伸縮組服務ESS。
配置條件：
1）使用專有雲的容器服務或者在ECS上手動部署敏捷PaaS。
2）開通雲專線，打通容器服務所在VPC與公共雲上的VPC。
3）開通公共雲彈性伸縮組服務（ESS）。
2. 背景信息
本實踐基於K8s的業務集群運行在專有雲上，對測試業務進行壓力測試，主要基於以下三種產品和能力：
①利用阿里雲的雲企業網專線打通專有雲和公共雲，實現兩朵雲上VPC網絡互通。
②利用K8s（Kubernetes）的HPA能力，實現容器的水平伸縮。
③利用K8s的Cluster Autoscaler和阿里雲彈性伸縮組ESS能力實現節點的自動伸縮。
HPA（Horizontal Pod Autoscaler）是K8s的一種資源對象，能夠根據CPU、內存等指標對statefulset、deployment等對象中的pod數量進行動態伸縮，使運行在上面的服務對指標的變化有一定的自適應能力。
當被測試業務指標達到上限時，觸發HPA自動擴容業務pod；當業務集群無法承載更多pod時，觸發公共雲的ESS服務，在公共雲內擴容出ECS並自動添加到專有雲的K8s集群。
圖 1：架構原理圖
3. 配置HPA
本示例創建了一個支持HPA的nginx應用，創建成功后，當Pod的利用率超過本例中設置的20%利用率時，則會進行水平擴容，低於20%的時候會進行縮容。
1.若使用自建K8s集群，則通過yaml文件配置HPA
1）創建一個nginx應用，必須為應用設置request值，否則HPA不會生效。
apiVersion: app/v1beta2 kind: Deployment spec: template: metadata: creationTimestamp: null labels: app: hpa-test spec: dnsPolicy: ClusterFirst terminationGracePeriodSeconds:30 containers: image: '192.168.**.***:5000/admin/hpa-example:v1' imagePullPolicy: IfNotPresent terminationMessagePolicy:File terminationMessagePath:/dev/termination-log name: hpa-test resources: requests: cpu: //必須設置request值 securityContext: {} restartPolicy:Always schedulerName:default-scheduler replicas: 1 selector: matchLabels: app: hpa-test revisionHistoryLimit: 10 strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 25% progressDeadlineSeconds: 6002）創建HPA。
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
annotations:
autoscaling.alpha.kubernetes.io/conditions:'[{"type":"AbleToScale","status":"True","lastTransitionTime":"2020-04-29T06:57:28Z","reason":"ScaleDownStabilized","message":"recent recommendations were higher than current one, applying the highest recent recommendation"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2020-04-29T06:57:28Z","reason":"ValidMetricFound","message":"theHPA was able to successfully calculate a replica count from cpu resource utilization(percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2020-04-29T06:57:28Z","reason":"DesiredWithinRange","message":"thedesired count is within the acceptable range"}]' autoscaling.alpha.kubernetes.io/currentmetrics:'[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":0,"currentAverageValue":"0"}}]' creationTimestamp: 2020-04-29T06:57:13Z name: hpa-test namespace: default resourceVersion: "3092268" selfLink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/hpa01 uid: a770ca26-89e6-11ea-a7d7-00163e0106e9 spec: maxReplicas: //設置pod數量 minReplicas: 1 scaleTargetRef: apiVersion: apps/v1beta2 kind: Deployment name: centos targetCPUUtilizationPercentage://設置CPU閾值2.若使用阿里雲容器服務，需要在部署應用時選擇配置HPA
圖2：訪問設置
4. 配置Cluster Autoscaler
資源請求（Request）的正確、合理設置，是彈性伸縮的前提條件。節點自動伸縮組件基於K8s資源調度的分配情況進行伸縮判斷，節點中資源的分配通過資源請（Request）進行計算。
當Pod由於資源請求（Request）無法滿足並進入等待（Pending）狀態時，節點自動伸縮組件會根據彈性伸縮組配置信息中的資源規格以及約束配置，計算所需的節點數目。
如果可以滿足伸縮條件，則會觸發伸縮組的節點加入。而當一個節點在彈性伸縮組中且節點上Pod的資源請求低於閾值時，節點自動伸縮組件會將節點進行縮容。
1.配置彈性伸縮組ESS
1）創建ESS彈性伸縮組，記錄最小實例數和最大實例數。
圖3：修改伸縮組
2）創建伸縮配置，記錄伸縮配置的id。
圖4：伸縮配置
#!/bin/sh yum install -y ntpdate && ntpdate -u ntp1.aliyun.com && curl http:// example.com/public/hybrid/attach_local_node_aliyun.sh | bash -s -- --docker-version 17.06.2-ce-3 --token 9s92co.y2gkocbumal4fz1z --endpoint 192.168.**.***:6443 --cluster-dns 10.254.**.** --region cn-huhehaote echo "{" > /etc/docker/daemon.json echo "\"registry-mirrors\": [" >> /etc/docker/daemon.json echo "\"https://registry-vpc.cn-huhehaote.aliyuncs.com\"" >> /etc/docker/daemon.json echo "]," >> /etc/docker/daemon.json echo "\"insecure-registries\": [\"https://192.168.**.***:5000\"]" >> /etc/docker/daemon.json echo "}" >> /etc/docker/daemon.json systemctl restart docker 2.K8s集群部署autoscaler
kubectl apply -f ca.yml參考ca.yml創建autoscaler，注意修改如下配置與實際環境相對應。
access-key-id: "TFRBSWlCSFJyeHd2QXZ6****" access-key-secret: "bGIyQ3NuejFQOWM0WjFUNjR4WTVQZzVPRXND****" region-id: "Y24taHVoZWhh****"ca.yal代碼如下：
--- apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-addon: cluster-autoscaler.addons.k8s.io k8s-app: cluster-autoscaler name: cluster-autoscaler namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: cluster-autoscaler labels: k8s-addon: cluster-autoscaler.addons.k8s.io k8s-app: cluster-autoscaler rules: - apiGroups: [""] resources: ["events","endpoints"] verbs: ["create", "patch"] - apiGroups: [""] resources: ["pods/eviction"] verbs: ["create"] - apiGroups: [""] resources: ["pods/status"] verbs: ["update"] - apiGroups: [""] resources: ["endpoints"] resourceNames: ["cluster-autoscaler"] verbs: ["get","update"] - apiGroups: [""] resources: ["nodes"] verbs: ["watch","list","get","update"] - apiGroups: [""] resources: ["pods","services","replicationcontrollers","persistentvolumeclaims","persistentvolumes"] verbs: ["watch","list","get"] - apiGroups: ["extensions"] resources: ["replicasets","daemonsets"] verbs: ["watch","list","get"] - apiGroups: ["policy"] resources: ["poddisruptionbudgets"] verbs: ["watch","list"] - apiGroups: ["apps"] resources: ["statefulsets"] verbs: ["watch","list","get"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["watch","list","get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: cluster-autoscaler namespace: kube-system labels: k8s-addon: cluster-autoscaler.addons.k8s.io k8s-app: cluster-autoscaler rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["create","list","watch"] - apiGroups: [""] resources: ["configmaps"] resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"] verbs: ["delete","get","update","watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: cluster-autoscaler labels: k8s-addon: cluster-autoscaler.addons.k8s.io k8s-app: cluster-autoscaler roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-autoscaler subjects: - kind: ServiceAccount name: cluster-autoscaler namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: cluster-autoscaler namespace: kube-system labels: k8s-addon: cluster-autoscaler.addons.k8s.io k8s-app: cluster-autoscaler roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: cluster-autoscaler subjects: - kind: ServiceAccount name: cluster-autoscaler namespace: kube-system --- apiVersion: v1 kind: Secret metadata: name: cloud-config namespace: kube-system type: Opaque data: access-key-id: "TFRBSWlCSFJyeHd2********" access-key-secret: "bGIyQ3NuejFQOWM0WjFUNjR4WTVQZzVP*********" region-id: "Y24taHVoZW********" --- apiVersion: apps/v1 kind: Deployment metadata: name: cluster-autoscaler namespace: kube-system labels: app: cluster-autoscaler spec: replicas: 1 selector: matchLabels: app: cluster-autoscaler template: metadata: labels: app: cluster-autoscaler spec: dnsConfig: nameservers: - 100.XXX.XXX.XXX