k8s的彈性伸縮（HPA）

在kubernetes中，我們使用pod對外提供服務。這時候，我們需要以下兩種情形需要關注：

• pod因為不明原因掛掉，導致服務不可用
• Pod在高負荷的情況下，不能支撐我們的服務

如果我們人工監控pods，人工進行調整副本那么這個工作量無疑是巨大的，但kubernetes已經有了相應的機制來應對了。

那么今天就來介紹一下在k8s 1.6中的彈性伸縮的實施

k8s是kubernetes的官方簡稱
HPA全稱Horizontal Pod Autoscaler

HPA的原理

Kubernetes有一個HPA(Horizontal Pod Autoscaler)的資源，可以實現基於CPU使用率的Pod自動伸縮的功能。HPA基於Master Node上的kube-controller-manager服務啟動參數–horizontal-pod-autoscaler-sync-period定義的時長(默認為30秒)，周期性的檢測Pod的CPU使用率(需要事先安裝heapster)。如果需要設置–horizontal-pod-autoscaler-sync-period可以在Master Node上的/etc/default/kube-controller-manager中修改。

graph TD A[ms-a] --> B(RC/Deployment) A1[ms-b] --> B(RC/Deployment) A3[ms-c] --> B(RC/Deployment) A4[其他服務] --> B(RC/Deployment) B --- C(Horizontal Pod Autoscaler)

安裝Heapster

K8S從1.8版本開始，CPU、內存等資源的metrics信息可以通過 Metrics API來獲取，用戶可以直接獲取這些metrics信息（例如通過執行kubect top命令），HPA使用這些metics信息來實現動態伸縮，但是在之前我們使用Heapster來收集節點的相關數據

導入相關鏡像

我們在實施的時候一般會創建/data目錄，把所有的deployment放在此目錄下，因此在k8s master創建kube-system目錄

[root@master data]# mkdir kube-system

上傳相鏡像，並導入

# 導入heasper
[root@master kube-system]# docker load < heapster_3.tar 
38ac8d0f5bb3: Loading layer [==================================================>]  1.312MB/1.312MB
388f58c4d5b0: Loading layer [==================================================>]  99.87MB/99.87MB
c6772246bc46: Loading layer [==================================================>]  281.1kB/281.1kB
Loaded image: registry.cn-hangzhou.aliyuncs.com/lczean/heapster-amd64-v1.3.0-beta.1:v1.3.0-beta.1
# 導入influxdb數據庫
[root@master kube-system]# docker load < influxdb13.tar 
7da815924651: Loading layer [==================================================>]  10.48MB/10.48MB
2d447b9e914f: Loading layer [==================================================>]   5.12kB/5.12kB
Loaded image: registry.cn-hangzhou.aliyuncs.com/golden/heapster-influxdb-amd64:latest

查看導入images

[root@master kube-system]# docker images |grep heapster
registry.cn-hangzhou.aliyuncs.com/lczean/heapster-amd64-v1.3.0-beta.1   v1.3.0-beta.1       6393b81e2220        17 months ago       101MB
registry.cn-hangzhou.aliyuncs.com/golden/heapster-influxdb-amd64        latest              d3fccbedd180        22 months ago       11.6MB

修改images tag以便我們可以導入到私有registry中

[root@master kube-system]# docker tag registry.cn-hangzhou.aliyuncs.com/lczean/heapster-amd64-v1.3.0-beta.1:v1.3.0-beta.1  registry.k8s.osc:5000/heapster:v1.3.0
[root@master kube-system]# docker tag registry.cn-hangzhou.aliyuncs.com/golden/heapster-influxdb-amd64 registry.k8s.osc:5000/heapster-influxdb
# 查看修改后的images
[root@master kube-system]# docker images |grep heapster
registry.cn-hangzhou.aliyuncs.com/lczean/heapster-amd64-v1.3.0-beta.1   v1.3.0-beta.1       6393b81e2220        17 months ago       101MB
registry.k8s.osc:5000/heapster                                          v1.3.0              6393b81e2220        17 months ago       101MB
registry.cn-hangzhou.aliyuncs.com/golden/heapster-influxdb-amd64        latest              d3fccbedd180        22 months ago       11.6MB
registry.k8s.osc:5000/heapster-influxdb                                 latest              d3fccbedd180        22 months ago       11.6MB

推送到私有倉庫

[root@master kube-system]# docker push registry.k8s.osc:5000/heapster:v1.3.0
The push refers to repository [registry.k8s.osc:5000/heapster]
c6772246bc46: Pushed 
388f58c4d5b0: Pushed 
38ac8d0f5bb3: Pushed 
v1.3.0: digest: sha256:e23b30d2e131e042eec9b5fdc30af905b63e454d140dc335246e74a4e8b4c857 size: 949
[root@master kube-system]# docker push registry.k8s.osc:5000/heapster-influxdb 
The push refers to repository [registry.k8s.osc:5000/heapster-influxdb]
2d447b9e914f: Pushed 
7da815924651: Pushed 
38ac8d0f5bb3: Mounted from heapster 
latest: digest: sha256:d2ecd285eb6585d56e8853da7b9fd8f4a57de4a3006f6720173a3f3942c0e7c9 size: 945

influxdb時間序列庫介紹

創建deployment

[root@master kube-system]# vim influxdb-deployment.yaml
[root@master kube-system]# vim influxdb-service.yaml
[root@master kube-system]# vim heapster-deployment.yaml
[root@master kube-system]# vim heapster-service.yaml   

分別看一下yaml：
influxdb-deployment.yaml
修改image

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: monitoring-influxdb
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: influxdb
    spec:
      volumes:
      - name: influxdb-storage
        emptyDir: {}
      containers:
      - name: influxdb
        image: registry.k8s.osc:5000/heapster-influxdb
        volumeMounts:
        - mountPath: /data
          name: influxdb-storage

influxdb-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: monitoring-influxdb
  name: monitoring-influxdb
  namespace: kube-system
spec:
  ports:
  - name: http
    port: 8083
    targetPort: 8083
  - name: api
    port: 8086
    targetPort: 8086
  selector:
    k8s-app: influxdb

創建deployment、service

[root@master kube-system]# kubectl create -f influxdb-deployment.yaml 
[root@master kube-system]# kubectl create -f influxdb-service.yaml 

安裝這兩個后查看influxdb坐在的pod ip

[root@master kube-system]# kubectl get pods  -n kube-system -o wide
NAME                                   READY     STATUS    RESTARTS   AGE       IP            NODE
monitoring-influxdb-3696415694-q9tds   1/1       Running   0          16m       172.99.39.6   172.16.187.158

測試安裝正常，再安裝flanneld的node訪問以下鏈接，如果無報錯說明安裝成功

[root@node0 ~]# curl http://172.99.39.6:8086/ping

創建heapster-deployment.yaml
修改image、--source、--sink

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
        version: v6
    spec:
      containers:
      - name: heapster
        image: registry.k8s.osc:5000/heapster:v1.3.0
        imagePullPolicy: Always
        command:
        - /heapster
        - --source=kubernetes:http://172.16.187.162:8080
        - --sink=influxdb:http://172.99.39.6:8086

創建heapster-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring  
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8082
  selector:
    k8s-app: heapster

創建heapster的deployment、service

[root@master kube-system]# kubectl create -f heapster-deployment.yaml 
deployment "heapster" created
[root@master kube-system]# kubectl create -f heapster-service.yaml 
service "heapster" created

全部安裝后可以查看日志是否是正常啟動的

[root@master kube-system]# kubectl get pods  -n kube-system -o wide   
NAME                                   READY     STATUS    RESTARTS   AGE       IP             NODE
heapster-1258036176-sjg7s              1/1       Running   0          1m        172.99.93.13   172.16.187.160
monitoring-influxdb-3696415694-q9tds   1/1       Running   0          26m       172.99.39.6    172.16.187.158


[root@master kube-system]# kubectl logs -f monitoring-influxdb-3696415694-q9tds  -n kube-system                         

 8888888           .d888 888                   8888888b.  888888b.
   888            d88P"  888                   888  "Y88b 888  "88b
   888            888    888                   888    888 888  .88P
   888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
   888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
   888   888  888 888    888 888  888   X88K   888    888 888    888
   888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
 8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"

[run] 2018/12/07 05:27:33 InfluxDB starting, version unknown, branch unknown, commit unknown
[run] 2018/12/07 05:27:33 Go version go1.7.4, GOMAXPROCS set to 16
[run] 2018/12/07 05:27:33 Using configuration at: /etc/config.toml
[store] 2018/12/07 05:27:33 Using data dir: /data/data
[subscriber] 2018/12/07 05:27:33 opened service
[monitor] 2018/12/07 05:27:33 Starting monitor system
[monitor] 2018/12/07 05:27:33 'build' registered for diagnostics monitoring
[monitor] 2018/12/07 05:27:33 'runtime' registered for diagnostics monitoring
[monitor] 2018/12/07 05:27:33 'network' registered for diagnostics monitoring
[monitor] 2018/12/07 05:27:33 'system' registered for diagnostics monitoring
[shard-precreation] 2018/12/07 05:27:33 Starting precreation service with check interval of 10m0s, advance period of 30m0s
[snapshot] 2018/12/07 05:27:33 Starting snapshot service
[continuous_querier] 2018/12/07 05:27:33 Starting continuous query service
[httpd] 2018/12/07 05:27:33 Starting HTTP service
[httpd] 2018/12/07 05:27:33 Authentication enabled: false

## heapster

[root@master kube-system]# kubectl logs -f heapster-1258036176-sjg7s  -n kube-system
I1207 05:53:00.275512       1 heapster.go:71] /heapster --source=kubernetes:http://172.16.187.162:8080 --sink=influxdb:http://172.99.39.6:8086
I1207 05:53:00.275568       1 heapster.go:72] Heapster version v1.3.0-beta.1
I1207 05:53:00.275794       1 configs.go:61] Using Kubernetes client with master "http://172.16.187.162:8080" and version v1
I1207 05:53:00.275816       1 configs.go:62] Using kubelet port 10255
I1207 05:53:00.283647       1 influxdb.go:252] created influxdb sink with options: host:172.99.39.6:8086 user:root db:k8s
I1207 05:53:00.283680       1 heapster.go:193] Starting with InfluxDB Sink
I1207 05:53:00.283687       1 heapster.go:193] Starting with Metric Sink
I1207 05:53:00.294214       1 heapster.go:105] Starting heapster on port 8082
I1207 05:54:05.082812       1 influxdb.go:215] Created database "k8s" on influxDB server at "172.99.39.6:8086"

最后查看heapster，由於收集數據需要時間，過一段時間后，查看節點的node的監控數據

[root@master ~]# kubectl top node
NAME             CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
172.16.187.158   121m         0%        19721Mi         30%       
172.16.187.159   112m         0%        15805Mi         24%       
172.16.187.160   172m         1%        28090Mi         43%       

創建HPA

1創建HPA控制器自動管控Pod副本
 kubectl autoscale deploy ms-wechat   --min=3 --max=10 --cpu-percent=80

以上步驟都成功的時候，我們可以創建HorizontalPodAutoscaler來管理，下面就用ms-wechat來進行測試

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: ms-wechat  # 名稱
  namespace: default #k8s命名空間
spec:
  maxReplicas: 10  # 最大副本數
  minReplicas: 3   # 最小副本數
  scaleTargetRef:   
    apiVersion: apps/v1beta1
    kind: Deployment  
    name: ms-wechat   # 監控名為ms-wechat的Deployment
  targetCPUUtilizationPercentage: 80  # cpu 閾值


       （1）minReplicas：自動伸縮pod副本數下限

       （2）maxReplicas：自動伸縮可擴展至Pod副本數的上限

       （3）scaleTargetRef：要伸縮的目標資源

       （4）metrics：用於計算所需的Pod副本數量的指標列表

       （5）external：用於應用非附屬於任何對象的全局指標

       （6）object：應用描述集群中某單一對象的特定指標

       （7）pods：應用被彈性伸縮的pod對象的特定指標

       （8）resource：應用資源指標，即當前被彈性伸縮的pod對象中容器的requests和limits中定義的指標。

       （9）type：標識指標源的類型

查看hpa

[root@master ~]# kubectl get hpa 
NAME        REFERENCE              TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
ms-wechat   Deployment/ms-wechat   <unknown> / 80%   3         10        3          10m

大家看到 targets為unknown有兩種原因

• 查看原始deployment的resource有沒有設置cpu的限制如果沒有：kubectl set resources deployment/ms-wechat --limits=cpu=2000m動態設置
• 等一段時間再查看

查看結果

[root@master ~]# kubectl get hpa
NAME        REFERENCE              TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
ms-wechat   Deployment/ms-wechat   47% / 80%   3         10        3          11m

可以進行壓力測試，觀察REPLICAS變化