HPA（Horizontal Pod Autoscaler）在k8s集群中用於POD水平自動伸縮，它是基於CPU和內存利用率對Deployment和Replicaset控制器中的pod數量進行自動擴縮容（除了CPU和內存利用率之外，也可以基於其他應程序提供的度量指標custom metrics進行自動擴縮容）。pod自動縮放不適用於無法縮放的對象，比如DaemonSets。HPA由Kubernetes API資源和控制器實現。資源決定了控制器的行為，控制器會周期性的獲取CPU和內存利用率，並與目標值相比較后來調整replication controller或deployment中的副本數量。
HPA使用前提條件：
- 集群中部署了Metrics Server插件, 可以獲取到Pod的CPU和內存利用率。
- Pod部署的Yaml文件中必須設置資源限制和資源請求。
以下對K8S集群使用HPA進行Pod自動伸縮做個測試記錄
1. 編譯一個測試容器
[root@k8s-master01 work]# cat kevin-t.yaml
apiVersion: v1
kind: Service
metadata:
name: kevin-t
namespace: default
labels:
app: kevin-t
spec:
type: NodePort
selector:
app: kevin-t
ports:
- name: http
port: 8088
targetPort: 80
nodePort: 30888
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kevin-t
namespace: default
spec:
replicas: 1
minReadySeconds: 10
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
selector:
matchLabels:
app: kevin-t
template:
metadata:
labels:
app: kevin-t
spec:
containers:
- name: kevin-t
image: nginx:1.7.9
imagePullPolicy: IfNotPresent
ports:
- name: kport
containerPort: 80
resources:
requests:
cpu: 100m
memory: 50Mi
limits:
cpu: 100m
memory: 50Mi
lifecycle:
postStart:
exec:
command: ["/bin/sh","-c","touch /tmp/health"]
livenessProbe:
exec:
command: ["test","-e","/tmp/health"]
initialDelaySeconds: 5
timeoutSeconds: 3
periodSeconds: 5
readinessProbe:
tcpSocket:
port: kport
initialDelaySeconds: 5
timeoutSeconds: 3
periodSeconds: 5
執行創建
[root@k8s-master01 work]# kubectl apply -f kevin-t.yaml
查看pod
[root@k8s-master01 work]# kubectl get pods|grep kevin-t
kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 119s
[root@k8s-master01 work]# kubectl get svc|grep kevin-t
kevin-t NodePort 10.254.120.80 <none> 8088:30888/TCP 4m20s
訪問
[root@k8s-master01 work]# curl http://172.16.60.235:30888
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
查看metrcis-service采集到的該pod的cpu和內存數據
[root@k8s-master01 work]# kubectl top pods
NAME CPU(cores) MEMORY(bytes)
dnsutils-ds-22q87 0m 0Mi
dnsutils-ds-jcvng 0m 0Mi
dnsutils-ds-zjg5l 0m 0Mi
kevin-t-66cdc9cdbf-jwvnv 1m 3Mi
my-nginx-765d6dcff7-gs97l 0m 3Mi
my-nginx-765d6dcff7-lbjm6 0m 2Mi
nginx-ds-98rm2 0m 2Mi
nginx-ds-bbx68 0m 2Mi
nginx-ds-ngqcm 0m 2Mi
通過上面可知，使用kevin-t的deployment控制的kevin-t的pod容器資源請求和資源限制都是100m的CPU（即0.1核）和50M的內存。當前該pod所用CPU是1m，內存是3M
2. 創建HPA限制
設置kevin-t的 deployment的最大最小副本數，HPA對應pod的CPU和內存指標做限制。
參數解釋：
scaleTargetRef: 要縮放的目標資源
metrics: 計算所需Pod副本數量的指標列表，每個指標單獨計算，取所有計算結果的最大值作為最終副本數量
- external 引用非附屬於任何對象的全局指標，可以是集群之外的組件的指標數據，如消息隊列長度
- object 引用描述集群中某單一對象的特定指標，如Ingress對象上的hits-per-second等
- pods 引用當前被彈性伸縮的Pod對象的特定指標
- resource 引用資源指標，即當前被彈性伸縮的Pod對象中容器的requests和limits中定義的指標
- type 指標源的類型，可為Objects、Pods、Resource
另外注意兩個指標閾值參數：
- targetAverageUtilization 表示的是百分比
- targetAverageValue 表示的是數值，比如100m的CPU、100Mi的內存
[root@k8s-master01 work]# cat kevin-t-hap.yml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: kevin-t-hap
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: kevin-t
minReplicas: 1
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80
- type: Resource
resource:
name: memory
targetAverageValue: 30Mi
如上，設置了kevin-t的deployment控制的pod的HPA限制，當cpu使用超過設置的80%，內存使用超過30Mi時就觸發自動擴容，副本數最小為1，最大為3。
查看創建的HPA
[root@k8s-master01 work]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
kevin-t-hap Deployment/kevin-t 3194880/30Mi, 1%/80% 1 3 3 2m15s
[root@k8s-master01 work]# kubectl describe hpa kevin-t-hap
Name: kevin-t-hap
Namespace: default
Labels: <none>
Annotations: CreationTimestamp: Fri, 14 Aug 2020 17:00:01 +0800
Reference: Deployment/kevin-t
Metrics: ( current / target )
resource memory on pods: 3291818666m / 30Mi
resource cpu on pods (as a percentage of request): 1% (1m) / 80%
Min replicas: 1
Max replicas: 3
Deployment pods: 3 current / 3 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource
ScalingLimited True TooManyReplicas the desired replica count is more than the maximum replica count
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 4m3s horizontal-pod-autoscaler New size: 3; reaso
3. 壓力測試，驗證HPA的擴縮容
使用ab命令進行壓力測試，主要有兩個參數：
-n ：總共的請求執行數，缺省是1；
-c： 並發數，缺省是1；
測試之前kevin-t的pod是一個副本數 [root@k8s-master01 work]# kubectl get pods|grep kevin-t kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 119s 接下來進行壓力測試 [root@k8s-master01 work]# ab -c 1000 -n 500000000000 http://172.16.60.235:30888/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 172.16.60.235 (be patient) apr_socket_recv: Connection reset by peer (104) Total of 12623 requests completed 發現此時，kevin-t 已經擴到2個副本了 [root@k8s-master02 ~]# kubectl get pods|grep kevin-t kevin-t-66cdc9cdbf-5h564 1/1 Running 0 38s kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 35m 再加大壓力測試： [root@k8s-master01 work]# ab -c 100 -n 50000000000000000 http://172.16.60.235:30888/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 172.16.60.235 (be patient) 在上面壓力測試中，另外打開一個終端，發現kevin-t已經擴容到最大的3個副本了！ [root@k8s-master02 ~]# kubectl get pods|grep kevin-t kevin-t-66cdc9cdbf-5h564 1/1 Running 0 4m52s kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 39m kevin-t-66cdc9cdbf-zpbfs 1/1 Running 0 80s 查看hpa，實時查看pod的cpu和內存狀態 [root@k8s-master01 work]# kubectl get hpa -w NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE kevin-t-hap Deployment/kevin-t 4434602666m/30Mi, 1%/80% 1 3 3 19m kevin-t-hap Deployment/kevin-t 4405930666m/30Mi, 1%/80% 1 3 3 19m kevin-t-hap Deployment/kevin-t 4441429333m/30Mi, 1%/80% 1 3 3 20m kevin-t-hap Deployment/kevin-t 4441429333m/30Mi, 1%/80% 1 3 3 20m kevin-t-hap Deployment/kevin-t 4945920/30Mi, 1%/80% 1 3 2 20m kevin-t-hap Deployment/kevin-t 4945920/30Mi, 1%/80% 1 3 2 21m ............... 觀察kevin-t的pod情況 [root@k8s-master02 ~]# kubectl get pods|grep kevin-t -w kevin-t-66cdc9cdbf-5h564 1/1 Running 0 11m kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 46m kevin-t-66cdc9cdbf-zpbfs 1/1 Running 0 7m36s [root@k8s-master02 ~]# kubectl get pods|grep kevin-t -w kevin-t-66cdc9cdbf-5h564 1/1 Terminating 0 12m kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 47m 發現壓力測試過一段時間后，隨着pod的cpu和內存使用數值下降，副本數也在逐步減少到最小的1個副本 [root@k8s-master02 ~]# kubectl get pods|grep kevin-t kevin-t-66cdc9cdbf-jwvnv 1/1 Running 0 49m