k8s之自定義指標API部署prometheus

1.自定義指標-prometheus

node_exporter是agent;PromQL相當於sql語句來查詢數據;

k8s-prometheus-adapter:prometheus是不能直接解析k8s的指標的,需要借助k8s-prometheus-adapter轉換成api;

kube-state-metrics是用來整合數據的.

訪問:https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus

git clone https://github.com/iKubernetes/k8s-prom.git
cd k8s-prom && kubectl apply -f namespace.yaml
# 部署node_exporter
cd node_exporter/ && kubectl apply -f .
# 部署prometheus,注釋掉資源限制limit,
cd prometheus/ && vim prometheus-deploy.yaml && kubectl apply -f .
#resources:
#  limits:
#    memory: 200Mi
這個pod沒有部署好,prometheus就無法收集到數據,導致grafana界面沒有數據,浪費了一天時間
kubectl get pods -n prom
prometheus-server-64877844d4-gx4jr 1/1 Running 0 <invalid>

訪問NodePort,訪問prometheus

部署k8s-prometheus-adapter,需要自制證書

cd kube-state-metrics/ && kubectl apply -f .
cd /etc/kubernetes/pki/
(umask 077; openssl genrsa -out serving.key 2048)
openssl req -new -key serving.key -out serving.csr -subj "/CN=serving"
openssl  x509 -req -in serving.csr -CA ./ca.crt -CAkey ./ca.key -CAcreateserial -out serving.crt -days 3650
# custom-metrics-apiserver-deployment.yaml會用到secretName: cm-adapter-serving-certs
kubectl create secret generic cm-adapter-serving-certs --from-file=serving.crt=./serving.crt --from-file=serving.key=./serving.key  -n prom

# 部署k8s-prometheus-adapter,由於版本問題,需要下載兩個文件,將兩個文件中的名稱空間改為prom
cd k8s-prometheus-adapter/
mv custom-metrics-apiserver-deployment.yaml ..
wget https://raw.githubusercontent.com/DirectXMan12/k8s-prometheus-adapter/master/deploy/manifests/custom-metrics-apiserver-deployment.yam
wget https://raw.githubusercontent.com/DirectXMan12/k8s-prometheus-adapter/master/deploy/manifests/custom-metrics-config-map.yaml
kubectl apply -f .

kubectl api-versions  # 必須出現這個api,並且開啟代理可以訪問到數據
custom.metrics.k8s.io/v1beta1
kubectl proxy --port=8080
curl http://localhost:8080/apis/custom.metrics.k8s.io/v1beta1/
# prometheus和grafana整合
wget https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/influxdb/grafana.yaml
把namespace: kube-system改成prom,有兩處;
把env里面的下面兩個注釋掉：
- name: INFLUXDB_HOST
 value: monitoring-influxdb
在最有一行加個type: NodePort
 ports:
  - port: 80
    targetPort: 3000
  selector:
    k8s-app: grafana
  type: NodePort
kubectl apply -f grafana.yaml
kubectl get svc -n prom
monitoring-grafana  NodePort    10.96.228.0      <none>  80:30336/TCP     13h

prom名稱空間內的所有pod

訪問:10.0.0.20:30336

兩個k8s模板:https://grafana.com/dashboards/6417 https://grafana.com/dashboards/315

一切順利的話,立馬就能看到監控數據

2.HPA(水平pod自動擴展)

當pod壓力大了,會根據負載自動擴展Pod個數以緩解壓力

kubectl api-versions |grep auto
創建一個帶有資源限制的pod
kubectl run myapp --image=ikubernetes/myapp:v1 --replicas=1 \
--requests='cpu=50m,memory=256Mi' --limits='cpu=50m,memory=256Mi' \
--labels='app=myapp' --expose --port=80
# 讓myapp這個控制器支持自動擴展,--cpu-percent表示cpu超過這個值就開始擴展
kubectl autoscale deployment myapp --min=1 --max=5 --cpu-percent=60
kubectl get hpa
# 對pod進行壓力測試
kubectl patch svc myapp -p '{"spec":{"type": "NodePort"}}'
yum install httpd-tools
# 隨着cpu壓力的上升,會看到自動擴展為4個或更多的pod
ab -c 1000 -n 5000000 http://172.16.1.100:31990/index.html
# hpa v1版本只能根據cpu利用率擴展pod,hpa v2可以根據自定義指標利用率水平擴展pod
kubectl delete hpa myapp

cat hpa-v2-demo.yaml 
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa-v2
spec:
  scaleTargetRef: # 根據什么指標來做評估壓力
    apiVersion: apps/v1 
    kind: Deployment
    name: myapp  # 對哪個控制器做自動擴展
  minReplicas: 1
  maxReplicas: 10
  metrics: # 依據哪些指標來進行評估
  - type: Resource # 基於資源進行評估
    resource: 
      name: cpu
      targetAverageUtilization: 55 # cpu使用率超過55%,就自動水平擴展pod個數
  - type: Resource
    resource:
      name: memory  # v2版可以根據內存進行評估
      targetAverageValue: 50Mi # 內存使用超過50M,就自動水平擴展pod個數
kubectl apply -f hpa-v2-demo.yaml
# 進行壓測即可看到pod會自動擴展
# 自定義的資源指標,pod被開發好之后,得支持這些指標,否則就是白寫
# 下面這個例子中支持並發參數的鏡像地址:https://hub.docker.com/r/ikubernetes/metrics-app/
cat hpa-v2-custom.yaml 
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa-v2
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods  # 利用pod中定義的指標進行擴縮
    pods: 
      metricName: http_requests  # 自定義的資源指標
        targetAverageValue: 800m # m表示個數,並發數800

 

參考博客:http://blog.itpub.net/28916011/viewspace-2216340/

prometheus監控mysql、k8s:https://www.cnblogs.com/sfnz/p/6566951.html