prometheus（九） 收集kube-state-metrics指標信息

安裝kube-state-metrics

1. 下載

   $ cd /opt/k8s/work/
   $ wget -O kube-state-metrics.zip  https://github.com/kubernetes/kube-state-metrics/archive/master.zip
   $ unzip kube-state-metrics.tar.gz
   $ cd kube-state-metrics-master/examples/standard
   $ ls
   cluster-role-binding.yaml  cluster-role.yaml  deployment.yaml  service-account.yaml  service.yaml
   

2. 修改namespace

   將默認的namespace從kube-system修改成monitoring

   $ cd /opt/k8s/work/kube-state-metrics-master/examples/standard
   $ ls | xargs -I {} sed -i "s/namespace: kube-system/namespace: monitoring/g" {}
   

3. 在service.yaml追加如下annotation

   annotations:
     prometheus.io/scraped: "true"
     
   

   修改后的server.yml文件

   cat service.yaml
    apiVersion: v1
    kind: Service
    metadata:
      annotations:
        prometheus.io/scraped: "true"
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: 1.9.5
      name: kube-state-metrics
      namespace: monitoring
    spec:
      clusterIP: None
      ports:
      - name: http-metrics
        port: 8080
        targetPort: http-metrics
      - name: telemetry
        port: 8081
        targetPort: telemetry
      selector:
        app.kubernetes.io/name: kube-state-metrics
    
   

4. 啟動服務

   $ cd /opt/k8s/work/kube-state-metrics-master/examples/standard
   $ kubectl create -f .
   
   

prometheus 收集指標

1. prometheus 追加配置，僅抓取帶有prometheus.io/scraped: "true" 注解的endpoint

   - job_name: "kubernetes-service-endpoints"
     kubernetes_sd_configs:
     - role: endpoints
     relabel_configs:
     - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scraped]
       action: keep
       regex: true
     - action: labelmap
       regex: __meta_kubernetes_service_label_(.+)
     - source_labels: [__meta_kubernetes_namespace]
       action: replace
       target_label: kubernetes_namespace
     - source_labels: [__meta_kubernetes_service_name]
       action: replace
       target_label: service_name
    
   

2. prometheus重新加載配置后，查看對應的target

   

   kube-state-metrics對應的pod有兩個端口，在prometheus中顯示了兩個target

配置grafana

1. 從grafana.com 下載state-metrics監控模版1. Kubernetes Deployment Statefulset Daemonset metrics,導入模版

   grafana效果圖