Prometheus + Consul 自動發現服務監控

一、Prometheus支持的多種服務發現機制（常用如下）

1. static_configs: 靜態服務發現
2. file_sd_configs: 文件服務發現
3. dns_sd_configs: DNS 服務發現
4. kubernetes_sd_configs: Kubernetes 服務發現
5. consul_sd_configs: Consul 服務發現

二、Prometheus主要配置prometheus.yml中的scrape_configs以及consul_sd_configs如下

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'consul'
    consul_sd_configs:
      - server: 'localhost:8500'
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*,prome,.*
        action: keep
      - source_labels: [__meta_consul_service]
        target_label: job

通過重新加載prometheus.yml配置添加

./prometheus --config.file="prometheus.yml" --web.listen-address="0.0.0.0:9090" --web.enable-admin-api --web.enable-lifecycle &

--web.enable-admin-api 運行通過web方式管理prometheus（刪除清空數據等操作）
--web.enable-lifecycle 運行通過web方式重新加載prometheus配置（相當於reload）

 

三、k8s環境實現

將配置制作成secret掛在到prometheus中

1、編寫prometheus-additional-consul.yaml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'consul'
    consul_sd_configs:
      - server: 'localhost:8500'
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*,prome,.*
        action: keep
      - source_labels: [__meta_consul_service]
        target_label: job

2、制作secret

kubectl create secret generic additional-consul-configs --from-file=prometheus-additional-consul.yaml -n monitoring

3、添加配置到prometheus的pod中，修改prometheus-prometheus.yaml，並更新資源

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.11.0
  additionalScrapeConfigs:
    name: additional-consul-configs
    key: prometheus-additional-consul.yaml

4、注意事項：權限不夠（配置更新，但是沒有對應的監控任務生成）

修改名為prometheus-k8s的ClusterRole權限，並更新資源

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get