prometheus + alertmanager 實現報警

1. 部署alertmanager

• 鏡像 prom/alertmanager
• 配置文件在 /etc/alertmanager/alertmanager.yml

alertmanager.yml: |
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 1h
      receiver: 'default'
    receivers:
      - name: "default"
        webhook_configs:
        - url: "http://192.168.0.103:5002/api/hooks"
          send_resolved: false

• 由於沒有企業微信、也懶得配置郵件，就先用webhook測試。
• group_by 表示分組，alertname 是 prometheus 中配置的 rule 的名稱，如下：memory_too_high、cpu_too_high、disk_will_full

2. 配置prometheus

prometheus.yml: |
    global:
      scrape_interval: 30s
      scrape_timeout: 30s
    alerting:
      alertmanagers:
      - static_configs:
        - targets: ["prometheus-alertmanager-svc:9093"]
    rule_files:
      - "alertmanager_rules.yml"
    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
        - targets: ['localhost:9090']
    - job_name: 'k8s-node'
      static_configs:
        - targets: 
          - 192.168.0.200:9100
          - 192.168.0.201:9100

• alerting 節點配置 alertmanager的地址
• rule_files 節點配置 告警的 rule 文件，比如監控cpu、內存、硬盤的監控

alertmanager_rules.yml: |
    groups:
    - name: node-alert
      rules:
      - alert: node-up
        expr: up{job="node-exporter"} == 0
        for: 15s
        labels:
            severity: 1
            team: node
        annotations:
            summary: "{{ $labels.instance }} 已停止運行超過 15s！"
      - alert: memory_too_high
        expr: (1 - (node_memory_MemAvailable_bytes / (node_memory_MemTotal_bytes)))* 100 > 90
        for: 10s  # 告警持續時間，超過這個時間才會發送給alertmanager
        labels:
          severity: warning
        annotations:
          summary: "Instance {{ $labels.instance }} 內存使用率過高"
          description: "{{ $labels.instance }} of job {{$labels.job}}內存使用率超過90%,當前使用率[{{ $value }}]."
      - alert: cpu_too_high
        expr: 100-avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by(instance)*100 > 80
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Instance {{ $labels.instance }} cpu使用率過高"
          description: "{{ $labels.instance }} of job {{$labels.job}}cpu使用率超過80%,當前使用率[{{ $value }}]."
      - alert: disk_will_full
        expr: 100 - node_filesystem_avail_bytes{fstype=~"ext4|xfs",mountpoint="/"}  * 100 / node_filesystem_size_bytes{fstype=~"ext4|xfs",mountpoint="/"} > 80
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Instance {{ $labels.instance }} disk使用率過高"
          description: "{{ $labels.instance }} of job {{$labels.job}}disk使用率超過80%,當前使用率[{{ $value }}]."

• $labels 可獲得數據的標簽
• $value 可獲得[expr]計算的值