Prometheus部署+郵箱告警+企業微信告警+釘釘告警

Prometheus部署+郵箱報警+企業微信報警+釘釘報警

1 部署Prometheus server

1.1 下載二進制包

$ wget https://github.com/prometheus/prometheus/releases/download/v2.12.0/prometheus-2.12.0.linux-amd64.tar.gz

1.2 解壓並move至/work/admin目錄下

$ tar zcvf prometheus-2.12.0.linux-amd64.tar.gz

$ mv prometheus-2.12.0.linux-amd64 /work/admin/prometheus

1.3 配置並啟動

$ cat prometheus.yml
# my global config
global:
  scrape_interval:     15s # 默認抓取間隔, 15秒向目標抓取一次數據。
  scrape_timeout: 15s
  evaluation_interval: 20s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
       #- alertmanager: ['localhost:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "/work/admin/prometheus/alerts/*.rules"
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus_product'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    #metrics_path: /metrics
    #scheme: http

    static_configs:
    - targets: ['localhost:9090','localhost:9100']
      labels: {cluster: 'product',type: 'basic',env: 'prometheus',job: 'prometheus',export: 'prometheus'}

$ /work/admin/prometheus/prometheus --config.file=/work/admin/prometheus/prometheus.yml --storage.tsdb.path=/work/admin/prometheus/data

2 部署node_exporter

2.1 下載二進制包

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz

2.2 解壓並move至/work/admin目錄下

$ tar zcvf node_exporter-0.17.0.linux-amd64.tar.gz

$ mv node_exporter-0.17.0.linux-amd64 /work/admin/node_exporter

2.3 啟動

$ /work/admin/node_exporter/node_exporter --web.listen-address=:9100

3 部署alertmanager

3.1 下載二進制包

$ wget https://github.com/prometheus/alertmanager/releases/download/v0.18.0/alertmanager-0.18.0.linux-amd64.tar.gz

3.2 解壓並move至/work/admin目錄下

$ tar zcvf alertmanager-0.18.0.linux-amd64.tar.gz

$ mv alertmanager-0.18.0.linux-amd64 /work/admin/alertmanager

3.3 修改配置文件並啟動

$ cat alertmanager.yml

global:

  resolve_timeout: 5m

  smtp_smarthost: 'smtp.163.com:25' # 郵箱smtp服務器代理

  smtp_from: '[e-mail]' # 發送郵箱名稱

  smtp_auth_username: '[e-mail]' # 郵箱名稱

  smtp_auth_password: 'XXXXXXXX' # 郵箱密碼或授權碼

templates:

  - 'template/*.tmpl'

route:

  group_by: ['alertname']

  group_wait: 10s

  group_interval: 10s

  repeat_interval: 24h

  receiver: 'ops_dingding'

receivers:

  - name: 'email'

    email_configs:

    - to: '[e-mail]'  # 接收警報的email配置

      html: '{{ template "test.html" . }}' # 設定郵箱的內容模板

      headers: { Subject: "[WARN] 報警郵件"} # 接收郵件的標題

  - name: 'wechat'

    wechat_configs:

    - corp_id: 'XXXXX'

      to_party: '1'

      agent_id: '1000002'

      api_secret: 'XXXXX'

  - name: 'ops_dingding'

    webhook_configs:

    - url: 'http://localhost:8060/dingtalk/ops_dingding/send'

inhibit_rules:

  - source_match:

      severity: 'critical'

    target_match:

      severity: 'warning'

    equal: ['alertname', 'dev', 'instance']

$ /work/admin/alertmanager/alertmanager --config.file=/work/admin/alertmanager/alertmanager.yml

4 prometheus通過webhook推送告警至釘釘

4.1 添加釘釘機器人，獲取webhook

參考 https://open-doc.dingtalk.com/docs/doc.htm?treeId=257&articleId=105735&docType=1

4.2 下載插件（二進制文件）

$ wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v0.3.0/prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz

4.3 解壓並move至/usr/local/prometheus目錄下

$ tar zxvf prometheus-webhook-dingtalk-0.3.0.linux-amd64.tar.gz

$ mv prometheus-webhook-dingtalk-0.3.0.linux-amd64/prometheus-webhook-dingtalk /work/admin/alertmanager

4.4 編輯啟動腳本（請替換為自己的webhook URL 及 ding.profile）

$ cat dingding_start.sh

nohup /work/admin/alertmanager/prometheus-webhook-dingtalk --ding.profile="ops_dingding=https://oapi.dingtalk.com/robot/send?access_token=XXXXXXX"  2>&1 1>/work/admin/alertmanager/dingding.log &

$ sh dingding_start.sh

4.5 編輯alertmanager.yml，增加web_hook配置並重啟alertmanager

  - name: 'ops_dingding'

    webhook_configs:

    - url: 'http://localhost:8060/dingtalk/ops_dingding/send'