VictoriaMetrics vmalert 說明

vmalert 可以執行一系列給定的rule（基於metricsql），然后發送報警到Alertmanager

特性

• 集成VictoriaMetrics TSDB
• MetricsQL 表達式校驗
• prometheus 報警規則格式支持
• 集成Alertmanager
• 輕量級沒有額外的依賴

使用

• 構建
  目前需要自己構建，很簡單 make vmalert 就可以了
• 啟動
  依賴一系列的規則（promql&&metricssql），數據源地址，通知地址Alertmanager地址，方便處理，聚合報警以及發送通知
  命令:

 

./bin/vmalert -rule=alert.rules \

        -datasource.url=http://localhost:8428 \

        -notifier.url=http://localhost:9093

一個參考規則

groups:

  - name: groupGorSingleAlert

    rules:

      - alert: VMRows

        for: 10s

        expr: vm_rows > 0

        labels:

          label: bar

          host: "{{ $labels.instance }}"

        annotations:

          summary: "{{ $value|humanize }}"

          description: "{{$labels}}"

​

  - name: TestGroup

    rules:

      - alert: Conns

        expr: sum(vm_tcplistener_conns) by(instance) > 1

        annotations:

          summary: "Too high connection number for {{$labels.instance}}"

          description: "It is {{ $value }} connections for {{$labels.instance}}"

      - alert: ExampleAlertAlwaysFiring

        expr: sum by(job)

          (up == 1)

• 參考命令

vmalert-20200511-085826-heads-cluster-0-g6c88e352

Usage of ./vmalert:

  -datasource.basicAuth.password string

        Optional basic auth password for -datasource.url

  -datasource.basicAuth.username string

        Optional basic auth username for -datasource.url

  -datasource.url string

        Victoria Metrics or VMSelect url. Required parameter. E.g. http://127.0.0.1:8428

  -enableTCP6

        Whether to enable IPv6 for listening and dialing. By default only IPv4 TCP is used

  -envflag.enable

        Whether to enable reading flags from environment variables additionally to command line. Command line flag values have priority over values from environment vars. Flags are read only from command line if this flag isn't set

  -envflag.prefix string

        Prefix for environment variables if -envflag.enable is set

  -evaluationInterval duration

        How often to evaluate the rules. Default 1m (default 1m0s)

  -external.url string

        External URL is used as alert's source for sent alerts to the notifier

  -http.disableResponseCompression

        Disable compression of HTTP responses for saving CPU resources. By default compression is enabled to save network bandwidth

  -http.maxGracefulShutdownDuration duration

        The maximum duration for graceful shutdown of HTTP server. Highly loaded server may require increased value for graceful shutdown (default 7s)

  -http.pathPrefix string

        An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus

  -http.shutdownDelay duration

        Optional delay before http server shutdown. During this dealy the servier returns non-OK responses from /health page, so load balancers can route new requests to other servers

  -httpListenAddr string

        Address to listen for http connections (default ":8880")

  -loggerFormat string

        Format for logs. Possible values: default, json (default "default")

  -loggerLevel string

        Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO")

  -loggerOutput string

        Output for the logs. Supported values: stderr, stdout (default "stderr")

  -memory.allowedPercent float

        Allowed percent of system memory VictoriaMetrics caches may occupy. Too low value may increase cache miss rate, which usually results in higher CPU and disk IO usage. Too high value may evict too much data from OS page cache, which will result in higher disk IO usage (default 60)

  -notifier.url string

        Prometheus alertmanager URL. Required parameter. e.g. http://127.0.0.1:9093

  -remoteread.basicAuth.password string

        Optional basic auth password for -remoteread.url

  -remoteread.basicAuth.username string

        Optional basic auth username for -remoteread.url

  -remoteread.lookback duration

        Lookback defines how far to look into past for alerts timeseries. For example, if lookback=1h then range from now() to now()-1h will be scanned. (default 1h0m0s)

  -remoteread.url vmalert

        Optional URL to Victoria Metrics or VMSelect that will be used to restore alerts state. This configuration makes sense only if vmalert was configured with `remotewrite.url` before and has been successfully persisted its state. E.g. http://127.0.0.1:8428

  -remotewrite.basicAuth.password string

        Optional basic auth password for -remotewrite.url

  -remotewrite.basicAuth.username string

        Optional basic auth username for -remotewrite.url

  -remotewrite.url string

        Optional URL to Victoria Metrics or VMInsert where to persist alerts state in form of timeseries. E.g. http://127.0.0.1:8428

  -rule value

        Path to the file with alert rules. 

        Supports patterns. Flag can be specified multiple times. 

        Examples:

         -rule /path/to/file. Path to a single file with alerting rules

         -rule dir/*.yaml -rule /*.yaml. Relative path to all .yaml files in "dir" folder, 

       absolute path to all .yaml files in root.

 -rule.validateTemplates

       Indicates to validate annotation and label templates (default true)

 -version

       Show VictoriaMetrics version

vmalert 暴露的服務

http://<vmalert-addr>/api/v1/alerts - 所有激活的報警;

http://<vmalert-addr>/api/v1/<groupName>/<alertID>/status" - 根據id獲取報警狀態

http://<vmalert-addr>/metrics - 應用程序的metrics

http://<vmalert-addr>/-/reload - 熱加載（目前暫時還沒實現）

說明

vmalert 是一個很不錯的工具，彌補了VictoriaMetrics以前需要依賴外部進行報警的缺陷，以前我們需要基於原生prometheus
或者promxy或者grafana 等工具進行報警的處理，vmalert依托metrcisql 增強的擴展完善了VictoriaMetrics

參考資料

https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmalert