Prometheus 監控 Nginx

適用於Nginx的Prometheus指標庫

使用的是 nginx-lua-prometheus 這個庫，負責去采集 nginx 內部的指標，暴露給 Prometheus 拉取。

安裝

要使用這個庫，需要啟用 Nginx 對於 lua 的支持，看了 Nginx 編譯 lua 很麻煩，於是直接換 openresty

wget -O /etc/yum.repos.d/openresty.repo  https://openresty.org/package/centos/openresty.repo
yum check-update
yum install -y openresty
# 裝好了。感覺比 Nginx 簡單，重點是直接支持 lua了

配置

在 nginx.conf 的 http 塊添加如下配置

# lua
lua_shared_dict prometheus_metrics 10M;
# 注意這個文件
lua_package_path "/usr/local/nginx/conf/prometheus.lua";
init_by_lua '
  prometheus = require("prometheus").init("prometheus_metrics")
  metric_requests = prometheus:counter(
    "nginx_http_requests_total", "Number of HTTP requests", {"host", "status"})
  metric_latency = prometheus:histogram(
    "nginx_http_request_duration_seconds", "HTTP request latency", {"host"})
  metric_connections = prometheus:gauge(
    "nginx_http_connections", "Number of HTTP connections", {"state"})
';
log_by_lua '
  metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
  metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
';

/usr/local/nginx/conf/prometheus.lua 這個文件內容太長了，請自行從 github 上下載保存。

https://github.com/knyar/nginx-lua-prometheus/blob/master/prometheus.lua

新增一個用於暴露指標的服務器配置文件

server {
  listen 9145;
  server_name localhost;
  location /metrics {
    content_by_lua '
      metric_connections:set(ngx.var.connections_active, {"active"})
      metric_connections:set(ngx.var.connections_reading, {"reading"})
      metric_connections:set(ngx.var.connections_waiting, {"waiting"})
      metric_connections:set(ngx.var.connections_writing, {"writing"})
      prometheus:collect()
    ';
  }
}

reload 一下 Nginx，就可以去訪問指標了

[root@iZ1rp1vunvZ vhost]# curl http://127.0.0.1:9145/metrics
# HELP nginx_http_connections Number of HTTP connections
# TYPE nginx_http_connections gauge
nginx_http_connections{state="active"} 879
nginx_http_connections{state="reading"} 0
nginx_http_connections{state="waiting"} 851
nginx_http_connections{state="writing"} 25
......
# HELP nginx_http_requests_total Number of HTTP requests
# TYPE nginx_http_requests_total counter
nginx_http_requests_total{host="",status="302"} 39
nginx_http_requests_total{host="",status="400"} 48
nginx_http_requests_total{host="",status="404"} 4

Prometheus 配置文件 Nginx 的配置

  - job_name: 'Nginx'
    static_configs:
      - targets: ['192.168.201.179:9145','192.168.201.180:9145']

告警規則

4xx 過多

  - alert: NginxHighHttp4xxErrorRate
    expr: sum(rate(nginx_http_requests_total{status=~"^4.."}[1m])) / sum(rate(nginx_http_requests_total[1m])) * 100 > 5
    for: 5m
    labels:
      severity: error
    annotations:
      summary: "Nginx high HTTP 4xx error rate (instance {{ $labels.instance }})"
      description: "Too many HTTP requests with status 4xx (> 5%)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

5xx 過多

  - alert: NginxHighHttp5xxErrorRate
    expr: sum(rate(nginx_http_requests_total{status=~"^5.."}[1m])) / sum(rate(nginx_http_requests_total[1m])) * 100 > 5
    for: 5m
    labels:
      severity: error
    annotations:
      summary: "Nginx high HTTP 5xx error rate (instance {{ $labels.instance }})"
      description: "Too many HTTP requests with status 5xx (> 5%)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"