Prometheus搜集mysql和nginx log指標

• Prometheus+Grafana+Mysql-exporter+Nginx_logs-exporter
• 給予docker的方式部署運行
• 以下配置涉及的192.168.1.11為服務器IP，請根據實際而修改

一、部署 Prometheus+Grafana

Docker Compose For Prometheus+Grafana YML File

$ cat docker-compose-monitoring.yml

version: "2.3"
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    hostname: prometheus
    volumes:
      - ./prometheus/config:/etc/prometheus
      - ./prometheus/data:/prometheus
      - /etc/localtime:/etc/localtime
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--log.level=info'
      - '--web.listen-address=0.0.0.0:9090'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention=30d'
      - '--query.max-concurrency=50'
      - '--web.enable-lifecycle'
    ports:
      - "11090:9090"
    logging:
      driver: "json-file"
    restart: always
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    hostname: grafana
    restart: always
    volumes:
      - /data/work/monitoring_data/grafana/logs:/var/log/grafana
      - /data/work/monitoring_data/grafana/data:/var/lib/grafana
      - ./grafana/grafana.ini:/etc/grafana/grafana.ini
      - /etc/localtime:/etc/localtime
    ports:
      - "11091:3000"
    user: "104"
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

Grafana 配置文件

• 由於grafana.ini配置文件內容太多就不一一貼出來了，本文僅體現修改。
• grafana.ini文件可以從grafana容器內copy出來(先運行grafana的容器，然后拷貝配置文件：docker cp grafana:/etc/grafana/grafana.ini .)

$ mkdir grafana
$ vim grafana/grafana.ini
[paths]
data = /var/lib/grafana
logs = /var/log/grafana

Prometheus 配置文件

$ mkdir -p prometheus/config
$ cat prometheus/config/prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    monitor: 'Monitoring'

scrape_configs:

啟動&停止

docker-compose -f docker-compose-monitoring.yml up -d
docker-compose -f docker-compose-monitoring.yml down

• 瀏覽器打開Prometheus地址驗證http://Prometheus_IP:11090/
  

• 瀏覽器打開Grafana地址驗證http://Prometheus_IP:11091/，默認用戶名密碼為admin/admin
  

二、MySQL Exporter

$ cat docker-compose-mysql_exporter.yml
version: "2.3"
services:
  mysql_exporter:
    image: prom/mysqld-exporter:latest
    container_name: mysql_exporter
    hostname: mysql_exporter
    environment:
      DATA_SOURCE_NAME: "exporter:PWD123@(192.168.1.11:11009)/"
    ports:
      - "11092:9104"
    restart: always
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

登錄需要監控的mysql服務，創建監控用戶

$ mysql -uroot -p123456 -h 192.168.1.11 -P 11009

// 此處創建的監控用戶名為exporter 密碼為PWD123 根據實際情況進行修改.
> create user 'exporter'@'%';
> alter user 'exporter'@'%' identified with mysql_native_password by 'PWD123';
> GRANT REPLICATION CLIENT, PROCESS ON *.* TO 'exporter'@'%';
> GRANT SELECT ON performance_schema.* TO 'exporter'@'%';
> flush privileges;
> exit

根據MySQL的連接信息，修改compose相關的變量

      DATA_SOURCE_NAME: "exporter:PWD123@(192.168.1.11:11009)/"
#說明      DATA_SOURCE_NAME: "mysql用戶:mysql密碼@(mysql IP地址:mysql端口)/"

啟動mysql exporter

docker-compose -f docker-compose-mysql_exporter.yml up -d
docker-compose -f docker-compose-mysql_exporter.yml down

訪問

瀏覽器訪問mysql exporter的接口即可獲取相應的mysql metrics
http://服務器IP:11092/metrics

將MySQL Exporter添加到prometheus.yml文件

$ vim prometheus/config/prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    monitor: 'Monitoring'

scrape_configs:
# MySQL
  - job_name: 'MySQL'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.1.11:11092']
        labels:
          instance: '192.168.1.11'

重新加載配置

curl -X POST http://192.168.1.11:11090/-/reload

登錄prometheus輸入mysql關鍵詞可以看的mysql相關指標

三、Nginx Logs Exporter

$ cat docker-compose-nginxlogs_exporter.yml
version: "2.3"
services:
  nginx_exporter:
    image: quay.io/martinhelmich/prometheus-nginxlog-exporter
    command: -config-file /etc/prometheus-nginxlog-exporter.hcl
    container_name: nginx_exporter
    hostname: nginx_exporter
    volumes:
      - /usr/local/var/log/nginx:/mnt/nginxlogs # 本機實際的nginx日志問價位置：/usr/local/var/log/nginx
      - ./config.hcl:/etc/prometheus-nginxlog-exporter.hcl
    ports:
      - "11093:4040"
    restart: always
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

配置文件

listen {
  port = 4040
  address = "0.0.0.0"
}

namespace "nginx" {
  format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_time $upstream_response_time"

  source_files = ["/mnt/nginxlogs/access.log"]
  labels {
    app = "nginx-log"
  }

  relabel "request_url" {
    from = "request"
    split = 2

#    match "^/api/*" {
#      replacement = "api/"
#    }

    match "^/api/([a-z]+|[a-z]+_+[a-z]+)/([a-z]+|[a-z]+_+[a-z]+|[a-z]+_+[a-z]+_+[a-z]+|[a-z]+_+[a-z]+_+[a-z]+_+[a-z]+|[a-z]+_+[a-z]+_+[a-z]+_+[a-z]+_+[a-z]+)(/.*)" {
      replacement = "api/$1/$2"
    }

  }

}

Nginx logs exports config file

// config.hcl

定義的 format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_time $upstream_response_time" # nginx日志保存格式，需要與nginx.conf配置文件相同
source_files: # nginx日志文件位置
match "^/api/*" {  : # 僅匹配以api關鍵詞開頭的接口信息，支持正則表達式過濾。

修改nginx的log輸出格式：需要修改nginx的 nginx.conf 配置文件如下，重啟后記得reload nginx

  log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                         '"$request" $status $body_bytes_sent '
                         '"$http_referer" "$http_user_agent" '
                         '$request_time $upstream_response_time';

  access_log /usr/local/var/log/nginx/access.log upstream_time;

啟動&停止

docker-compose -f docker-compose-nginxlogs_exporter.yml up -d
docker-compose -f docker-compose-nginxlogs_exporter.yml down

prometheus.yml增加

# Nginx
  - job_name: 'Nginx'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.1.11:11093']
        labels:
          instance: '192.168.1.11'

重新加載prometheus

curl -X POST http://192.168.1.11:11090/-/reload

登錄prometheus輸入nginx關鍵詞可以看的nginx相關指標

四、配置Grafana通過圖表顯示

• 登錄Grafana，默認用戶名密碼為admin/admin

添加數據源（prometheus），名稱以APP為例，URL為prometheus的地址

點擊最后的Save&Test即可

導入Mysql 監控模版

• 推薦模版
• 更多模版可以在grafana官網查找

導入Nginx 監控模版

• 推薦模版
• 更多模版可以在grafana官網查找
• Import導入模版即可