prometheus學習系列七： Prometheus promQL查詢語言

 Prometheus promQL查詢語言

Prometheus提供了一種名為PromQL (Prometheus查詢語言)的函數式查詢語言，允許用戶實時選擇和聚合時間序列數據。表達式的結果既可以顯示為圖形，也可以在Prometheus的表達式瀏覽器中作為表格數據查看，或者通過HTTP API由外部系統使用。

准備工作

在進行查詢，這里提供下我的配置文件如下

[root@node00 prometheus]# cat prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']
  - job_name: "node"
    file_sd_configs:
    - refresh_interval: 1m
      files: 
      - "/usr/local/prometheus/prometheus/conf/node*.yml"
remote_write:
  - url: "http://localhost:8086/api/v1/prom/write?db=prometheus"

remote_read:
  - url: "http://localhost:8086/api/v1/prom/read?db=prometheus"


[root@node00 prometheus]# cat conf/node-dis.yml 
- targets: 
  - "192.168.100.10:20001"
  labels: 
    __datacenter__: dc0
    __hostname__: node00
    __businees_line__: "line_a"
    __region_id__: "cn-beijing"
    __availability_zone__: "a"
- targets: 
  - "192.168.100.11:20001"
  labels: 
    __datacenter__: dc1
    __hostname__: node01
    __businees_line__: "line_a"
    __region_id__: "cn-beijing"
    __availability_zone__: "a"
- targets: 
  - "192.168.100.12:20001"
  labels: 
    __datacenter__: dc0
    __hostname__: node02
    __businees_line__: "line_c"
    __region_id__: "cn-beijing"
    __availability_zone__: "b"

簡單時序查詢

直接查詢特定metric_name

# 節點的forks的總次數
node_forks_total
#結果如下

Element	Value
node_forks_total{instance="192.168.100.10:20001",job="node"}	201518
node_forks_total{instance="192.168.100.11:20001",job="node"}	23951
node_forks_total{instance="192.168.100.12:20001",job="node"}	24127

帶標簽的查詢

node_forks_total{instance="192.168.100.10:20001"}
# 結果如下

Element	Value
node_forks_total{instance="192.168.100.10:20001",job="node"}	201816

多標簽查詢

node_forks_total{instance="192.168.100.10:20001",job="node"}

# 結果如下

Element	Value
node_forks_total{instance="192.168.100.10:20001",job="node"}	201932

查詢2分鍾的時序數值

node_forks_total{instance="192.168.100.10:20001",job="node"}[2m]

Element	Value
node_forks_total{instance="192.168.100.10:20001",job="node"}	201932 @1569492864.036 201932 @1569492879.036 201932 @1569492894.035 201932 @1569492909.036 201985 @1569492924.036 201989 @1569492939.036 201993 @1569492954.036

 正則匹配

node_forks_total{instance=~"192.168.*:20001",job="node"}

Element	Value
node_forks_total{instance="192.168.100.10:20001",job="node"}	202107
node_forks_total{instance="192.168.100.11:20001",job="node"}	24014
node_forks_total{instance="192.168.100.12:20001",job="node"}	24186

常用函數查詢

官方提供的函數比較多， 具體可以參考地址如下： https://prometheus.io/docs/prometheus/latest/querying/functions/

這里主要就常用函數進行演示。

irate

irate用於計算速率。

# 通過標簽查詢，特定實例特定job，特定cpu 在idle狀態下的cpu次數速率
irate(node_cpu_seconds_total{cpu="0",instance="192.168.100.10:20001",job="node",mode="idle"}[1m])

Element	Value
{cpu="0",instance="192.168.100.10:20001",job="node",mode="idle"}	0.9833988932595507

count_over_time

計算特定的時序數據中的個數。

# 這個數值個數和采集頻率有關， 我們的采集間隔是15s，在一分鍾會有4個點位數據。
count_over_time(node_boot_time_seconds[1m])

Element	Value
{instance="192.168.100.10:20001",job="node"}	4
{instance="192.168.100.11:20001",job="node"}	4
{instance="192.168.100.12:20001",job="node"}	4

 

子查詢

 

# 過去的10分鍾內， 每分鍾計算下過去5分鍾的一個速率值。 一個采集10m/1m一共10個值。
rate(node_cpu_seconds_total{cpu="0",instance="192.168.100.10:20001",job="node",mode="idle"}[5m])[10m:1m]

Element	Value
{cpu="0",instance="192.168.100.10:20001",job="node",mode="idle"}	0.9865228543057867 @1569494040 0.9862807017543735 @1569494100 0.9861087231885309 @1569494160 0.9864946894550303 @1569494220 0.9863192502430038 @1569494280 0.9859649122807017 @1569494340 0.9859298245613708 @1569494400 0.9869122807017177 @1569494460 0.9867368421052672 @1569494520 0.987438596491273 @1569494580

 

復雜查詢

計算內存使用百分比

node_memory_MemFree_bytes / node_memory_MemTotal_bytes  * 100 

Element	Value
{instance="192.168.100.10:20001",job="node"}	9.927579722322251
{instance="192.168.100.11:20001",job="node"}	59.740727403673034
{instance="192.168.100.12:20001",job="node"}	63.2080982675149

獲取所有實例的內存使用百分比前2個

topk(2,node_memory_MemFree_bytes / node_memory_MemTotal_bytes  * 100 )

Element	Value
{instance="192.168.100.12:20001",job="node"}	63.20129636298163
{instance="192.168.100.11:20001",job="node"}	59.50586164125955

實用查詢樣例

獲取cpu核心個數

# 計算所有的實例cpu核心數
count by (instance) ( count by (instance,cpu) (node_cpu_seconds_total{mode="system"}) ) 
# 計算單個實例的
count by (instance) ( count by (instance,cpu) (node_cpu_seconds_total{mode="system",instance="192.168.100.11:20001"})  

計算內存使用率

(1 - (node_memory_MemAvailable_bytes{instance=~"192.168.100.10:20001"} / (node_memory_MemTotal_bytes{instance=~"192.168.100.10:20001"})))* 100

Element	Value
{instance="192.168.100.10:20001",job="node"}	87.09358620413717

計算根分區使用率

100 - ((node_filesystem_avail_bytes{instance="192.168.100.10:20001",mountpoint="/",fstype=~"ext4|xfs"} * 100) / node_filesystem_size_bytes {instance=~"192.168.100.10:20001",mountpoint="/",fstype=~"ext4|xfs"})

Element	Value
{device="/dev/mapper/centos-root",fstype="xfs",instance="192.168.100.10:20001",job="node",mountpoint="/"}	4.175111443575972

 預測磁盤空間

 # 整體分為 2個部分， 中間用and分割， 前面部分計算根分區使用率大於85的， 后面計算根據近6小時的數據預測接下來24小時的磁盤可用空間是否小於0 。
 (1-  node_filesystem_avail_bytes{fstype=~"ext4|xfs",mountpoint="/"}
  / node_filesystem_size_bytes{fstype=~"ext4|xfs",mountpoint="/"}) * 100 >= 85     and (predict_linear(node_filesystem_avail_bytes[6h],3600 * 24) < 0)