prometheus數據持久化設計

一、prometheus數據持久化設計

1、需求

1分鍾級別的數據保留 10天，10分鍾級別的數據保留20天，1小時級別的數據保留30天，1天級別的保留3個月

2、方案

1、創建4個數據庫，分別存儲不同頻率的數據，prometheus：原始數據 、prometheus_10m：按10分鍾頻率的數據、prometheus_1h：按1小時頻率的數據、prometheus_1d：按天頻率的數據

2、創建CQ，統計上級頻率數據並寫入相應頻率的數據庫中；10m由原始數據統計，1h由10m數據統計，1d數據由1h數據統計

3、開發influxdb remote read adapter，根據查詢的開始時間來確定從哪個數據庫查數據

3、db設計

prometheus , 數據保留：15d，分區：1d

prometheus_10m , 數據保留：20d，分區：1d

prometheus_1h , 數據保留：30d，分區：1d

prometheus_1d , 數據保留：730d，分區：7d

4、相應的sql

CREATE DATABASE "prometheus" WITH DURATION 15d SHARD DURATION 1d NAME "prometheus"
CREATE DATABASE "prometheus_10m" WITH DURATION 20d SHARD DURATION 1d NAME "prometheus_10m"
CREATE DATABASE "prometheus_1h" WITH DURATION 30d SHARD DURATION 1d NAME "prometheus_1h"
CREATE DATABASE "prometheus_1d" WITH DURATION 730d SHARD DURATION 7d NAME "prometheus_1d"

 

5、CQ

 

CREATE CONTINUOUS QUERY "cq_prometheus_10m" ON "prometheus" BEGIN SELECT mean(value) as "value" INTO "prometheus_10m"..:MEASUREMENT FROM prometheus../.*/ GROUP BY time(10m),* END 

 

CREATE CONTINUOUS QUERY "cq_prometheus_1h" ON "prometheus_10m" BEGIN SELECT mean(value) as "value" INTO "prometheus_1h"..:MEASUREMENT FROM prometheus_10m../.*/ GROUP BY time(1h),* END 

 

CREATE CONTINUOUS QUERY "cq_prometheus_1d" ON "prometheus_1h" BEGIN SELECT mean(value) as "value" INTO "prometheus_1d"..:MEASUREMENT FROM prometheus_1h../.*/ GROUP BY time(1d),* END 

adapter

主要是做http請求轉發，基於prometheus remote read的請求中Query的StartTimestampMs來確定/api/v1/prom/read?db=%s最終db調用哪一個

StartTimestampMs在7天內轉發到/api/v1/prom/read?db=prometheus

StartTimestampMs在7-15天內轉發到/api/v1/prom/read?db=prometheus_10m

StartTimestampMs在15-30天內轉發到/api/v1/prom/read?db=prometheus_1h

StartTimestampMs在30天以上轉發到/api/v1/prom/read?db=prometheus_1d