VictoriaMetrics：使用vmctl來實現vm-storage向victoria-metrics-prod(單機版)遷移數據

前一篇提到了，vm-storage的備份數據，無法被victoria-metrics-prod(單機版)讀取。
繼續翻文檔發現vmctl可以實現這個效果：

1.啟動vm-restore恢復數據

vmrestore-prod \
  -configFilePath="/etc/cos/config.ini" \
  -credsFilePath="/etc/cos/creds.ini" \
  -customS3Endpoint="https://cos.ap-guangzhou.myqcloud.com" \
  -src="s3://bucket/vm-storage/2021-12-27/" \
  -storageDataPath="/var/victoria-metrics/data/2021-12-27/" \
  -httpListenAddr="0.0.0.0:8420" \
  -memory.allowedBytes=2GB \
  -concurrency=10 \
  -maxBytesPerSecond=100MB 

2.啟動vm-storage

vmstorage-prod \
  -memory.allowedBytes 30GB \
  -storageDataPath /var/victoria-metrics/data/2021-12-27/ \
  -retentionPeriod=15d \
  -httpListenAddr 0.0.0.0:8482 \
  -vminsertAddr 0.0.0.0:8400 \
  -vmselectAddr 0.0.0.0:8401 \
  -search.maxTagKeys 100000 \
  -search.maxTagValues 1000000 \
  -search.maxUniqueTimeseries=10000000

3.啟動一個vm-select來查詢這個節點

vmselect-prod \
  -cacheDataPath /var/victoria-metrics/cache \
  -memory.allowedBytes 10GB \
  -replicationFactor 1 \
  -storageNode 127.0.0.1:8401 \
  -httpListenAddr 0.0.0.0:8481 \
  -search.maxConcurrentRequests 2000 \
  -search.maxPointsPerTimeseries 5000000 \
  -search.maxQueryDuration 600s

4.啟動victoria-metrics-prod(單機版)

victoria-metrics-prod \
  -dedup.minScrapeInterval=30s \
  -httpListenAddr="0.0.0.0:8428" \
  -insert.maxQueueDuration=2m \
  -maxInsertRequestSize=500MB \
  -memory.allowedBytes=7GB \
  -retentionPeriod=1y \
  -search.maxConcurrentRequests=500 \
  -search.maxPointsPerTimeseries=100000 \
  -search.maxUniqueTimeseries=1000000 \
  -storageDataPath=/var/victoria-metrics/data/vm-single/data/

5.啟動vmctl來傳輸數據

vmctl-prod vm-native \
  --vm-native-filter-time-start=0 \
  --vm-native-filter-time-end=$(date +"%s") \
  --vm-native-src-addr="http://127.0.0.1:8481/select/0/prometheus/" \
  --vm-native-dst-addr="http://127.0.0.1:8428/"

運行結果：

VictoriaMetrics Native import mode
Initing export pipe from "http://127.0.0.1:8481/select/0/prometheus" with filters:
        filter: match[]={__name__!=""}
        start: 0
        end: 1640919407
Initing import process to "http://127.0.0.1:8428":
Total: 5.81 GiB ↖ Speed: 1.29 MiB p/s                                                                                                                                                                                                                                       
2021/12/31 12:13:22 Total time: 1h16m35.022151869s

結論：

• 用vmctl來傳輸數據，奇慢，每秒傳輸幾百KB數據。生產環境中用於遷移數據，簡直是找死！
  • 統計下來，每核傳輸數速率 663kb/s (主要是vm單機版耗CPU很高)
  • 傳輸的時候，所有核跑滿。可以通過增加CPU資源加快導入速度。
• 這個東西可能只適合特定的某些數據的導入導出