prometheus監控之自動發現

prometheus監控之自動發現,這里采用服務端添加配置文件,具體操作如下,目前prometheus server只有如下節點:

 

 

 現在開始添加配置文件:

1.首先創建存放配置文件的目錄:

# mkdir /usr/local/prometheus/target/node/ -p

2.然后在prometheus server配置文件中定義file類型:

  - job_name: 'host_discovery'
    file_sd_configs:
    - files:
      - "/usr/local/prometheus/target/node/host_discovery.json"
      refresh_interval: 6s

3.添加targets到json配置文件中:

[root@master node]# cat host_discovery.json 
[{
    "targets": ["172.16.23.121:9100"],
    "labels": {
        "instance": "172.16.23.121",
        "role": "node1"
    }
}]

上面雙引號一定要注意不要使用單引號,不然日志會出現如下報錯:

[root@master node]# tail -f /var/log/messages
Sep 22 23:06:50 master prometheus: level=error ts=2020-09-22T15:06:50.849Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:06:50 master prometheus: level=error ts=2020-09-22T15:06:50.849Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:06:50 master prometheus: level=error ts=2020-09-22T15:06:50.849Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:06:54 master prometheus: level=error ts=2020-09-22T15:06:54.378Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:06:54 master prometheus: level=error ts=2020-09-22T15:06:54.866Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"
Sep 22 23:07:00 master prometheus: level=error ts=2020-09-22T15:07:00.378Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/usr/local/prometheus/target/node/host_discovery.json err="invalid character '\\'' looking for beginning of value"

4.配置完成后,最后校驗prometheus server配置文件,校驗沒問題后就加載配置文件:

# /usr/local/prometheus/promtool check config /usr/local/prometheus/prometheus.yml
# curl -X POST http://172.16.23.120:9090/-/reload

5.刷新控制台,查看targets節點:

 

 接下來通過向json文件host_discovery.json增加節點,然后不加載prometheus,刷新控制台看效果:

[root@master node]# cat host_discovery.json 
[{
    "targets": ["172.16.23.121:9100","172.16.23.122:9100"],
    "labels": {
        "instance": "172.16.23.121",
        "role": "node1"
    }
}]

刷新prometheus控制台:

 

 可以看見新增了122節點,但是label屬性都是相同,這種明顯不是我們希望看見的,所以繼續修改:

[root@master node]# cat host_discovery.json 
[{
    "targets": ["172.16.23.121:9100"],
    "labels": {
        "instance": "172.16.23.121",
        "role": "node1"
    }
},
{
    "targets": ["172.16.23.122:9100"],
    "labels": {
        "instance": "172.16.23.122",
        "role": "node2"
    }
}]

上面配置就是一個targets對應自己私有的label,然后刷新控制台:

 

 當然基於配置文件發現的方式除了json文件,yaml文件也是可以,操作如下:

[root@master node]# cat host_discovery.yml 
- targets:
  - "172.16.23.121:9100"
  labels:
    instance: "172.16.23.121"

然后重載prometheus server:

# curl -X POST http://172.16.23.120:9090/-/reload

刷新控制台:

 

 繼續向yml配置文件添加節點:

[root@master node]# cat host_discovery.yml 
- targets:
  - "172.16.23.121:9100"
  labels:
    instance: "172.16.23.121"
- targets:
  - "172.16.23.122:9100"
  labels:
    instance: "172.16.23.122"

然后刷新控制台: