blackbox_exporter+grafana+prometheus監控主機存活，端口存活及網站狀態

一、簡介：

blackbox_exporter是Prometheus 官方提供的 exporter 之一，可以提供 http、dns、tcp、icmp 的監控數據采集。

Blackbox_exporter 應用場景

HTTP 測試
定義 Request Header 信息
判斷 Http status / Http Respones Header / Http Body 內容
TCP 測試
業務組件端口狀態監聽
應用層協議定義與監聽
ICMP 測試
主機探活機制
POST 測試
接口聯通性
SSL 證書過期時間

2、安裝blackbox_exporter：

$ wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
$ cd /usr/local/src
$ tar -xvf blackbox_exporter-0.16.0.linux-amd64.tar.gz
$ mv blackbox_exporter-0.16.0.linux-amd64.tar.gz blackbox_exporter

驗證是安裝版本

cd /usr/local/src/blackbox_exporter
[root@localhost blackbox_exporter]# ./blackbox_exporter --version
blackbox_exporter, version 0.16.0 (branch: HEAD, revision: 991f89846ae10db22a3933356a7d196642fcb9a9)
build user: root@64f600555645
build date: 20191111-16:27:24
go version: go1.13.4

創建systemd服務

 vim /lib/systemd/system/blackbox_exporter.service

[Unit]
Description=blackbox_exporter
After=network.target

[Service]
User=root
Type=simple
ExecStart=/usr/local/src/blackbox_exporter/blackbox_exporter --config.file=/usr/local/src/blackbox_exporter/blackbox.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target

如果以非root用戶運行blackbox_exporter，為了使用icmp prober，需要設置CAP_NET_RAW，即對可執行文件blackbox_exporter執行下面的命令:

$ cd /usr/local/src/blackbox_exporter
$ setcap cap_net_raw+ep blackbox_exporter

啟動blackbox_exporter

$ systemctl daemon-reload
$ systemctl start blackbox_exporter

驗證是否啟動成功

默認監聽端口為9115

$ systemctl status blackbox_exporter
$ netstat -lnpt|grep 9115

3、安裝prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.19.2/prometheus-2.19.2.linux-amd64.tar.gz
tar -xvf prometheus-2.19.2.linux-amd64.tar.gz
mv  prometheus-2.19.2.linux-amd64 prometheus

驗證安裝版本:

[root@localhost prometheus]# ./prometheus --version
prometheus, version 2.19.2 (branch: HEAD, revision: c448ada63d83002e9c1d2c9f84e09f55a61f0ff7)
  build user:       root@dd72efe1549d
  build date:       20200626-09:02:20
  go version:       go1.14.4

創建systemd服務

vim /lib/systemd/system/prometheus.service
[Unit]
Description=prometheus
After=network.target

[Service]
User=root
Type=simple
ExecStart=/usr/local/src/prometheus/prometheus --config.file=/usr/local/src/prometheus/prometheus.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target

啟動 prometheus:

systemctl daemon-reload
systemctl start prometheus.service

4、grafana安裝部署：

參考Grafana官網文檔：https://grafana.com/docs/grafana/latest/installation/rpm/#install-from-yum-repository

下載rpm文件，並使用yum install命令安裝

 

wget https://dl.grafana.com/oss/release/grafana-7.0.4-1.x86_64.rpm
sudo yum install grafana-7.0.4-1.x86_64.rpm

 

grafana啟動和查看狀態：

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server

grafana查看版本：

grafana-server -v

grafana設置開機自啟：

systemctl enable grafana-server

grafana安裝包信息：

二進制文件： /usr/sbin/grafana-server
init.d 腳本： /etc/init.d/grafana-server
環境變量文件： /etc/sysconfig/grafana-server
配置文件： /etc/grafana/grafana.ini
啟動項： grafana-server.service
日志文件：/var/log/grafana/grafana.log
默認配置的sqlite3數據庫：/var/lib/grafana/grafana.db

grafana訪問地址：

默認端口是3000，訪問地址：http://IP:3000

默認賬號/密碼：admin/admin

 

 5、prometheus.yml中加入blackbox_exporter

1.檢查主機存活狀態：

$ vim /usr/local/prometheus/prometheus.yml

  - job_name: 'node_status'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets: ['127.0.0.1']
        labels:
          instance: 'node_status'
          group: 'node'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
        #      - source_labels: [__param_target]
        #        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

2.監控主機端口存活狀態

$ vim /usr/local/prometheus/prometheus.yml

  - job_name: 'port_status'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets: ['127.0.0.1:9100','127.0.0.1:9090']
        labels:
          instance: 'port_status'
          group: 'tcp'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
        #      - source_labels: [__param_target]
        #        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

3.監控網站狀態

$ vim /usr/local/prometheus/prometheus.yml

  - job_name: web_status
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://www.baidu.com']
        labels:
          instance: web_status
          group: web
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 127.0.0.1:9115

檢查配置文件是否書寫正確：

$ cd /usr/local/src/prometheus

[root@localhost prometheus]# ./promtool check config prometheus.yml
Checking prometheus.yml
SUCCESS: 0 rule files found

重新加載prometheus的配置

systemctl reload prometheus
或
curl -X POST http://127.0.0.1:9090/-/reload (啟用了--web.enable-lifecycle選項)

訪問web界面

訪問：http://ip:9090/targets

 

 6、grafana中加入blackbox_exporter監控數據

導入blackbox_exporter模板  此模板為9965號模板，數據源選擇Prometheus 模板下載地址 https://grafana.com/grafana/dashboards/9965

 

此模板需要安裝餅狀圖插件 下載地址 https://grafana.com/grafana/plugins/grafana-piechart-panel
安裝插件，重啟grafana生效。

$ grafana-cli plugins install grafana-piechart-panel
$ service grafana-server restart

 查看監控數據：

 

 備注：本文是監控本機的相關端口進行測試的，如果需要監聽其它主機。需要在其它主機上面安裝 blackbox_exportex

安裝完成后，需要在prometheus.yml今天添加相關監控指標：