本文主要是針對prometheus的blackbox_exporter，對集群中的服務進行狀態的監控
因為OpenShift 3.11版本自己帶的promethues修改起來有一些問題，所以自己安裝和部署了自己的prometheus和grafana
1.BlackBox安裝
blackbox的詳細情況可以參考
https://github.com/prometheus/blackbox_exporter
安裝過程如下
git clone https://github.com/prometheus/blackbox_exporter.git
在https://github.com/prometheus/blackbox_exporter/releases
下載最新版本的blackbox_exporter-0.14.0.linux-amd64.tar.gz , 解壓放到相應的目錄下，然后構建鏡像
docker build -t blackbox_exporter .
blackbox可以直接通過docker run來運行在本地，比如
docker run -d -p 9115:9115 --name blackbox blackbox_exporter
但問題是我們需要針對ocp內部的服務進行監控，因此最好還是作為pod運行在ocp的內部，這樣通過服務就可以獲取固定的url.
2.Blackbox部署成為Pod
下面三條語句，比較簡單了...
oc new-project blackbox oc import-image blackbox:latest --from=registry.example.com/openshift/blackbox_exporter:latest --insecure --confirm oc new-app blackbox:latest --name=blackbox
完成后看到有個blackbox pod在運行
3.部署prometheus
Prometheus部署使用的是
https://github.com/openshift/origin/tree/master/examples/prometheus
下載相關版本鏡像，推送到registry.example.com,然后
oc new-project prometheus oc new-app -f prometheus.yaml -p NAMESPACE=prometheus
因為原本prometheus的配置是在內部的，為了便於修改，直接放到宿主機上。
將需要監控的url寫入到prometheus.yml的配置
配置文件如下
[root@master prometheus]# cat prometheus.yml rule_files: - '*.rules' ..... - job_name: blackbox metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - http://ericapp1.ericproject1:8080 - http://ericapp2.ericproject1:8080 relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox.blackbox:9115 # Blackbox exporter. alerting: alertmanagers: - scheme: http static_configs: - targets: - "localhost:9093"
主要看blackbox這段.
oc create configmap prom-config --from-file=prometheus.yml oc set volume sts/prometheus --add --overwrite --name=config-volume --mount-path=/etc/prometheus/myconfig --source='{"configMap": { "name": "prom-config"}}'
然后進入console修改prometheus的yaml
一切完成，進入prometheus的主頁面，看到相關的指標
4.部署grafana
可以參考
https://github.com/openshift/origin/tree/master/examples/grafana
https://labs.consol.de/development/2018/01/19/openshift_application_monitoring.html
oc new-project grafana oc new-app -f grafana.yaml -p NAMESPACE=grafana oc policy add-role-to-user view system:serviceaccount:grafana:grafana-ocp -n prometheus
登錄grafana界面，添加數據源，這個數據源這里折騰了一段時間
因為是走的https,所以必須需要token,這個token值為
[root@master prometheus]# oc sa get-token grafana-ocp eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJncmFmYW5hIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImdyYWZhbmEtb2NwLXRva2VuLXhodDl2Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImdyYWZhbmEtb2NwIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2IyYmU5YjQtNThkMS0xMWU5LThlMDQtMDgwMDI3ZGM5OTFhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmdyYWZhbmE6Z3JhZmFuYS1vY3AifQ.NAf03C7v42aBNrRbwLcIosKDFgv3WyBA2-jZ73RqmX-OyNkshzgkvO6RZDGU7Oi9x27Hu7S69CW-2nZBDqSuCKP_fOyr7pKkOZTHfmgnXQdOeqnG8WSGM5HMbMsLbrAgOMZSehG5jFos-FaJb6jYJwQx485u0ViwWcSN58y7m4unIyrXgvzvjCoFijIjoCcYOvJklZVAsToVIFnevOcXEdJhtoitFiG2Agg4_KSiEUOhhKzyvKCIOLfSZN1UGsicUx_vFmqix8ARmOegILJNdRHksBAXRdReZuaJlJ1-FJnh1j9-QVVSSgw2DP6JrMEhS9gV38VWiWhoizSZLqsgCw
Save Test后一定需要出現is working啊.
New Dashboard
singlestat->edit->metrics,修改options
修改完最后
好啦，搞定！