prometheus系列監控：jvm,mongodb,mysql,redis,consul

jvm：

maven添加dependence

<!-- https://mvnrepository.com/artifact/io.micrometer/micrometer-registry-prometheus -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <version>1.3.5</version>
</dependency>

 

編輯springboot項目的yml文件

yml配置參考https://blog.csdn.net/u014401141/article/details/84784422

 

server:
  port: 8085


spring:
  #for monitor
  application:
    name: mall_prometheus






management:
  #TODO endpoint 解釋查詢
  endpoints:
    web:
      exposure:
        #include: "*"
        include: info, health, beans, env, metrics, mappings, scheduledtasks, sessions, threaddump, docs, logfile, jolokia, prometheus
      base-path: /actuator #默認該路徑，不更改可不用配置
      #cors跨域支持
      cors:
        allowed-origins: http://example.com
        allowed-methods: GET,PUT,POST,DELETE
    prometheus:
      id: springmetrics
  endpoint:
    beans:
      cache:
        time-to-live: 10s #端點緩存響應的時間量
    health:
      show-details: always #詳細信息顯示給所有用戶
  server:
    port: 8001 #默認8888
    #address: 127.0.0.1 #配置此項表示不允許遠程連接
  #monitor
  metrics:
    export:
      datadog:
        application-key: ${spring.application.name}
    web:
      server:
        auto-time-requests: true

　　

配置prometheus.yml

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus
  - job_name: linux
    static_configs:
      - targets: ['47.112.188.174:9100']
        labels:
          instance: node
  - job_name: 'spring'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['47.112.188.174:8001']

　　

docker 啟動服務時，開放8001端口

# 9100是exporter的端口
docker run -p 8001:8001 -p 8085:8085 --name mall-portal \
 --link mall-mysql:db \
 --link mall-redis:redis \
 --link mongo:mongo \
 --link rabbitmq:rabbit \
 -v /etc/localtime:/etc/localtime \
 -v/usr/local/dockerdata/mall-project/mall-port/logs:/var/logs \
 -d mall/mall-portal:1.0-SNAPSHOT

　　

啟動prometheus時加載配置

docker run --name prometheus -d -p 9090:9090 --privileged=true -v /usr/local/dockerdata/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

　　

grafana添加對應dashboard:4701

https://grafana.com/grafana/dashboards/4701

 

 

 mysql

 

啟動mysql的時候

數據庫內執行：(添加exporter的權限)

use mall;
GRANT REPLICATION CLIENT, PROCESS ON *.* to 'exporter'@'%' identified by 'exporter';
GRANT SELECT ON performance_schema.* TO 'exporter'@'%';
flush privileges;

　　

docker 啟動 mysqld_exporter（使用9104端口向prometheus傳遞數據）

docker run -d --restart=always --name mysqld-exporter -p 9104:9104 -e DATA_SOURCE_NAME='exporter:exporter@(47.112.188.174:3306)/' prom/mysqld-exporter

　　

　

curl 驗證一下

curl localhost:9104/metrics
process_virtual_memory_bytes 1.16281344e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

　　 

然后修改一下prometheus的yaml。啟動prometheus。。。。。具體編寫和運行不再贅述。。。

 

 

--------------------------------------------這里是服務發現---------------------------------------------------------------------

上面這種方式太過繁瑣，幾乎每次加一個exporter，都要修改prometheus.yml，並重啟。

優化方案：可以使用consul來自動搜尋服務，每當要加一個exporter ，就通過consul的自動發現來加到prometheus，不用重新配置yml。

consul官網：https://www.consul.io/downloads.html

步驟1 ：docker 啟動consul，啟用8500端口

consul 服務發現
docker run --restart=always --name consul -d -p 8500:8500 consul

　

服務啟動后訪問url看一下： curl localhost:8500/ui　

 

步驟2：修改prometheus.yml　　，添加consul的job，這里配置成根據tag里面有“mall"字樣來添加。

　

  - job_name: consul
    consul_sd_configs:
      - server: '47.112.188.174:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*mall.*
        action: keep

 

啟動prometheus時，傳入該配置：

docker run --name prometheus -d -p 9090:9090 --privileged=true -v /usr/local/dockerdata/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml
　　

　　

curl localhost:9090/target 看一下是否啟動成功，這里根據配置會有兩個job：prometheus，node

 

步驟3：注冊服務(即傳入exporter的開放端口，這里傳入mysqld_exporter的9104)

curl -X PUT -d '{"id": "mysql01","name": "mysql01","address": "47.112.188.174","port": 9104,"tags": ["mall"],"checks": [{"http": "http://47.112.188.174:9104/","interval": "5s"}]}' http://localhost:8500/v1/agent/service/register

 

 

 

 

-----------------------------------------------添加redis,mongodb 只需要啟動對應的docker exporter，然后步驟3傳入服務就可以了--------------------------------------------------------------

添加一下mongodb redis的exporter

mongodb / redis
./redis_exporter -redis.addr 47.93.32.161:6379 & -web.listenaddress 0.0.0.0:9122
./mongodb_exporter-linux-amd64 -mongodb.uri mongodb://47.93.32.161:27017/admin http://localhost:8500/v1/agent/service/register

 

mongodb_exporter我用的docker

docker run -d --name=mongodb_exporter -p 9001:9001  mongodb_exporter

　

添加mongodb的服務

curl -X PUT -d '{"id": "mongo01","name": "mongo01","address": "47.112.188.174","port": 9001,"tags": ["mall"],"checks": [{"http": "http://47.112.188.174/","interval": "5s"}]}' http://localhost:8500/v1/agent/service/register

　

 

 

對應的dashboard 參考github給的吧。。。懶得搞了

https://github.com/dcu/mongodb_exporter　

 

 

添加spring的服務

 

curl -X PUT -d '{"id": "spring01","name": "spring01","address": "47.112.188.174","port": 8001,"tags": ["mall"],"checks": [{"http": "http://47.112.188.174:8001/actuator/prometheus","interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register　　

　　

　

這里因為spring的exporter 路徑不是/metrics，在curl后 服務從 localhost:8001/metrics讀不到有效數據，會有報錯。這里就需要將默認的/metrics轉換成/actuator/prometheus，更改一下yml文件：

  - job_name: consul
    consul_sd_configs:
      - server: '47.112.188.174:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*mall.*
        action: keep
      - source_labels: [__meta_consul_service]
        regex: spring01
        target_label: __metrics_path__
        replacement: /actuator/prometheus
        action: replace

　　

------------------------------------其他---------------------------------

幾個exporter的git地址，這里要特別注意，搜索exporter的時候要注意是否有自帶的dashboard，否則直接從grafana添加，可能dashboard並不匹配

https://github.com/prometheus/mysqld_exporter/releases
https://github.com/oliver006/redis_exporter/releases
https://github.com/percona/mongodb_exporter/releases

 

 

最后：

1.curl后 consul的展示

 

 

2. consul自動接入的服務，在prometheus上是否存儲成功

 

mysql的grafana視圖選6239