基於consul服務發現監控prometheus數據采集

業務服務使用micrometer + prometheus + grafana模式進行監控，如果服務機器列表發生變化，需要改動prometheus.yml機器域名或者IP，如果發生機器遷移或者擴縮容則容易遺忘，因此，我們可以通過prometheus與consul集成來實現動態監測機器列表變動來獲取對應的監控服務機器信息。

1、業務服務目前基本是基於consul做的服務發現，可以直接在promethus.yml里配置，注意通過drop過濾非業務服務，並進行標簽的重命名；

 - job_name: 'consul-prometheus'
    metrics_path: /actuator/prometheus
    scheme: http
    consul_sd_configs:
    -  server: 'consul_ip:8500'
       services: [] 
    relabel_configs:
       - source_labels:
           -  __meta_consul_service
         regex: 'consul|nginx|comment'
         action: drop
       - source_labels:
           -  __meta_consul_service
         target_label: group

2、將nginx服務注冊到consul上，在promethus.yml里配置，注意通過nginx服務，並進行標簽的重命名；

- job_name: 'nginx-consul-prometheus'
  metrics_path: /metrics
  consul_sd_configs:
  - server: 'consul_ip:8500'
    services: 
    - nginx 
 relabel_configs:
  - source_labels: ["__meta_consul_service"]
    target_label: "group"

nginx服務機器上需安裝consul客戶端，模板文件nginx.json，啟動consul客戶端時記得要額外加載該配置文件

{
  "service":{
  "id": "nginx",
  "name": "nginx",
  "address": "{{hostIp['stdout']}}",
  "port": 9145,
  "meta": {
    "sogou_host": "{{hostIp['stdout']}}"
  },
  "tags": ["nginx"],
  "checks": [
  {
   "http": "http://{{hostIp['stdout']}}:9145/metrics",
   "interval": "30s"
  }
  ]
 }
}

 3、ansible腳本如下，將nginx服務注冊consul：

- name: register hostIp
  shell: /sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:" |head -n 1
  register: hostIp
  ignore_errors: True
  tags:
    - consul

- name: register nginx_metrics
  shell: netstat -ntlp | grep 9145
  register: nginx_metrics
  ignore_errors: True
  failed_when: nginx_metrics.rc != 1 and nginx_metrics.rc != 0

- name: Copy consul.json
  template: src=consul.hcl.j2 dest=/search/odin/consul/conf/consul.hcl

- name: Copy nginx.json
  template: src=nginx.json.j2 dest=/search/odin/consul/conf/nginx.json
  when: nginx_metrics.rc == 0
  tags:
    - consul

- name: Copy systemd script
  template: src=consul-promethus.service dest=/etc/systemd/system/consul-promethus.service
  tags:
    - consul

- name: chown /search/odin
  file:
    path: /search/odin/consul
    owner: odin
    group: odin
    mode: 0755
    recurse: yes
  tags:
    - consul

- name: Service consul started
  systemd:
    daemon_reload: yes
    enabled: yes
    name: consul-promethus
    state: restarted
  tags:
    - consul

4、安裝consul太麻煩，可以直接使用api將服務注冊到consul server，可以忽略上面的了

curl http://consulip:8500/v1/agent/service/register -X PUT -i -H "Content-Type:application/json" -d '{
 "ID": "id",  
 "Name": "name",
 "Tags": [
   "nginx"
 ],
 "Address": "ip",
 "Port": 9145,
 "Check": {
   "DeregisterCriticalServiceAfter": "30s",
   "HTTP": "http://ip:9145/metrics",
   "Interval": "30s"
 }
}'

 

。。。