prometheus+grafana+cadvisor監控docker容器

1、配置CAdvisor：

CAdvisor是谷歌開發的用於分析運行中容器的資源占用和性能指標的開源工具。CAdvisor是一個運行時的守護進程，負責收集、聚合、處理和輸出運行中容器的信息,他將以容器的方式運行在所有的host上。

{
  "id": "/monitor/cadvisor106",
  "role": "monitor",
  "cmd": null,
  "cpus": 0.1,
  "mem": 256,
  "disk": 0,
  "gpus": 0,
  "instances": 1,
  "constraints": [
    [
      "hostname",
      "CLUSTER",
      "10.20.31.106"
    ]
  ],
  "acceptedResourceRoles": [
    "*"
  ],
  "container": {
    "type": "DOCKER",
    "docker": {
      "forcePullImage": true,
      "image": "10.20.31.104:5000/library/google/cadvisor:latest",
      "parameters": [],
      "privileged": false
    },
    "volumes": [
      {
        "containerPath": "/etc/localtime",
        "hostPath": "/etc/localtime",
        "mode": "RO"
      },
      {
        "containerPath": "/rootfs",
        "hostPath": "/",
        "mode": "RO"
      },
      {
        "containerPath": "/var/run",
        "hostPath": "/var/run",
        "mode": "RO"
      },
      {
        "containerPath": "/sys",
        "hostPath": "/sys",
        "mode": "RO"
      },
      {
        "containerPath": "/dev/disk/",
        "hostPath": "/dev/disk/",
        "mode": "RO"
      },
      {
        "containerPath": "/var/lib/docker",
        "hostPath": "/var/lib/docker",
        "mode": "RO"
      },
      {
        "containerPath": "/cgroup",
        "hostPath": "/cgroup",
        "mode": "RO"
      }
    ],
    "portMappings": [
      {
        "containerPort": 8080,
        "hostPort": 0,
        "labels": {},
        "protocol": "tcp",
        "servicePort": 10130
      }
    ]
  },
  "labels": {
    "HAPROXY_GROUP": "external"
  },
  "networks": [
    {
      "mode": "container/bridge"
    }
  ],
  "portDefinitions": [],
  "maxLaunchDelaySeconds": 300
}

2、配置prometheus：

修改prometheus.yml文件增加如下：

 - job_name: 'docker161'
    scrape_interval: 5s

    file_sd_configs:
      - files: ['/etc/prometheus/fileconfig/docker-nodes161.json']

cat  /etc/prometheus/fileconfig/docker-nodes161.json

[
    {
        "targets": ["10.20.31.164:10128"],
        "labels": {
            "instance": "10.20.31.161",
            "alias": "161-docker",
            "job": "docker161"
        }
    }
]

因為是修改了prometheus.yml文件，所以要重新加載一下prometheus服務

curl -XPOST http://10.20.31.100:9090/-/reload

訪問prometheus頁面——Status——Targets可以查看到剛添加的docker主機信息。

3、配置grafana：

{
  "id": "/grafana",
  "role": "slave_public",
  "cmd": null,
  "cpus": 0.3,
  "mem": 512,
  "disk": 0,
  "gpus": 0,
  "instances": 1,
  "constraints": [
    [
      "hostname",
      "CLUSTER",
      "10.20.31.161"
    ]
  ],
  "acceptedResourceRoles": [
    "*"
  ],
  "container": {
    "type": "DOCKER",
    "docker": {
      "forcePullImage": false,
      "image": "10.20.31.104:5000/library/grafana",
      "parameters": [],
      "privileged": false
    },
    "volumes": [
      {
        "containerPath": "/etc/localtime",
        "hostPath": "/etc/localtime",
        "mode": "RO"
      },
      {
        "containerPath": "/var/lib/grafana",
        "hostPath": "/opt/GrafanaData",
        "mode": "RW"
      }
    ],
    "portMappings": [
      {
        "containerPort": 3000,
        "hostPort": 0,
        "labels": {},
        "protocol": "tcp",
        "servicePort": 10135
      }
    ]
  },
  "labels": {
    "HAPROXY_GROUP": "external"
  },
  "networks": [
    {
      "mode": "container/bridge"
    }
  ],
  "portDefinitions": [],
  "maxLaunchDelaySeconds": 300
}

注意：默認用戶名和密碼都是 admin。

grafana添加普羅米修斯服務器:

 如果一切順利該DataSource可以正常工作了，也就是說Grafana可以正常跟Prometheus正常通信了，接下來通過dashboard展示數據。

自己手工創建dashboard有點困難，可以借助開元的力量訪問 監控模板地址 將會看到很多用於監控 Docker 的 Dashboard。監控模板地址(多種監控模板根據自己需求下載不同的模板)

 有些dashboard可以下載后直接導入，而有些需要修改后再導入，需要看dashboard的overview

最后效果（首頁左上角Home---->dashboards）

但是這個模板，無法選擇根據主機選擇。推薦另外一個模板，它是可以選擇主機的，並且這個模板指標更多。

https://grafana.com/grafana/dashboards/10566