1、淺析整個監控流程

heapster是一個監控計算、存儲、網絡等集群資源的工具，以k8s內置的cAdvisor作為數據源收集集群信息，並匯總出有價值的性能數據(Metrics)：cpu、內存、網絡流量等，然后將這些數據輸出到外部存儲，如InfluxDB，最后就可以通過相應的UI界面顯示出來，如grafana。另外heapster的數據源和外部存儲都是可插拔的，所以可以很靈活的組建出很多監控方案，如：Heapster+ElasticSearch+Kibana等等。

2、創建k8s資源對象

使用官方提供的yml文件有一些小問題，請參考以下改動和說明：

2.1、創建InfluxDB資源對象

apiVersion: apps/v1 kind: Deployment metadata: name: monitoring-influxdb namespace: kube-system spec: replicas: 1 selector: matchLabels: task: monitoring k8s-app: influxdb template: metadata: labels: task: monitoring k8s-app: influxdb spec: containers: - name: influxdb image: k8s.gcr.io/heapster-influxdb-amd64:v1.3.3 volumeMounts: - mountPath: /data  name: influxdb-storage  volumes:  - name: influxdb-storage  emptyDir: {} --- apiVersion: v1 kind: Service metadata:  labels:  task: monitoring  kubernetes.io/cluster-service: 'true' kubernetes.io/name: monitoring-influxdb name: monitoring-influxdb namespace: kube-system spec: type: NodePort ports: - nodePort: 31001 port: 8086 targetPort: 8086 selector: k8s-app: influxdb

注意：這里我們使用NotePort暴露monitoring-influxdb服務在主機的31001端口上，那么InfluxDB服務端的地址：http://[host-ip]:31001 ，記下這個地址，以便創建heapster和為grafana配置數據源時，可以直接使用。

2.1、創建Grafana資源對象

apiVersion: apps/v1 kind: Deployment metadata: name: monitoring-grafana namespace: kube-system spec: replicas: 1 selector: matchLabels: task: monitoring k8s-app: grafana template: metadata: labels: task: monitoring k8s-app: grafana spec: containers: - name: grafana image: k8s.gcr.io/heapster-grafana-amd64:v4.4.3 ports: - containerPort: 3000 protocol: TCP volumeMounts: - mountPath: /etc/ssl/certs  name: ca-certificates  readOnly: true  - mountPath: /var name: grafana-storage env: - name: INFLUXDB_HOST value: monitoring-influxdb - name: GF_SERVER_HTTP_PORT value: "3000" # The following env variables are required to make Grafana accessible via # the kubernetes api-server proxy. On production clusters, we recommend # removing these env variables, setup auth for grafana, and expose the grafana # service using a LoadBalancer or a public IP. - name: GF_AUTH_BASIC_ENABLED value: "false" - name: GF_AUTH_ANONYMOUS_ENABLED value: "true" - name: GF_AUTH_ANONYMOUS_ORG_ROLE value: Admin - name: GF_SERVER_ROOT_URL # If you're only using the API Server proxy, set this value instead: # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy value: / volumes: - name: ca-certificates hostPath: path: /etc/ssl/certs  - name: grafana-storage  emptyDir: {} --- apiVersion: v1 kind: Service metadata:  labels:  # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)  # If you are NOT using this as an addon, you should comment out this line.  kubernetes.io/cluster-service: 'true' kubernetes.io/name: monitoring-grafana name: monitoring-grafana namespace: kube-system spec: # In a production setup, we recommend accessing Grafana through an external Loadbalancer # or through a public IP. # type: LoadBalancer # You could also use NodePort to expose the service at a randomly-generated port type: NodePort ports: - nodePort: 30108 port: 80 targetPort: 3000 selector: k8s-app: grafana

注意：這里我們使用NotePort暴露monitoring-grafana服務在主機的30108上，那么Grafana服務端的地址：http://registry.wuling.com:30108 ，通過瀏覽器訪問，為Grafana修改數據源，如下：

標紅的地方，為上一步記錄下的InfluxDB服務端的地址。

2.2、創建Heapster資源對象

apiVersion: v1 kind: ServiceAccount metadata: name: heapster namespace: kube-system --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: heapster namespace: kube-system spec: replicas: 1 selector: matchLabels: task: monitoring k8s-app: heapster template: metadata: labels: task: monitoring k8s-app: heapster spec: serviceAccountName: heapster containers: - name: heapster image: k8s.gcr.io/heapster-amd64:v1.4.2 imagePullPolicy: IfNotPresent command: - /heapster  - --source=kubernetes:https://kubernetes.default   - --sink=influxdb:http://150.109.39.33:31001 # 這里填寫剛剛記錄下的InfluxDB服務端的地址。 --- apiVersion: v1 kind: Service metadata:  labels:  task: monitoring  # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)  # If you are NOT using this as an addon, you should comment out this line.  kubernetes.io/cluster-service: 'true' kubernetes.io/name: Heapster name: heapster namespace: kube-system spec: ports: - port: 80 targetPort: 8082 selector: k8s-app: heapster

--source 為heapster指定獲取集群信息的數據源。參考：https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md
--sink 為heaster指定后端存儲，這里我們使用InfluxDB，其他的，請參考：https://github.com/kubernetes/heapster/blob/master/docs/sink-owners.md
這里heapster留下了一個的坑，請繼續往下看，當我部署完heapster，查看Heapster容器組的標准輸出：

很多人都以為是https或者k8s配置的問題，於是去就慌忙的去配置InSecure http方式，導致坑越來越深，透明度越來越低，更是無從下手，我也是這樣弄了很久，都較上勁了，此處省略一萬字。。。，當這些路子都走遍了，再次品讀下面的原文：

才發現是權限的問題，heaster默認使用一個令牌(Token)與ApiServer進行認證，通過查看heapster.yml發現 serviceAccountName: heapster ，現在明白了吧，就是heaster沒有權限，那么如何授權呢-----給heaster綁定一個有權限的角色就行了，如下：

apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: heapster roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: heapster namespace: kube-system