使用prometheus-operator監控 etcd狀態

一、查看 Etcd 信息

獲取 Etcd Pod 名稱

$ kubectl get pods -n kube-system | grep etcd
etcd-k8s-master-2-11                      1/1     Running   9          55d

查看 Etcd 描述信息

$ kubectl describe pod etcd-k8s-master-2-11 -n kube-system ...... Containers:  Command: etcd  --advertise-client-urls=https://192.168.2.11:2379  --cert-file=/etc/kubernetes/pki/etcd/server.crt  --client-cert-auth=true  --data-dir=/var/lib/etcd  --initial-advertise-peer-urls=https://192.168.2.11:2380  --initial-cluster=k8s-master-2-11=https://192.168.2.11:2380  --key-file=/etc/kubernetes/pki/etcd/server.key  --listen-client-urls=https://127.0.0.1:2379,https://192.168.2.11:2379  --listen-peer-urls=https://192.168.2.11:2380  --name=k8s-master-2-11  --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt  --peer-client-cert-auth=true  --peer-key-file=/etc/kubernetes/pki/etcd/peer.key  --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt  --snapshot-count=10000  --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt ...... 

可以看到 ETCD 的證書文件在 Kubernetes Master 節點的 “/etc/kubernetes/pki/etcd/” 文件夾下。

二、將證書存入 Kubernetes

利用 kubectl 命令將三個證書文件存入 Kubernetes 的 Secret 資源下。

$ kubectl create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key --from-file=/etc/kubernetes/pki/etcd/ca.crt -n monitoring

查看剛剛創建的資源

$ kubectl get secret etcd-certs -n monitoring 

NAME         TYPE     DATA   AGE
etcd-certs   Opaque   3      1m

三、將證書掛入 Prometheus

編譯 prometheus 資源

$ kubectl edit prometheus k8s -n monitoring

將證書掛入

apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata:  annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{},"labels":{"prometheus":"k8s"},"name":"k8s","namespace":"monitoring"},"spec":{"alerting":{"alertmanagers":[{"name":"alertmanager-main","namespace":"monitoring","port":"web"}]},"baseImage":"quay-mirror.qiniu.com/prometheus/prometheus","nodeSelector":{"beta.kubernetes.io/os":"linux"},"replicas":2,"resources":{"requests":{"memory":"400Mi"}},"ruleSelector":{"matchLabels":{"prometheus":"k8s","role":"alert-rules"}},"securityContext":{"fsGroup":2000,"runAsNonRoot":true,"runAsUser":1000},"serviceAccountName":"prometheus-k8s","serviceMonitorNamespaceSelector":{},"serviceMonitorSelector":{},"storage":{"volumeClaimTemplate":{"spec":{"resources":{"requests":{"storage":"8Gi"}},"storageClassName":"fast"}}},"version":"v2.7.2"}}  creationTimestamp: "2019-06-07T22:15:37Z"  generation: 5  labels:  prometheus: k8s  name: k8s  namespace: monitoring  resourceVersion: "2128109"  selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheuses/k8s  uid: c6daa0a1-8971-11e9-bc01-000c29d98697 spec:  alerting:  alertmanagers:  - name: alertmanager-main  namespace: monitoring  port: web  baseImage: quay-mirror.qiniu.com/prometheus/prometheus  nodeSelector: beta.kubernetes.io/os: linux  replicas: 2  resources:  requests:  memory: 400Mi  ruleSelector:  matchLabels:  prometheus: k8s  role: alert-rules  secrets: #------新增證書配置，將etcd證書掛入  - etcd-certs 

更新完成后就可以在 Prometheus Pod 中看到上面掛入的 etcd 證書，我們可以進入 Pod 中查看：

$ kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring /prometheus $ ls /etc/prometheus/secrets/etcd-certs/ ca.crt healthcheck-client.crt healthcheck-client.key 

四、創建 Etcd Service & Endpoints

因為 ETCD 是獨立於集群之外的，所以我們需要創建一個 Endpoints 將其代理到 Kubernetes 集群，然后創建一個 Service 綁定 Endpoints，然后 Kubernetes 集群的應用就可以訪問 ETCD 了。

etcd-service.yaml

apiVersion: v1 kind: Service metadata:  name: etcd-k8s  namespace: kube-system  labels:  k8s-app: etcd spec:  type: ClusterIP  clusterIP: None #設置為None，不分配Service IP  ports:  - name: port  port: 2379  protocol: TCP --- apiVersion: v1 kind: Endpoints metadata:  name: etcd-k8s  namespace: kube-system  labels:  k8s-app: etcd subsets: - addresses:  - ip: 192.168.2.11 #Etcd 所在節點的IP  ports:  - port: 2379 #Etcd 端口號 

創建 Service & Endpoints

$ kubectl apply -f etcd-service.yaml

五、創建 ServiceMonitor

創建 Prometheus 監控資源，配置用於監控 Etcd 參數。

etcd-monitor.yaml

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata:  name: etcd-k8s  namespace: monitoring  labels:  k8s-app: etcd-k8s spec:  jobLabel: k8s-app  endpoints:  - port: port  interval: 30s  scheme: https  tlsConfig:  caFile: /etc/prometheus/secrets/etcd-certs/ca.crt  certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt  keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key  insecureSkipVerify: true  selector:  matchLabels:  k8s-app: etcd  namespaceSelector:  matchNames:  - kube-system 

創建 Etcd ServiceMonitor

$ kubectl apply -f etcd-monitor.yaml

六、查看 Prometheus 規則

創建完成后查看 Prometheus UI，可以看到已經有對應監控數據

七、Grafana 引入 ETCD 儀表盤

完成 Prometheus 配置后，直接打開 Grafana 頁面，引入Dashboard，輸入編號 “3070” 的儀表盤

可以看到監控 ETCD 的各個看板