碼上歡樂
相關內容    簡體    繁體

kubernetes之監控Operator部署Prometheus

本文轉載自   查看原文   2020-06-27 15:41   644    k8s-docker

什么是Operator

Operator是由 CoreOS 公司開發的，用來擴展 Kubernetes API，特定的應用程序控制器。它被用來創建、配置和管理復雜的有狀態應用，如數據庫、緩存和監控系統。Operator 是基於 Kubernetes 的資源和控制器概念之上構建，但同時又包含了應用程序特定的一些專業知識：比如創建一個數據庫的Operator，則必須對創建的數據庫的各種運維方式非常了解，創建Operator的關鍵是CRD（自定義資源）的設計。

Operator是將運維人員對軟件操作的知識給代碼化，同時利用 Kubernetes 強大的抽象來管理大規模的軟件應用。目前CoreOS官方提供了幾種Operator的實現，其中就包括我們今天的主角：Prometheus Operator，Operator的核心實現就是基於 Kubernetes 的以下兩個概念：

  • 資源：對象的狀態定義
  • 控制器：觀測、分析和行動，以調節資源的分布

當前CoreOS提供的以下四種Operator：

  • etcd：創建etcd集群
  • Rook：雲原生環境下的文件、塊、對象存儲服務
  • Prometheus：創建Prometheus監控實例
  • Tectonic：部署Kubernetes集群

Kube-Prometheus（Operator） 框架

 

注：CRD是對 Kubernetes API 的擴展，Kubernetes 中的每個資源都是一個 API 對象的集合，例如我們在 YAML文件里定義的那些spec都是對 Kubernetes 中的資源對象的定義，所有的自定義資源可以跟 Kubernetes 中內建的資源一樣使用 kubectl 操作。

 

 

接下來我們將使用Operator創建Prometheus。

安裝

我們這里直接通過 Prometheus-Operator 的源碼來進行安裝，當然也可以用 Helm 來進行一鍵安裝，我們采用源碼安裝可以去了解更多的實現細節。首頁將源碼 Clone 下來：

git clone https://github.com/coreos/prometheus-operator
cd prometheus-operator/contrib/kube-prometheus/manifests

進入到 manifests 目錄下面，這個目錄下面包含我們所有的資源清單文件，直接在該文件夾下面執行創建資源命令即可：

kubectl apply -f .

部署完成后，會創建一個名為monitoring的 namespace，所以資源對象對將部署在改命名空間下面，此外 Operator 會自動創建4個 CRD 資源對象：

 kubectl get crd |grep coreos
alertmanagers.monitoring.coreos.com           2019-03-18T02:43:57Z
prometheuses.monitoring.coreos.com            2019-03-18T02:43:58Z
prometheusrules.monitoring.coreos.com         2019-03-18T02:43:58Z
servicemonitors.monitoring.coreos.com         2019-03-18T02:43:58Z

可以在 monitoring 命名空間下面查看所有的 Pod，其中 alertmanager 和 prometheus 是用 StatefulSet 控制器管理的，其中還有一個比較核心的 prometheus-operator 的 Pod，用來控制其他資源對象和監聽對象變化的：

kubectl get pods -n monitoring
NAME                                   READY     STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2       Running   0          37m
alertmanager-main-1                    2/2       Running   0          34m
alertmanager-main-2                    2/2       Running   0          33m
grafana-7489c49998-pkl8w               1/1       Running   0          40m
kube-state-metrics-d6cf6c7b5-7dwpg     4/4       Running   0          27m
node-exporter-dlp25                    2/2       Running   0          40m
node-exporter-fghlp                    2/2       Running   0          40m
node-exporter-mxwdm                    2/2       Running   0          40m
node-exporter-r9v92                    2/2       Running   0          40m
prometheus-adapter-84cd9c96c9-n92n4    1/1       Running   0          40m
prometheus-k8s-0                       3/3       Running   1          37m
prometheus-k8s-1                       3/3       Running   1          37m
prometheus-operator-7b74946bd6-vmbcj   1/1       Running   0          40m

查看創建的 Service:

  
kubectl get svc -n monitoring
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
alertmanager-main       ClusterIP   10.110.43.207   <none>        9093/TCP            40m
alertmanager-operated   ClusterIP   None            <none>        9093/TCP,6783/TCP   38m
grafana                 ClusterIP   10.109.160.0    <none>        3000/TCP            40m
kube-state-metrics      ClusterIP   None            <none>        8443/TCP,9443/TCP   40m
node-exporter           ClusterIP   None            <none>        9100/TCP            40m
prometheus-adapter      ClusterIP   10.105.174.21   <none>        443/TCP             40m
prometheus-k8s          ClusterIP   10.97.195.143   <none>        9090/TCP            40m
prometheus-operated     ClusterIP   None            <none>        9090/TCP            38m
prometheus-operator     ClusterIP   None            <none>        8080/TCP            40m
 

可以看到上面針對 grafana 和 prometheus 都創建了一個類型為 ClusterIP 的 Service，當然如果我們想要在外網訪問這兩個服務的話可以通過創建對應的 Ingress 對象或者使用 NodePort 類型的 Service，我們這里為了簡單，直接使用 NodePort 類型的服務即可，編輯 grafana 和 prometheus-k8s 這兩個 Service，將服務類型更改為 NodePort:

  
kubectl edit svc grafana -n monitoring
kubectl edit svc prometheus-k8s -n monitoring
kubectl get svc -n monitoring
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
.....
grafana                 NodePort    10.109.160.0    <none>        3000:31740/TCP      42m
prometheus-k8s          NodePort    10.97.195.143   <none>        9090:31310/TCP      42m
 

更改完成后，我們就可以通過去訪問上面的兩個服務了，比如查看 prometheus 的 targets 頁面：

我們可以看到大部分的配置都是正常的，只有兩三個沒有管理到對應的監控目標，比如 kube-controller-manager 和 kube-scheduler 這兩個系統組件，這就和 ServiceMonitor 的定義有關系了，我們先來查看下 kube-scheduler 組件對應的 ServiceMonitor 資源的定義：(prometheus-serviceMonitorKubeScheduler.yaml)

 

Kube-Prometheus配置

我們可以看到大部分的配置都是正常的，只有兩個沒有管理到對應的監控目標，比如 kube-controller-manager 和 kube-scheduler 這兩個系統組件，這就和 ServiceMonitor 的定義有關系了，我們先來查看下 kube-scheduler 組件對應的 ServiceMonitor 資源的定義（prometheus-serviceMonitorKubeScheduler.yaml）：

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kube-scheduler
  name: kube-scheduler
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s  # 每30s獲取一次信息
    port: http-metrics  # 對應service的端口名
  jobLabel: k8s-app
  namespaceSelector:  # 表示去匹配某一命名空間中的service，如果想從所有的namespace中匹配用any: true
    matchNames:
    - kube-system
  selector:  # 匹配的 Service 的labels，如果使用mathLabels，則下面的所有標簽都匹配時才會匹配該service，如果使用matchExpressions，則至少匹配一個標簽的service都會被選擇
    matchLabels:
      k8s-app: kube-scheduler

 

上面是一個典型的 ServiceMonitor 資源文件的聲明方式，上面我們通過selector.matchLabels在 kube-system 這個命名空間下面匹配具有k8s-app=kube-scheduler這樣的 Service，但是我們系統中根本就沒有對應的 Service，所以我們需要手動創建一個 Service（prometheus-kubeSchedulerService.yaml）：

apiVersion: v1
kind: Service
metadata:
  namespace: kube-system
  name: kube-scheduler
  labels:
    k8s-app: kube-scheduler
spec:
  #selector:
  #  component: kube-scheduler
  ports:
  - name: http-metrics
    port: 10251
    targetPort: 10251
    protocol: TCP

我們采用相同的方法創建Kube-Controller-Manager的Service（prometheus-KubeControllerManagerService.yaml）：

apiVersion: v1
kind: Service
metadata:
  namespace: kube-system
  name: kube-controller-manager
  labels:
    k8s-app: kube-controller-manager
spec:
  #selector:
  #  component: kube-controller-manager
  ports:
  - name: https-metrics
    port: 10252
    targetPort: 10252
    protocol: TCP

 

創建完成之后我們通過 kubectl get svc -n kube-system 命令看到對應SVC資源已經生成了：

 

 但是此時Prometheus面板的 targets 還是沒有任何顯示，我們通過 kubectl get ep -n kube-system 命令去查看一下 kube-controller-manager 和 kube-scheduler 的endpoints發現並沒有任何endpoints，這里我們需要手動來添加endpoints。我們定義兩個endpoints資源文件，如下：

 

 編輯 prometheus-kubeSchedulerServiceEnpoints.yaml

[root@master1 manifests]# cat prometheus-kubeSchedulerServiceEnpoints.yaml 
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: kube-scheduler
  name: kube-scheduler
  namespace: kube-system
subsets:
  - addresses:
      - ip: 192.168.200.11
      - ip: 192.168.200.12
      - ip: 192.168.200.13
    ports:
    - name: http-metrics
      port: 10251
      protocol: TCP

 

編輯 prometheus-KubeControllerManagerServiceEnpoints.yaml

[root@master1 manifests]# cat prometheus-KubeControllerManagerServiceEnpoints.yaml 
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: kube-controller-manager
  name: kube-controller-manager
  namespace: kube-system
subsets:
  - addresses:
      - ip: 192.168.200.11
      - ip: 192.168.200.12
      - ip: 192.168.200.13
    ports:
    - name: https-metrics
      port: 10252
      protocol: TCP

 

定義完成之后我們直接去創建上面的兩個endpoints資源，然后我們繼續查看 kube-controller-manager 和 kube-scheduler 的endpoints資源狀態：

 

我們已經成功為 kube-controller-manager 和 kube-scheduler 這兩個SVC資源添加了 Endpoints 資源。但是此時我們去刷新Prometheus的 target 可以看到 monitoring/kube-scheduler/0 數據正常、但是 monitoring/kube-controller-manager/0 狀態顯示為DOWN。錯誤信息為：Server Returned HTTP Status 400 Bad Request。仔細觀察我們會發現、Endpoints 信息為：http://172.16.200.11:10252/metrics，我們在前面部署 kube-controller-manager 明明用的是HTTPS、但是為什么這里就變成 HTTP 了呢？

快速部署Kubernetes監控系統Kube-Prometheus

 

我們去查看 prometheus-serviceMonitorKubeControllerManager.yaml 文件發現 kube-controller-manager 的 ServiceMonitor kind 端口寫的是 http-metrics。我們需要把Port修改為https-metrics，並添加token信息，這里我們可以 insecureSkipVerify: true 參數禁止掉證書驗證，詳細信息如下：

 

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kube-controller-manager
  name: kube-controller-manager
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token # 添加token
    interval: 30s
......
    - action: drop
      regex: etcd_(debugging|disk|request|server).*
      sourceLabels:
      - __name__
    port: https-metrics # 把默認http-metrics修改為https-metrics
    scheme: https # 添加https
    tlsConfig:
      insecureSkipVerify: true # 禁止驗證證書信息

 

我們直接通過 kubectl apply -f prometheus-serviceMonitorKubeControllerManager.yaml 更新上面的資源文件，然后稍等30秒重新刷新 prometheus 頁面。我們可以看到 monitoring/kube-controller-manager/0 這個  target  狀態已經變成UP了。這時我們去看看 Grafana 里面是否有對應數據，當然你也可以通過 PromSQL 來查詢驗證：

 

 

自定義 Kube-Prometheus 監控項

前面我們講解了 如何快速部署 Kube-Prometheus 監控系統，下面我們繼續介紹如何在 Kube-Prometheus 中添加一個自定義的監控項。除了 Kubernetes 集群中的一些資源對象、節點以及組件需要監控，有的時候我們可能還需要根據實際的業務需求去添加自定義的監控項，添加一個自定義監控的步驟也是非常簡單的。

  • 第一步：建立一個 ServiceMonitor 對象，用於 Prometheus 添加監控項；
  • 第二步：為 ServiceMonitor 對象關聯 metrics 數據接口的一個 Service 對象；
  • 第三步：確保 Service 對象可以正確獲取到 metrics 數據。

接下來我們就來看看如何添加 Etcd 集群的監控。無論是 Kubernetes 集群外的還是使用 Kubeadm 安裝在集群內部的 Etcd 集群，我們這里都將其視作集群外的獨立集群，因為對於二者的使用方法沒什么特殊之處。

 

3.1、獲取ETCD證書

對於 Etcd 集群，在搭建的時候我們就采用了https證書認證的方式，所以這里如果想用 Kube-Prometheus 訪問到 Etcd 集群的監控數據，就需要添加證書。我們可以通過 systemctl status etcd 查看證書路徑：

systemctl status etcd

快速部署Kubernetes監控系統Kube-Prometheus

接下來我們通過啟動文件，查看ETCD證書路徑：

cat /etc/systemd/system/etcd.service

快速部署Kubernetes監控系統Kube-Prometheus

注：這里我查閱了很多資料、發現大部分都是基於 Kubeadm 的監控；而這里我是針對 Kubernetes 二進制搭建的監控。基於 Kubeadm 這里就不再詳細描述了、請大家自行百度。
接下來我們需要創建一個 secret，讓 Kube-Prometheus pod 節點掛載：

kubectl create secret generic etcd-ssl --from-file=/etc/kubernetes/cert/ca.pem --from-file=/etc/etcd/cert/etcd.pem --from-file=/etc/etcd/cert/etcd-key.pem -n monitoring

 

創建完完成以后我們可以通過下面的命令來檢查一下：

kubectl describe secrets -n monitoring etcd-ssl

快速部署Kubernetes監控系統Kube-Prometheus

我們可以看到證書已經創建成功，然后我們將 etcd-ssl secret 對象配置到 Kube-Prometheus 資源對象中。這里我們可以通過 edit 命令直接編輯 Kube-Prometheus 或者 修改 prometheus-prometheus.yaml  文件，然后更新：

# 通過 edit 命令直接編輯
kubectl edit prometheus k8s -n monitoring

# 修改 prometheus-prometheus.yaml  文件
vim kube-prometheus-master/manifests/prometheus-prometheus.yaml

 

這里我們直接修改 prometheus-prometheus.yaml  資源文件，並添加內容如下：

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
......
  replicas: 2
  secrets:
  - etcd-ssl#  添加secret名稱
......

 

修改完成之后我們直接更新上面的資源文件，然后我們就可以在 Kube-Prometheus pod 中查看到對象的目錄了：

# 更新資源文件
kubectl apply -f prometheus-prometheus.yaml

# 查看pod狀態
kubectl get pod -n monitoring

# 進入prometheus-k8s-0 Pod中
kubectl exec -it -n monitoring prometheus-k8s-0 /bin/sh

# 查看證書
ls /etc/prometheus/secrets/etcd-ssl/

快速部署Kubernetes監控系統Kube-Prometheus

 

3.2、創建ServiceMonitor

前面Kube-Prometheus已經掛載了ETCD證書文件，下面我們就可以直接來創建ServiceMonitor了：

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd-k8s
  namespace: monitoring
  labels:
    k8s-app: etcd-k8s
spec:
  jobLabel: k8s-app
  endpoints:
  - port: port
    interval: 30s
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-ssl/ca.pem #證書路徑 (Pod里的路徑)
      certFile: /etc/prometheus/secrets/etcd-ssl/etcd.pem
      keyFile: /etc/prometheus/secrets/etcd-ssl/etcd-key.pem
      insecureSkipVerify: true
  selector:
    matchLabels:
      k8s-app: etcd
  namespaceSelector:
    matchNames:
    - kube-system

上面這個文件我們匹配 Kube-system 這個命名空間下面具有 k8s-app=etcd 這個label標簽的Service，job label用於檢索job任務名稱的標簽。由於證書 serverName 和 etcd 中簽發的證書可能不匹配，所以添加了 insecureSkipVerify=true 將不再對服務端的證書進行校驗。接下來我們直接創建這個ServiceMonitor：

# 創建資源文件
kubectl apply -f prometheus-serviceMonitorEtcd.yaml

# 查看servicemonitors資源
kubectl get servicemonitors -n monitoring |grep etcd

快速部署Kubernetes監控系統Kube-Prometheus

 

ServiceMonitor資源創建完成以后、我們等30秒然后刷新Prometheus Targets數據。此時我們發現新增了一條 monitoring/etcd-k8s/0 (0/0 up) ，但是沒有任何數據。那是因為我們雖然創建了ServiceMonitor，但是還沒有關聯對應的Service對象，所以需要創建一個service對象。

快速部署Kubernetes監控系統Kube-Prometheus

我們需要定義一個 Service 對象和一個 Endpoints，對應資源文件如下。

prometheus-EtcdService.yaml：

apiVersion: v1
kind: Service
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: port
    port: 2379
    protocol: TCP

 

prometheus-EtcdServiceEnpoints.yaml

apiVersion: v1
kind: Endpoints
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
subsets:
- addresses:
  - ip: 172.16.200.11     #etcd節點名稱
    #nodeName: k8s-01     #kubelet名稱 (kubectl get node)顯示的名稱
  - ip: 172.16.200.12
    #nodeName: k8s-02
  - ip: 172.16.200.13
    #nodeName: k8s-03
  ports:
  - name: port
    port: 2379
    protocol: TCP

 

然后我們直接創建上面的 Service 對象和 Endpoints 對象：

kubectl apply -f  prometheus-EtcdService.yaml
kubectl apply -f  prometheus-EtcdServiceEnpoints.yaml

# 查看ETCD狀態
kubectl describe svc -n kube-system etcd-k8s

快速部署Kubernetes監控系統Kube-Prometheus

 

創建完成后，稍等一會我們可以去Prometheus 里面查看targets，便會出現etcd監控信息：

快速部署Kubernetes監控系統Kube-Prometheus

注：如果提示ip:2379 connection refused，首先檢查本地Telnet 是否正常，在檢查etcd配置文件是否是監聽0.0.0.0:2379。

 

數據采集完成后，接下來可以在grafana中導入dashboard。這里我們可以導入 ：https://grafana.com/grafana/dashboards/3070；還可以導入中文版ETCD集群插件：https://grafana.com/grafana/dashboards/9733；導入過程這里就不再詳細描述了、不會的小伙伴請自行百度：

快速部署Kubernetes監控系統Kube-Prometheus

 

4、Kube-Prometheus數據持久化

前面我們需改完Prometheus的相關配置后，重啟了 Prometheus 的 Pod，如果我們仔細觀察的話會發現我們之前采集的數據已經沒有了，這是因為我們通過 Prometheus 這個 CRD 創建的 Prometheus 並沒有做數據的持久化，我們可以直接查看生成的 Prometheus Pod 的掛載情況就清楚了：

kubectl get pod prometheus-k8s-0 -n monitoring -o yaml

快速部署Kubernetes監控系統Kube-Prometheus

快速部署Kubernetes監控系統Kube-Prometheus

 

從上圖我們可以看到 Prometheus 的數據目錄 /prometheus 實際上是通過 emptyDir 進行掛載的，我們知道 emptyDir 掛載的數據的生命周期和 Pod 生命周期一致的，所以如果 Pod 掛掉了，數據也就丟失了，這也就是為什么我們重建 Pod 后之前的數據就沒有了的原因，對應線上的監控數據肯定需要做數據的持久化的，同樣的 prometheus 這個 CRD 資源也為我們提供了數據持久化的配置方法，由於我們的 Prometheus 最終是通過 Statefulset 控制器進行部署的，所以我們這里需要通過 storageclass 來做數據持久化，首先創建一個 StorageClass 對象（prometheus-storageclass.yaml）：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: prometheus-data-db
provisioner: fuseim.pri/ifs

 

這里我們聲明一個 StorageClass 對象，其中 provisioner=fuseim.pri/ifs，則是因為我們集群中使用的是 nfs 作為存儲后端，而前面我們創建的 nfs-client-provisioner 中指定的 PROVISIONER_NAME 就為 fuseim.pri/ifs，這個名字不能隨便更改。然后我們直接創建這個資源：

kubectl apply -f prometheus-storageclass.yaml
kubectl get storageclass

快速部署Kubernetes監控系統Kube-Prometheus

 

然后在 prometheus 的 CRD 資源對象中添加如下配置：

storage: volumeClaimTemplate: spec: storageClassName: prometheus-data-db resources: requests: storage: 100Gi

快速部署Kubernetes監控系統Kube-Prometheus

注意這里的 storageClassName 名字為上面我們創建的 StorageClass 對象名稱，然后更新 prometheus 這個 CRD 資源。更新完成后會自動生成兩個 PVC 和 PV 資源對象：

# 更新資源文件
kubectl apply -f prometheus-prometheus.yaml
# 查看PVC
kubectl get pvc -n monitoring
# 查看PV
kubectl get pv

快速部署Kubernetes監控系統Kube-Prometheus

現在我們再去看 Prometheus Pod 的數據目錄就可以看到是關聯到一個 PVC 對象上了。現在即使我們的 Pod 掛掉了，數據也不會丟失了。

 

5、Kube-Prometheus數據持久時間

前面說了prometheus operator持久化的問題，但是還有一個問題很多人都忽略了，那就是prometheus operator數據保留天數，根據官方文檔的說明，默認prometheus operator數據存儲的時間為1d，這個時候無論你prometheus operator如何進行持久化，都沒有作用，因為數據只保留了1天，那么你是無法看到更多天數的數據。

官方文檔可以配置的說明：https://github.com/coreos/prometheus-operator/blob/0e6ed120261f101e6f0dc9581de025f136508ada/Documentation/prometheus.md

快速部署Kubernetes監控系統Kube-Prometheus

實際上我們修改 Kube-Prometheus 時間是通過 retention 參數進行修改，上面也提示了在prometheus.spec下填寫。這里我們直接修改 prometheus-prometheus.yaml 文件，並添加下面的參數：

快速部署Kubernetes監控系統Kube-Prometheus

注：如果已經安裝了可以直接修改 prometheus-prometheus.yaml 然后通過kubectl apply -f 刷新即可，修改完成以后記得檢查Pod運行狀態是否正常。

接下來可以訪問grafana或者prometheus ui進行檢查 （我這里修改完畢后等待2天，檢查數據是否正常）。

修改前

快速部署Kubernetes監控系統Kube-Prometheus

 

修改后

快速部署Kubernetes監控系統Kube-Prometheus快速部署Kubernetes監控系統Kube-Prometheus

6、Grafana數據持久化

前面我們介紹了關於prometheus的數據持久化、但是沒有介紹如何針對Grafana做數據持久化；如果Grafana不做數據持久化、那么服務重啟以后，Grafana里面配置的Dashboard、賬號密碼等信息將會丟失；所以Grafana做數據持久化也是很有必要的。

原始的數據是以 emptyDir 形式存放在pod里面，生命周期與pod相同；出現問題時，容器重啟，在Grafana里面設置的數據就全部消失了。

 volumeMounts:  - mountPath: /var/lib/grafana  name: grafana-storage  readOnly: false ... volumes: - emptyDir: {}  name: grafana-storage

 

從上圖我們可以看出Grafana將dashboard、插件這些數據保存在/var/lib/grafana這個目錄下面。做持久化的話，就需要對這個目錄進行volume掛載聲明。

我們把emptyDir修改為pvc方式：

volumes: - name: grafana-storage  persistentVolumeClaim:  claimName: grafana

 

如果要使用一個 pvc 對象來持久化數據，我們就需要添加一個可用的 pv 供 pvc 綁定使用，grafana-volume.yaml內容如下：

apiVersion: v1 kind: PersistentVolume metadata:  name: grafana spec:  capacity:  storage: 10Gi  accessModes:  - ReadWriteOnce  persistentVolumeReclaimPolicy: Recycle  nfs:  server: 172.16.200.10  path: /mnt/lv/k8s --- apiVersion: v1 kind: PersistentVolumeClaim metadata:  name: grafana  namespace: monitoring spec:  accessModes:  - ReadWriteOnce  resources:  requests:  storage: 10Gi

 

然后我們直接創建上面的PV和PVC、更新 grafana-deployment.yaml 文件即可：

# kubectl apply -f grafana-volume.yaml 
# kubectl apply -f grafana-deployment.yaml

 

創建完成以后我們查看Pod狀態，我們發現Pod狀態一直是 CrashLoopBackOff  沒有正常啟動，我們再看一下這個 Pod 的日志，錯誤信息如下：

mkdir: cannot create directory '/var/lib/grafana/plugins': Permission denied

 

這個錯誤是 Grafana 5.1版本以后才會出現的。錯誤的原因很明顯，就是 /var/lib/grafana 目錄的權限不夠。在 `` 中有這樣一個屬性：

securityContext: 
  runAsNonRoot: true 
  runAsUser: 65534

 

我們查看一下65534是哪個用戶：

cat /etc/passwd | grep 65534

所以，我們只需要把 /mnt/lv/k8s 目錄的用戶改為 nfsnobody 就可以了。當然把屬性改為 777 也沒問題：

chown nfsnobody /mnt/lv/k8s

把剛才出錯的那個 Pod 刪除，新的 Grafana Pod 就成功啟動了。然后就可以添加 Dashboard 了，現在Pod 重建也不會丟失數據了。

 

7、Kube-Prometheus服務發現

前面我們在 Kube-Prometheus 下面自定義一個監控項，以及自定義報警規則的使用。那么我們還能夠直接使用前面課程中的自動發現功能嗎？如果在我們的 Kubernetes 集群中有了很多的 Service/Pod，那么我們都需要一個一個的去建立一個對應的 ServiceMonitor 對象來進行監控嗎？這樣豈不是又變得麻煩起來了？

為解決上面的問題，Kube-Prometheus 為我們提供了一個額外的抓取配置的來解決這個問題，我們可以通過添加額外的配置來進行服務發現進行自動監控。和前面自定義的方式一樣，我們可以在 Kube-Prometheus 當中去自動發現並監控具有prometheus.io/scrape=true 這個 annotations 的 Service，之前我們定義的 Prometheus 的配置如下：

- job_name: 'kubernetes-endpoints'  kubernetes_sd_configs:  - role: endpoints  relabel_configs:  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]  action: keep  regex: true  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]  action: replace  target_label: __scheme__  regex: (https?)  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]  action: replace  target_label: __metrics_path__  regex: (.+)  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]  action: replace  target_label: __address__  regex: ([^:]+)(?::\d+)?;(\d+)  replacement: $1:$2  - action: labelmap  regex: __meta_kubernetes_service_label_(.+)  - source_labels: [__meta_kubernetes_namespace]  action: replace  target_label: kubernetes_namespace  - source_labels: [__meta_kubernetes_service_name]  action: replace  target_label: kubernetes_name  - source_labels: [__meta_kubernetes_pod_name]  action: replace  target_label: kubernetes_pod_name

 

要想自動發現集群中的 Service，就需要我們在 Service 的 annotation 區域添加 prometheus.io/scrape=true 的聲明，將上面文件直接保存為 prometheus-additional.yaml，然后通過這個文件創建一個對應的 Secret 對象：

kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring

 

然后我們需要在聲明 prometheus 的資源對象文件中通過 additionalScrapeConfigs 屬性添加上這個額外的配置：(prometheus-prometheus.yaml)

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  retention: 365d
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: prometheus-data-db
        resources:
          requests:
            storage: 100Gi
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 2
  secrets:
  - etcd-ssl
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.11.0
  # 添加額外配置內容
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yam

 

添加完成后，直接更新 prometheus 這個 CRD 資源對象即可：

kubectl apply -f prometheus-prometheus.yaml

 

隔一小會兒，可以前往 Prometheus 的 Dashboard 中查看配置已經生效了：

快速部署Kubernetes監控系統Kube-Prometheus

 

但是我們切換到 targets 頁面下面卻並沒有發現對應的監控任務，查看 Prometheus 的 Pod 日志：

kubectl logs -f prometheus-k8s-0 prometheus -n monitoring | grep cluster

快速部署Kubernetes監控系統Kube-Prometheus

 

可以看到有很多錯誤日志出現，都是 xxx is forbidden，這說明是 RBAC 權限的問題，通過 prometheus 資源對象的配置可以知道 Prometheus 綁定了一個名為 prometheus-k8s 的 ServiceAccount 對象，而這個對象綁定的是一個名為 prometheus-k8s 的 ClusterRole：（prometheus-clusterRole.yaml）

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

 

上面的權限規則中我們可以看到明顯沒有對 Service 或者 Pod 的 list 權限，所以報錯了，要解決這個問題，我們只需要添加上需要的權限即可：

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata:  name: prometheus-k8s rules: - apiGroups:  - ""  resources:  - nodes  - services  - endpoints  - pods  - nodes/proxy  verbs:  - get  - list  - watch - apiGroups:  - ""  resources:  - configmaps  - nodes/metrics  verbs:  - get - nonResourceURLs:  - /metrics  verbs:  - get

 

更新上面的 ClusterRole 這個資源對象，然后重建 Prometheus 的所有 Pod，正常就可以看到 targets 頁面下面有 kubernetes-endpoints 這個監控任務了：

快速部署Kubernetes監控系統Kube-Prometheus

從上圖我們可以看到、上面抓取的目標是因為 Service 中有 prometheus.io/scrape=true 這個 annotation。

 

 

 

 

 

 

 

 

 

 

 

 
 
 
 
 
 
 
 
 
 
 
 
超炫grafana展示 
{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "Prometheus",
      "description": "",
      "type": "datasource",
      "pluginId": "prometheus",
      "pluginName": "Prometheus"
    }
  ],
  "__requires": [
    {
      "type": "panel",
      "id": "bargauge",
      "name": "Bar Gauge",
      "version": ""
    },
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "6.7.3"
    },
    {
      "type": "panel",
      "id": "graph",
      "name": "Graph",
      "version": ""
    },
    {
      "type": "datasource",
      "id": "prometheus",
      "name": "Prometheus",
      "version": "1.0.0"
    },
    {
      "type": "panel",
      "id": "singlestat",
      "name": "Singlestat",
      "version": ""
    },
    {
      "type": "panel",
      "id": "table",
      "name": "Table",
      "version": ""
    }
  ],
  "annotations": {
    "list": [
      {
        "$$hashKey": "object:5598",
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "【中文版本】2020.06.28更新，增加整體資源展示！支持 Grafana6&7，Node Exporter v0.16及以上的版本，優化重要指標展示。包含整體資源展示與資源明細圖表：CPU 內存 磁盤 IO 網絡等監控指標。https://github.com/starsliao/Prometheus",
  "editable": true,
  "gnetId": 8919,
  "graphTooltip": 0,
  "id": null,
  "iteration": 1593281500982,
  "links": [
    {
      "icon": "external link",
      "tags": [],
      "targetBlank": true,
      "title": "更新node_exporter",
      "tooltip": "",
      "type": "link",
      "url": "https://github.com/prometheus/node_exporter/releases"
    },
    {
      "icon": "external link",
      "tags": [],
      "targetBlank": true,
      "title": "更新當前儀表板",
      "tooltip": "",
      "type": "link",
      "url": "https://grafana.com/dashboards/8919"
    },
    {
      "icon": "external link",
      "tags": [],
      "targetBlank": true,
      "title": "StarsL.cn",
      "tooltip": "",
      "type": "link",
      "url": "https://starsl.cn"
    },
    {
      "asDropdown": true,
      "icon": "external link",
      "tags": [],
      "targetBlank": true,
      "title": "",
      "type": "dashboards"
    }
  ],
  "panels": [
    {
      "collapsed": false,
      "datasource": "${DS_PROMETHEUS}",
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 187,
      "panels": [],
      "title": "資源總覽（關聯JOB項）當前選中主機：【$show_hostname】實例：$node",
      "type": "row"
    },
    {
      "columns": [],
      "datasource": "${DS_PROMETHEUS}",
      "description": "分區使用率、磁盤讀取、磁盤寫入、下載帶寬、上傳帶寬，如果有多個網卡或者多個分區，是采集的使用率最高的網卡或者分區的數值。",
      "fontSize": "100%",
      "gridPos": {
        "h": 12,
        "w": 24,
        "x": 0,
        "y": 1
      },
      "id": 185,
      "pageSize": 10,
      "showHeader": true,
      "sort": {
        "col": 5,
        "desc": false
      },
      "styles": [
        {
          "$$hashKey": "object:5955",
          "alias": "主機名",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 1,
          "link": false,
          "linkTooltip": "",
          "linkUrl": "",
          "mappingType": 1,
          "pattern": "nodename",
          "thresholds": [],
          "type": "string",
          "unit": "bytes"
        },
        {
          "$$hashKey": "object:5956",
          "alias": "IP（鏈接到明細）",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "link": true,
          "linkTargetBlank": false,
          "linkTooltip": "瀏覽主機明細",
          "linkUrl": "/d/9CWBz0bik/node-exporter?orgId=1&var-job=${job}&var-hostname=All&var-node=${__cell}&var-device=All",
          "mappingType": 1,
          "pattern": "instance",
          "thresholds": [],
          "type": "number",
          "unit": "short"
        },
        {
          "$$hashKey": "object:5957",
          "alias": "內存",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "link": false,
          "mappingType": 1,
          "pattern": "Value #B",
          "thresholds": [],
          "type": "number",
          "unit": "bytes"
        },
        {
          "$$hashKey": "object:5958",
          "alias": "CPU核",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": null,
          "mappingType": 1,
          "pattern": "Value #C",
          "thresholds": [],
          "type": "number",
          "unit": "short"
        },
        {
          "$$hashKey": "object:5959",
          "alias": " 運行時間",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "Value #D",
          "thresholds": [],
          "type": "number",
          "unit": "s"
        },
        {
          "$$hashKey": "object:5960",
          "alias": "分區使用率*",
          "align": "auto",
          "colorMode": "cell",
          "colors": [
            "rgba(50, 172, 45, 0.97)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(245, 54, 54, 0.9)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "Value #E",
          "thresholds": [
            "70",
            "85"
          ],
          "type": "number",
          "unit": "percent"
        },
        {
          "$$hashKey": "object:5961",
          "alias": "CPU使用率",
          "align": "auto",
          "colorMode": "cell",
          "colors": [
            "rgba(50, 172, 45, 0.97)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(245, 54, 54, 0.9)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "Value #F",
          "thresholds": [
            "70",
            "85"
          ],
          "type": "number",
          "unit": "percent"
        },
        {
          "$$hashKey": "object:5962",
          "alias": "內存使用率",
          "align": "auto",
          "colorMode": "cell",
          "colors": [
            "rgba(50, 172, 45, 0.97)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(245, 54, 54, 0.9)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "Value #G",
          "thresholds": [
            "70",
            "85"
          ],
          "type": "number",
          "unit": "percent"
        },
        {
          "$$hashKey": "object:5963",
          "alias": "磁盤讀取*",
          "align": "auto",
          "colorMode": "cell",
          "colors": [
            "rgba(50, 172, 45, 0.97)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(245, 54, 54, 0.9)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "Value #H",
          "thresholds": [
            "10485760",
            "20485760"
          ],
          "type": "number",
          "unit": "Bps"
        },
        {
          "$$hashKey": "object:5964",
          "alias": "磁盤寫入*",
          "align": "auto",
          "colorMode": "cell",
          "colors": [
            "rgba(50, 172, 45, 0.97)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(245, 54, 54, 0.9)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "Value #I",
          "thresholds": [
            "10485760",
            "20485760"
          ],
          "type": "number",
          "unit": "Bps"
        },
        {
          "$$hashKey": "object:5965",
          "alias": "下載帶寬*",
          "align": "auto",
          "colorMode": "cell",
          "colors": [
            "rgba(50, 172, 45, 0.97)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(245, 54, 54, 0.9)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "Value #J",
          "thresholds": [
            "30485760",
            "104857600"
          ],
          "type": "number",
          "unit": "bps"
        },
        {
          "$$hashKey": "object:5966",
          "alias": "上傳帶寬*",
          "align": "auto",
          "colorMode": "cell",
          "colors": [
            "rgba(50, 172, 45, 0.97)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(245, 54, 54, 0.9)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "Value #K",
          "thresholds": [
            "30485760",
            "104857600"
          ],
          "type": "number",
          "unit": "bps"
        },
        {
          "$$hashKey": "object:5967",
          "alias": "5m負載",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "Value #L",
          "thresholds": [],
          "type": "number",
          "unit": "short"
        },
        {
          "$$hashKey": "object:5968",
          "alias": "",
          "align": "right",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "decimals": 2,
          "pattern": "/.*/",
          "thresholds": [],
          "type": "hidden",
          "unit": "short"
        }
      ],
      "targets": [
        {
          "expr": "node_uname_info{job=~\"$job\"} - 0",
          "format": "table",
          "instant": true,
          "interval": "",
          "legendFormat": "主機名",
          "refId": "A"
        },
        {
          "expr": "sum(time() - node_boot_time_seconds{job=~\"$job\"})by(instance)",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "運行時間",
          "refId": "D"
        },
        {
          "expr": "node_memory_MemTotal_bytes{job=~\"$job\"} - 0",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "總內存",
          "refId": "B"
        },
        {
          "expr": "count(node_cpu_seconds_total{job=~\"$job\",mode='system'}) by (instance)",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "總核數",
          "refId": "C"
        },
        {
          "expr": "node_load5{job=~\"$job\"}",
          "format": "table",
          "instant": true,
          "interval": "",
          "legendFormat": "5分鍾負載",
          "refId": "L"
        },
        {
          "expr": "(1 - avg(irate(node_cpu_seconds_total{job=~\"$job\",mode=\"idle\"}[5m])) by (instance)) * 100",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "CPU使用率",
          "refId": "F"
        },
        {
          "expr": "(1 - (node_memory_MemAvailable_bytes{job=~\"$job\"} / (node_memory_MemTotal_bytes{job=~\"$job\"})))* 100",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "內存使用率",
          "refId": "G"
        },
        {
          "expr": "max((node_filesystem_size_bytes{job=~\"$job\",fstype=~\"ext.?|xfs\"}-node_filesystem_free_bytes{job=~\"$job\",fstype=~\"ext.?|xfs\"}) *100/(node_filesystem_avail_bytes {job=~\"$job\",fstype=~\"ext.?|xfs\"}+(node_filesystem_size_bytes{job=~\"$job\",fstype=~\"ext.?|xfs\"}-node_filesystem_free_bytes{job=~\"$job\",fstype=~\"ext.?|xfs\"})))by(instance)",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "分區使用率",
          "refId": "E"
        },
        {
          "expr": "max(irate(node_disk_read_bytes_total{job=~\"$job\"}[5m])) by (instance)",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "最大讀取",
          "refId": "H"
        },
        {
          "expr": "max(irate(node_disk_written_bytes_total{job=~\"$job\"}[5m])) by (instance)",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "最大寫入",
          "refId": "I"
        },
        {
          "expr": "max(irate(node_network_receive_bytes_total{job=~\"$job\"}[5m])*8) by (instance)",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "下載帶寬",
          "refId": "J"
        },
        {
          "expr": "max(irate(node_network_transmit_bytes_total{job=~\"$job\"}[5m])*8) by (instance)",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "上傳帶寬",
          "refId": "K"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "服務器資源總覽表(每頁10行)",
      "transform": "table",
      "type": "table"
    },
    {
      "aliasColors": {
        "192.168.200.241:9100_Total": "dark-red",
        "Idle - Waiting for something to happen": "#052B51",
        "guest": "#9AC48A",
        "idle": "#052B51",
        "iowait": "#EAB839",
        "irq": "#BF1B00",
        "nice": "#C15C17",
        "sdb_每秒I/O操作%": "#d683ce",
        "softirq": "#E24D42",
        "steal": "#FCE2DE",
        "system": "#508642",
        "user": "#5195CE",
        "磁盤花費在I/O操作占比": "#ba43a9"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": null,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 0,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 0,
        "y": 13
      },
      "hiddenSeries": false,
      "id": 191,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": false,
        "min": false,
        "rightSide": false,
        "show": true,
        "sideWidth": null,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "maxPerRow": 6,
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "repeat": null,
      "seriesOverrides": [
        {
          "alias": "總平均使用率",
          "lines": false,
          "pointradius": 1,
          "points": true,
          "yaxis": 2
        },
        {
          "alias": "總核數",
          "color": "#C4162A"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "count(node_cpu_seconds_total{job=~\"$job\", mode='system'})",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "總核數",
          "refId": "B",
          "step": 240
        },
        {
          "expr": "sum(node_load5{job=~\"$job\"})",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "總5分鍾負載",
          "refId": "A",
          "step": 240
        },
        {
          "expr": "avg(1 - avg(irate(node_cpu_seconds_total{job=~\"$job\",mode=\"idle\"}[5m])) by (instance)) * 100",
          "format": "time_series",
          "hide": false,
          "interval": "30m",
          "intervalFactor": 1,
          "legendFormat": "總平均使用率",
          "refId": "F",
          "step": 240
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "$job：整體總負載與整體平均CPU使用率",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "short",
          "label": "總負載",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "decimals": 0,
          "format": "percent",
          "label": "平均使用率",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "192.168.200.241:9100_總內存": "dark-red",
        "內存_Avaliable": "#6ED0E0",
        "內存_Cached": "#EF843C",
        "內存_Free": "#629E51",
        "內存_Total": "#6d1f62",
        "內存_Used": "#eab839",
        "可用": "#9ac48a",
        "總內存": "#bf1b00"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 1,
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 0,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 8,
        "y": 13
      },
      "height": "300",
      "hiddenSeries": false,
      "id": 195,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "max": false,
        "min": false,
        "rightSide": false,
        "show": true,
        "sort": "current",
        "sortDesc": false,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "總內存",
          "color": "#C4162A",
          "fill": 0
        },
        {
          "alias": "總平均使用率",
          "lines": false,
          "pointradius": 1,
          "points": true,
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(node_memory_MemTotal_bytes{job=~\"$job\"})",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "總內存",
          "refId": "A",
          "step": 4
        },
        {
          "expr": "sum(node_memory_MemTotal_bytes{job=~\"$job\"} - node_memory_MemAvailable_bytes{job=~\"$job\"})",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "總已用",
          "refId": "B",
          "step": 4
        },
        {
          "expr": "(sum(node_memory_MemTotal_bytes{job=~\"$job\"} - node_memory_MemAvailable_bytes{job=~\"$job\"}) / sum(node_memory_MemTotal_bytes{job=~\"$job\"}))*100",
          "format": "time_series",
          "hide": false,
          "interval": "30m",
          "intervalFactor": 1,
          "legendFormat": "總平均使用率",
          "refId": "H"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "$job：整體總內存與整體平均內存使用率",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "bytes",
          "label": "總內存量",
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "decimals": null,
          "format": "percent",
          "label": "平均使用率",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 1,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 0,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 16,
        "y": 13
      },
      "hiddenSeries": false,
      "id": 197,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "hideEmpty": false,
        "hideZero": false,
        "max": false,
        "min": false,
        "rightSide": false,
        "show": true,
        "sideWidth": null,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "總平均使用率",
          "lines": false,
          "pointradius": 1,
          "points": true,
          "yaxis": 2
        },
        {
          "alias": "總磁盤量",
          "color": "#C4162A"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(avg(node_filesystem_size_bytes{job=~\"$job\",fstype=~\"xfs|ext.*\"})by(device,instance))",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "總磁盤量",
          "refId": "E"
        },
        {
          "expr": "sum(avg(node_filesystem_size_bytes{job=~\"$job\",fstype=~\"xfs|ext.*\"})by(device,instance)) - sum(avg(node_filesystem_free_bytes{job=~\"$job\",fstype=~\"xfs|ext.*\"})by(device,instance))",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "總使用量",
          "refId": "C"
        },
        {
          "expr": "(sum(avg(node_filesystem_size_bytes{job=~\"$job\",fstype=~\"xfs|ext.*\"})by(device,instance)) - sum(avg(node_filesystem_free_bytes{job=~\"$job\",fstype=~\"xfs|ext.*\"})by(device,instance))) *100/(sum(avg(node_filesystem_avail_bytes{job=~\"$job\",fstype=~\"xfs|ext.*\"})by(device,instance))+(sum(avg(node_filesystem_size_bytes{job=~\"$job\",fstype=~\"xfs|ext.*\"})by(device,instance)) - sum(avg(node_filesystem_free_bytes{job=~\"$job\",fstype=~\"xfs|ext.*\"})by(device,instance))))",
          "format": "time_series",
          "instant": false,
          "interval": "30m",
          "intervalFactor": 1,
          "legendFormat": "總平均使用率",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "$job：整體總磁盤與整體平均磁盤使用率",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": 1,
          "format": "bytes",
          "label": "總磁盤量",
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "decimals": null,
          "format": "percent",
          "label": "平均使用率",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "datasource": "${DS_PROMETHEUS}",
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 21
      },
      "id": 189,
      "panels": [],
      "title": "資源明細：【$show_hostname】",
      "type": "row"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorPostfix": false,
      "colorPrefix": false,
      "colorValue": true,
      "colors": [
        "rgba(245, 54, 54, 0.9)",
        "rgba(237, 129, 40, 0.89)",
        "rgba(50, 172, 45, 0.97)"
      ],
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 0,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "format": "s",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "threshcisLabels": false,
        "threshcisMarkers": true
      },
      "gridPos": {
        "h": 2,
        "w": 2,
        "x": 0,
        "y": 22
      },
      "hideTimeOverride": true,
      "id": 15,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "null",
      "nullText": null,
      "pluginVersion": "6.4.2",
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": false
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "avg(time() - node_boot_time_seconds{instance=~\"$node\"})",
          "format": "time_series",
          "hide": false,
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A",
          "step": 40
        }
      ],
      "threshciss": "1,2",
      "thresholds": "1,3",
      "title": "運行時間",
      "type": "singlestat",
      "valueFontSize": "70%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "datasource": "${DS_PROMETHEUS}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {},
          "decimals": 2,
          "displayName": "",
          "mappings": [
            {
              "from": "",
              "id": 1,
              "operator": "",
              "text": "N/A",
              "to": "",
              "type": 1,
              "value": "0"
            }
          ],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 70
              },
              {
                "color": "#EAB839",
                "value": 90
              }
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 3,
        "x": 2,
        "y": 22
      },
      "id": 177,
      "options": {
        "displayMode": "lcd",
        "fieldOptions": {
          "calcs": [
            "last"
          ],
          "defaults": {
            "decimals": 1,
            "mappings": [
              {
                "from": "",
                "id": 1,
                "operator": "",
                "text": "N/A",
                "to": "",
                "type": 1,
                "value": "0"
              }
            ],
            "max": 100,
            "min": 0.1,
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "#EAB839",
                  "value": 70
                },
                {
                  "color": "red",
                  "value": 90
                }
              ]
            },
            "unit": "percent"
          },
          "overrides": [],
          "values": false
        },
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "mean"
          ],
          "values": false
        },
        "showUnfilled": true
      },
      "pluginVersion": "6.7.3",
      "targets": [
        {
          "expr": "100 - (avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"idle\"}[5m])) * 100)",
          "instant": true,
          "interval": "",
          "legendFormat": "總CPU使用率",
          "refId": "A"
        },
        {
          "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"iowait\"}[5m])) * 100",
          "hide": true,
          "instant": true,
          "interval": "",
          "legendFormat": "IOwait使用率",
          "refId": "C"
        },
        {
          "expr": "(1 - (node_memory_MemAvailable_bytes{instance=~\"$node\"} / (node_memory_MemTotal_bytes{instance=~\"$node\"})))* 100",
          "instant": true,
          "interval": "",
          "legendFormat": "內存使用率",
          "refId": "B"
        },
        {
          "expr": "(node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"}-node_filesystem_free_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"})*100 /(node_filesystem_avail_bytes {instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"}+(node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"}-node_filesystem_free_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"}))",
          "hide": false,
          "instant": true,
          "interval": "",
          "legendFormat": "最大分區({{mountpoint}})使用率",
          "refId": "D"
        },
        {
          "expr": "(1 - ((node_memory_SwapFree_bytes{instance=~\"$node\"} + 1)/ (node_memory_SwapTotal_bytes{instance=~\"$node\"} + 1))) * 100",
          "instant": true,
          "legendFormat": "交換分區使用率",
          "refId": "F"
        }
      ],
      "timeFrom": null,
      "timeShift": null,
      "title": "",
      "type": "bargauge"
    },
    {
      "columns": [],
      "datasource": "${DS_PROMETHEUS}",
      "description": "本看板中的：磁盤總量、使用量、可用量、使用率保持和df命令的Size、Used、Avail、Use% 列的值一致，並且Use%的值會四舍五入保留一位小數，會更加准確。\n\n注：df中Use%算法為：(size - free) * 100 / (avail + (size - free))，結果是整除則為該值，非整除則為該值+1，結果的單位是%。\n參考df命令源碼：",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fontSize": "100%",
      "gridPos": {
        "h": 6,
        "w": 10,
        "x": 5,
        "y": 22
      },
      "id": 181,
      "links": [
        {
          "targetBlank": true,
          "title": "https://github.com/coreutils/coreutils/blob/master/src/df.c",
          "url": "https://github.com/coreutils/coreutils/blob/master/src/df.c"
        }
      ],
      "pageSize": null,
      "scroll": true,
      "showHeader": true,
      "sort": {
        "col": 6,
        "desc": false
      },
      "styles": [
        {
          "$$hashKey": "object:818",
          "alias": "分區",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(50, 172, 45, 0.97)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(245, 54, 54, 0.9)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "mappingType": 1,
          "pattern": "mountpoint",
          "thresholds": [
            ""
          ],
          "type": "string",
          "unit": "bytes"
        },
        {
          "$$hashKey": "object:819",
          "alias": "可用空間",
          "align": "auto",
          "colorMode": "value",
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 1,
          "mappingType": 1,
          "pattern": "Value #A",
          "thresholds": [
            "10000000000",
            "20000000000"
          ],
          "type": "number",
          "unit": "bytes"
        },
        {
          "$$hashKey": "object:820",
          "alias": "使用率",
          "align": "auto",
          "colorMode": "cell",
          "colors": [
            "rgba(50, 172, 45, 0.97)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(245, 54, 54, 0.9)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 1,
          "mappingType": 1,
          "pattern": "Value #B",
          "thresholds": [
            "70",
            "85"
          ],
          "type": "number",
          "unit": "percent"
        },
        {
          "$$hashKey": "object:821",
          "alias": "總空間",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 0,
          "link": false,
          "mappingType": 1,
          "pattern": "Value #C",
          "thresholds": [],
          "type": "number",
          "unit": "bytes"
        },
        {
          "$$hashKey": "object:822",
          "alias": "文件系統",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "link": false,
          "mappingType": 1,
          "pattern": "fstype",
          "thresholds": [],
          "type": "string",
          "unit": "short"
        },
        {
          "$$hashKey": "object:823",
          "alias": "設備名",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "dateFormat": "YYYY-MM-DD HH:mm:ss",
          "decimals": 2,
          "link": false,
          "mappingType": 1,
          "pattern": "device",
          "preserveFormat": false,
          "sanitize": false,
          "thresholds": [],
          "type": "string",
          "unit": "short"
        },
        {
          "$$hashKey": "object:824",
          "alias": "",
          "align": "auto",
          "colorMode": null,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "decimals": 2,
          "pattern": "/.*/",
          "preserveFormat": true,
          "sanitize": false,
          "thresholds": [],
          "type": "hidden",
          "unit": "short"
        }
      ],
      "targets": [
        {
          "expr": "node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}-0",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "總量",
          "refId": "C"
        },
        {
          "expr": "node_filesystem_avail_bytes {instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}-0",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "10s",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        },
        {
          "expr": "(node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}-node_filesystem_free_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}) *100/(node_filesystem_avail_bytes {instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}+(node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}-node_filesystem_free_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}))",
          "format": "table",
          "hide": false,
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "B"
        }
      ],
      "title": "【$show_hostname】：各分區可用空間(EXT.*/XFS)",
      "transform": "table",
      "type": "table"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": true,
      "colors": [
        "rgba(50, 172, 45, 0.97)",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 2,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "format": "percent",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 2,
        "w": 2,
        "x": 15,
        "y": 22
      },
      "id": 20,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "pluginVersion": "6.4.2",
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": true,
        "lineColor": "#3274D9",
        "show": true,
        "ymax": null,
        "ymin": null
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"iowait\"}[5m])) * 100",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A",
          "step": 20
        }
      ],
      "thresholds": "20,50",
      "timeFrom": null,
      "timeShift": null,
      "title": "CPU iowait",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "avg"
    },
    {
      "aliasColors": {
        "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_cni0_in": "light-red",
        "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_cni0_in下載": "green",
        "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_cni0_out上傳": "yellow",
        "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_eth0_in下載": "purple",
        "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_eth0_out": "purple",
        "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_eth0_out上傳": "blue"
      },
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "editable": true,
      "error": false,
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "grid": {},
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 17,
        "y": 22
      },
      "hiddenSeries": false,
      "id": 183,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": false,
        "show": false,
        "sort": "current",
        "sortDesc": true,
        "total": true,
        "values": true
      },
      "lines": false,
      "linewidth": 2,
      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 1,
      "points": false,
      "renderer": "flot",
      "repeat": null,
      "seriesOverrides": [
        {
          "alias": "/.*_out上傳$/",
          "transform": "negative-Y"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "increase(node_network_receive_bytes_total{instance=~\"$node\",device=~\"$device\"}[60m])",
          "interval": "60m",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_in下載",
          "metric": "",
          "refId": "A",
          "step": 600,
          "target": ""
        },
        {
          "expr": "increase(node_network_transmit_bytes_total{instance=~\"$node\",device=~\"$device\"}[60m])",
          "hide": false,
          "interval": "60m",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_out上傳",
          "refId": "B",
          "step": 600
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "每小時流量$device",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "bytes",
          "label": "上傳（-）/下載（+）",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorPostfix": false,
      "colorValue": true,
      "colors": [
        "rgba(245, 54, 54, 0.9)",
        "rgba(237, 129, 40, 0.89)",
        "rgba(50, 172, 45, 0.97)"
      ],
      "datasource": "${DS_PROMETHEUS}",
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "format": "short",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 2,
        "w": 2,
        "x": 0,
        "y": 24
      },
      "id": 14,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "maxPerRow": 6,
      "nullPointMode": "null",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": false
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(node_cpu_seconds_total{instance=~\"$node\", mode='system'})",
          "format": "time_series",
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A",
          "step": 20
        }
      ],
      "thresholds": "1,2",
      "title": "CPU 核數",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorPostfix": false,
      "colorValue": true,
      "colors": [
        "rgba(245, 54, 54, 0.9)",
        "rgba(237, 129, 40, 0.89)",
        "rgba(50, 172, 45, 0.97)"
      ],
      "datasource": "${DS_PROMETHEUS}",
      "decimals": null,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "format": "short",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 2,
        "w": 2,
        "x": 15,
        "y": 24
      },
      "id": 179,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "maxPerRow": 6,
      "nullPointMode": "null",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": false
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "avg(node_filesystem_files_free{instance=~\"$node\",mountpoint=\"$maxmount\",fstype=~\"ext.?|xfs\"})",
          "format": "time_series",
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A",
          "step": 20
        }
      ],
      "thresholds": "100000,1000000",
      "title": "剩余節點數:$maxmount ",
      "type": "singlestat",
      "valueFontSize": "70%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": true,
      "colors": [
        "rgba(245, 54, 54, 0.9)",
        "rgba(237, 129, 40, 0.89)",
        "rgba(50, 172, 45, 0.97)"
      ],
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 0,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "format": "bytes",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 2,
        "w": 2,
        "x": 0,
        "y": 26
      },
      "id": 75,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "maxPerRow": 6,
      "nullPointMode": "null",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "70%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": false
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "sum(node_memory_MemTotal_bytes{instance=~\"$node\"})",
          "format": "time_series",
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "A",
          "step": 20
        }
      ],
      "thresholds": "2,3",
      "title": "總內存",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorPostfix": false,
      "colorValue": true,
      "colors": [
        "rgba(245, 54, 54, 0.9)",
        "rgba(237, 129, 40, 0.89)",
        "rgba(50, 172, 45, 0.97)"
      ],
      "datasource": "${DS_PROMETHEUS}",
      "decimals": null,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "format": "locale",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 2,
        "w": 2,
        "x": 15,
        "y": 26
      },
      "id": 178,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "maxPerRow": 6,
      "nullPointMode": "null",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": false
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "avg(node_filefd_maximum{instance=~\"$node\"})",
          "format": "time_series",
          "instant": true,
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A",
          "step": 20
        }
      ],
      "thresholds": "1024,10000",
      "title": "總文件描述符",
      "type": "singlestat",
      "valueFontSize": "70%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "aliasColors": {
        "192.168.200.241:9100_Total": "dark-red",
        "Idle - Waiting for something to happen": "#052B51",
        "guest": "#9AC48A",
        "idle": "#052B51",
        "iowait": "#EAB839",
        "irq": "#BF1B00",
        "nice": "#C15C17",
        "sdb_每秒I/O操作%": "#d683ce",
        "softirq": "#E24D42",
        "steal": "#FCE2DE",
        "system": "#508642",
        "user": "#5195CE",
        "磁盤花費在I/O操作占比": "#ba43a9"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 2,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 0,
        "y": 28
      },
      "hiddenSeries": false,
      "id": 7,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": true,
        "rightSide": false,
        "show": true,
        "sideWidth": null,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "maxPerRow": 6,
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "repeat": null,
      "seriesOverrides": [
        {
          "$$hashKey": "object:263",
          "alias": "/.*總使用率/",
          "color": "#C4162A",
          "fill": 0
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"system\"}[5m])) by (instance) *100",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "系統使用率",
          "refId": "A",
          "step": 20
        },
        {
          "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"user\"}[5m])) by (instance) *100",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "用戶使用率",
          "refId": "B",
          "step": 240
        },
        {
          "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"iowait\"}[5m])) by (instance) *100",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "磁盤IO使用率",
          "refId": "D",
          "step": 240
        },
        {
          "expr": "(1 - avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"idle\"}[5m])) by (instance))*100",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "總使用率",
          "refId": "F",
          "step": 240
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "CPU使用率",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:278",
          "decimals": 0,
          "format": "percent",
          "label": "",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "$$hashKey": "object:279",
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "192.168.200.241:9100_總內存": "dark-red",
        "使用率": "yellow",
        "內存_Avaliable": "#6ED0E0",
        "內存_Cached": "#EF843C",
        "內存_Free": "#629E51",
        "內存_Total": "#6d1f62",
        "內存_Used": "#eab839",
        "可用": "#9ac48a",
        "總內存": "#bf1b00"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 2,
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 8,
        "y": 28
      },
      "height": "300",
      "hiddenSeries": false,
      "id": 156,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": true,
        "rightSide": false,
        "show": true,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "總內存",
          "color": "#C4162A",
          "fill": 0
        },
        {
          "alias": "使用率",
          "color": "rgb(0, 209, 255)",
          "lines": false,
          "pointradius": 1,
          "points": true,
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "node_memory_MemTotal_bytes{instance=~\"$node\"}",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "總內存",
          "refId": "A",
          "step": 4
        },
        {
          "expr": "node_memory_MemTotal_bytes{instance=~\"$node\"} - node_memory_MemAvailable_bytes{instance=~\"$node\"}",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "已用",
          "refId": "B",
          "step": 4
        },
        {
          "expr": "node_memory_MemAvailable_bytes{instance=~\"$node\"}",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "可用",
          "refId": "F",
          "step": 4
        },
        {
          "expr": "node_memory_Buffers_bytes{instance=~\"$node\"}",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "內存_Buffers",
          "refId": "D",
          "step": 4
        },
        {
          "expr": "node_memory_MemFree_bytes{instance=~\"$node\"}",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "內存_Free",
          "refId": "C",
          "step": 4
        },
        {
          "expr": "node_memory_Cached_bytes{instance=~\"$node\"}",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "內存_Cached",
          "refId": "E",
          "step": 4
        },
        {
          "expr": "node_memory_MemTotal_bytes{instance=~\"$node\"} - (node_memory_Cached_bytes{instance=~\"$node\"} + node_memory_Buffers_bytes{instance=~\"$node\"} + node_memory_MemFree_bytes{instance=~\"$node\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "refId": "G"
        },
        {
          "expr": "(1 - (node_memory_MemAvailable_bytes{instance=~\"$node\"} / (node_memory_MemTotal_bytes{instance=~\"$node\"})))* 100",
          "format": "time_series",
          "hide": false,
          "interval": "30m",
          "intervalFactor": 10,
          "legendFormat": "使用率",
          "refId": "H"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "內存信息",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "bytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "percent",
          "label": "內存使用率",
          "logBase": 1,
          "max": "100",
          "min": "0",
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "192.168.10.227:9100_em1_in下載": "super-light-green",
        "192.168.10.227:9100_em1_out上傳": "dark-blue"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 2,
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 16,
        "y": 28
      },
      "height": "300",
      "hiddenSeries": false,
      "id": 157,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": true,
        "rightSide": false,
        "show": true,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_out上傳$/",
          "transform": "negative-Y"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "irate(node_network_receive_bytes_total{instance=~'$node',device=~\"$device\"}[5m])*8",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_in下載",
          "refId": "A",
          "step": 4
        },
        {
          "expr": "irate(node_network_transmit_bytes_total{instance=~'$node',device=~\"$device\"}[5m])*8",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_out上傳",
          "refId": "B",
          "step": 4
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "每秒網絡帶寬使用$device",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "bps",
          "label": "上傳（-）/下載（+）",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "15分鍾": "#6ED0E0",
        "1分鍾": "#BF1B00",
        "5分鍾": "#CCA300"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 2,
      "editable": true,
      "error": false,
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 1,
      "grid": {},
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 0,
        "y": 36
      },
      "height": "300",
      "hiddenSeries": false,
      "id": 13,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": true,
        "rightSide": false,
        "show": true,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "maxPerRow": 6,
      "nullPointMode": "null as zero",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "repeat": null,
      "seriesOverrides": [
        {
          "alias": "/.*總核數/",
          "color": "#C4162A"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "node_load1{instance=~\"$node\"}",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "1分鍾負載",
          "metric": "",
          "refId": "A",
          "step": 20,
          "target": ""
        },
        {
          "expr": "node_load5{instance=~\"$node\"}",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "5分鍾負載",
          "refId": "B",
          "step": 20
        },
        {
          "expr": "node_load15{instance=~\"$node\"}",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "15分鍾負載",
          "refId": "C",
          "step": 20
        },
        {
          "expr": " sum(count(node_cpu_seconds_total{instance=~\"$node\", mode='system'}) by (cpu,instance)) by(instance)",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "CPU總核數",
          "refId": "D",
          "step": 20
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "系統平均負載",
      "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 2,
        "value_type": "cumulative"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "vda_write": "#6ED0E0"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 2,
      "description": "Read bytes 每個磁盤分區每秒讀取的比特數\nWritten bytes 每個磁盤分區每秒寫入的比特數",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 1,
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 8,
        "y": 36
      },
      "height": "300",
      "hiddenSeries": false,
      "id": 168,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": true,
        "show": true,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_讀取$/",
          "transform": "negative-Y"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "irate(node_disk_read_bytes_total{instance=~\"$node\"}[5m])",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_讀取",
          "refId": "A",
          "step": 10
        },
        {
          "expr": "irate(node_disk_written_bytes_total{instance=~\"$node\"}[5m])",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_寫入",
          "refId": "B",
          "step": 10
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "每秒磁盤讀寫容量",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "Bps",
          "label": "讀取（-）/寫入（+）",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 1,
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 0,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 16,
        "y": 36
      },
      "hiddenSeries": false,
      "id": 174,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": true,
        "rightSide": false,
        "show": true,
        "sideWidth": null,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/Inodes.*/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "(node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}-node_filesystem_free_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}) *100/(node_filesystem_avail_bytes {instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}+(node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}-node_filesystem_free_bytes{instance=~'$node',fstype=~\"ext.*|xfs\",mountpoint !~\".*pod.*\"}))",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{mountpoint}}",
          "refId": "A"
        },
        {
          "expr": "node_filesystem_files_free{instance=~'$node',fstype=~\"ext.?|xfs\"} / node_filesystem_files{instance=~'$node',fstype=~\"ext.?|xfs\"}",
          "hide": true,
          "interval": "",
          "legendFormat": "Inodes：{{instance}}：{{mountpoint}}",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "磁盤使用率",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "percent",
          "label": "",
          "logBase": 1,
          "max": "100",
          "min": "0",
          "show": true
        },
        {
          "decimals": 2,
          "format": "percentunit",
          "label": null,
          "logBase": 1,
          "max": "1",
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "vda_write": "#6ED0E0"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 2,
      "description": "Reads completed: 每個磁盤分區每秒讀完成次數\n\nWrites completed: 每個磁盤分區每秒寫完成次數\n\nIO now 每個磁盤分區每秒正在處理的輸入/輸出請求數",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 0,
      "fillGradient": 0,
      "gridPos": {
        "h": 9,
        "w": 8,
        "x": 0,
        "y": 44
      },
      "height": "300",
      "hiddenSeries": false,
      "id": 161,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": true,
        "show": true,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_讀取$/",
          "transform": "negative-Y"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "irate(node_disk_reads_completed_total{instance=~\"$node\"}[5m])",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_讀取",
          "refId": "A",
          "step": 10
        },
        {
          "expr": "irate(node_disk_writes_completed_total{instance=~\"$node\"}[5m])",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_寫入",
          "refId": "B",
          "step": 10
        },
        {
          "expr": "node_disk_io_now{instance=~\"$node\"}",
          "format": "time_series",
          "hide": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "磁盤讀寫速率（IOPS）",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "iops",
          "label": "讀取（-）/寫入（+）I/O ops/sec",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "Idle - Waiting for something to happen": "#052B51",
        "guest": "#9AC48A",
        "idle": "#052B51",
        "iowait": "#EAB839",
        "irq": "#BF1B00",
        "nice": "#C15C17",
        "sdb_每秒I/O操作%": "#d683ce",
        "softirq": "#E24D42",
        "steal": "#FCE2DE",
        "system": "#508642",
        "user": "#5195CE",
        "磁盤花費在I/O操作占比": "#ba43a9"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": null,
      "description": "每一秒鍾的自然時間內，花費在I/O上的耗時。（wall-clock time）\n\nnode_disk_io_time_seconds_total：\n磁盤花費在輸入/輸出操作上的秒數。該值為累加值。（Milliseconds Spent Doing I/Os）\n\nirate(node_disk_io_time_seconds_total[1m])：\n計算每秒的速率：(last值-last前一個值)/時間戳差值，即：1秒鍾內磁盤花費在I/O操作的時間占比。",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 9,
        "w": 8,
        "x": 8,
        "y": 44
      },
      "hiddenSeries": false,
      "id": 175,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": false,
        "rightSide": false,
        "show": true,
        "sideWidth": null,
        "sort": null,
        "sortDesc": null,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "maxPerRow": 6,
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "irate(node_disk_io_time_seconds_total{instance=~\"$node\"}[5m])",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_每秒I/O操作%",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "每1秒內I/O操作耗時占比",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "percentunit",
          "label": "",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "vda": "#6ED0E0"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 2,
      "description": "Read time seconds 每個磁盤分區讀操作花費的秒數\n\nWrite time seconds 每個磁盤分區寫操作花費的秒數\n\nIO time seconds 每個磁盤分區輸入/輸出操作花費的秒數\n\nIO time weighted seconds每個磁盤分區輸入/輸出操作花費的加權秒數",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 1,
      "gridPos": {
        "h": 9,
        "w": 8,
        "x": 16,
        "y": 44
      },
      "height": "300",
      "hiddenSeries": false,
      "id": 160,
      "legend": {
        "alignAsTable": true,
        "avg": true,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": true,
        "show": true,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/,*_讀取$/",
          "transform": "negative-Y"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "irate(node_disk_read_time_seconds_total{instance=~\"$node\"}[5m]) / irate(node_disk_reads_completed_total{instance=~\"$node\"}[5m])",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_讀取",
          "refId": "B"
        },
        {
          "expr": "irate(node_disk_write_time_seconds_total{instance=~\"$node\"}[5m]) / irate(node_disk_writes_completed_total{instance=~\"$node\"}[5m])",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_寫入",
          "refId": "C"
        },
        {
          "expr": "irate(node_disk_io_time_seconds_total{instance=~\"$node\"}[5m])",
          "format": "time_series",
          "hide": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}",
          "refId": "A",
          "step": 10
        },
        {
          "expr": "irate(node_disk_io_time_weighted_seconds_total{instance=~\"$node\"}[5m])",
          "format": "time_series",
          "hide": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "{{device}}_加權",
          "refId": "D"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "每次IO讀寫的耗時（參考：小於100ms）(beta)",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "s",
          "label": "讀取（-）/寫入（+）",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "192.168.200.241:9100_TCP_alloc": "semi-dark-blue",
        "TCP": "#6ED0E0",
        "TCP_alloc": "blue"
      },
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "decimals": 2,
      "description": "Sockets_used - 已使用的所有協議套接字總量\n\nCurrEstab - 當前狀態為 ESTABLISHED 或 CLOSE-WAIT 的 TCP 連接數\n\nTCP_alloc - 已分配（已建立、已申請到sk_buff）的TCP套接字數量\n\nTCP_tw - 等待關閉的TCP連接數\n\nUDP_inuse - 正在使用的 UDP 套接字數量\n\nRetransSegs - TCP 重傳報文數\n\nOutSegs - TCP 發送的報文數\n\nInSegs - TCP 接收的報文數",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 0,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 16,
        "x": 0,
        "y": 53
      },
      "height": "300",
      "hiddenSeries": false,
      "id": 158,
      "interval": "",
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "hideEmpty": true,
        "hideZero": true,
        "max": true,
        "min": false,
        "rightSide": true,
        "show": true,
        "sideWidth": null,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*Sockets_used/",
          "color": "#E02F44",
          "lines": false,
          "pointradius": 1,
          "points": true,
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "node_netstat_Tcp_CurrEstab{instance=~'$node'}",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "CurrEstab",
          "refId": "A",
          "step": 20
        },
        {
          "expr": "node_sockstat_TCP_tw{instance=~'$node'}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "TCP_tw",
          "refId": "D"
        },
        {
          "expr": "node_sockstat_sockets_used{instance=~'$node'}",
          "hide": false,
          "interval": "30m",
          "intervalFactor": 1,
          "legendFormat": "Sockets_used",
          "refId": "B"
        },
        {
          "expr": "node_sockstat_UDP_inuse{instance=~'$node'}",
          "interval": "",
          "legendFormat": "UDP_inuse",
          "refId": "C"
        },
        {
          "expr": "node_sockstat_TCP_alloc{instance=~'$node'}",
          "interval": "",
          "legendFormat": "TCP_alloc",
          "refId": "E"
        },
        {
          "expr": "irate(node_netstat_Tcp_PassiveOpens{instance=~'$node'}[5m])",
          "hide": true,
          "interval": "",
          "legendFormat": "{{instance}}_Tcp_PassiveOpens",
          "refId": "G"
        },
        {
          "expr": "irate(node_netstat_Tcp_ActiveOpens{instance=~'$node'}[5m])",
          "hide": true,
          "interval": "",
          "legendFormat": "{{instance}}_Tcp_ActiveOpens",
          "refId": "F"
        },
        {
          "expr": "irate(node_netstat_Tcp_InSegs{instance=~'$node'}[5m])",
          "interval": "",
          "legendFormat": "Tcp_InSegs",
          "refId": "H"
        },
        {
          "expr": "irate(node_netstat_Tcp_OutSegs{instance=~'$node'}[5m])",
          "interval": "",
          "legendFormat": "Tcp_OutSegs",
          "refId": "I"
        },
        {
          "expr": "irate(node_netstat_Tcp_RetransSegs{instance=~'$node'}[5m])",
          "hide": false,
          "interval": "",
          "legendFormat": "Tcp_RetransSegs",
          "refId": "J"
        },
        {
          "expr": "irate(node_netstat_TcpExt_ListenDrops{instance=~'$node'}[5m])",
          "hide": true,
          "interval": "",
          "legendFormat": "",
          "refId": "K"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "網絡Socket連接信息",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "transformations": [],
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": "已使用的所有協議套接字總量",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {
        "filefd_192.168.200.241:9100": "super-light-green",
        "switches_192.168.200.241:9100": "semi-dark-red",
        "使用的文件描述符_10.118.72.128:9100": "red",
        "每秒上下文切換次數_10.118.71.245:9100": "yellow",
        "每秒上下文切換次數_10.118.72.128:9100": "yellow"
      },
      "bars": false,
      "cacheTimeout": null,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "description": "",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 0,
      "fillGradient": 1,
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 16,
        "y": 53
      },
      "hiddenSeries": false,
      "hideTimeOverride": false,
      "id": 16,
      "legend": {
        "alignAsTable": false,
        "avg": false,
        "current": true,
        "max": false,
        "min": false,
        "rightSide": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 2,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pluginVersion": "6.4.2",
      "pointradius": 1,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/每秒上下文切換次數.*/",
          "color": "#FADE2A",
          "lines": false,
          "pointradius": 1,
          "points": true,
          "yaxis": 2
        },
        {
          "alias": "/使用的文件描述符.*/",
          "color": "#F2495C"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "node_filefd_allocated{instance=~\"$node\"}",
          "format": "time_series",
          "instant": false,
          "interval": "",
          "intervalFactor": 5,
          "legendFormat": "使用的文件描述符",
          "refId": "B"
        },
        {
          "expr": "irate(node_context_switches_total{instance=~\"$node\"}[5m])",
          "interval": "",
          "intervalFactor": 5,
          "legendFormat": "每秒上下文切換次數",
          "refId": "A"
        },
        {
          "expr": "  (node_filefd_allocated{instance=~\"$node\"}/node_filefd_maximum{instance=~\"$node\"}) *100",
          "format": "time_series",
          "hide": true,
          "instant": false,
          "interval": "",
          "intervalFactor": 5,
          "legendFormat": "使用的文件描述符占比_{{instance}}",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "打開的文件描述符(左 )/每秒上下文切換次數(右)",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": "使用的文件描述符",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": "context_switches",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "refresh": "",
  "schemaVersion": 22,
  "style": "dark",
  "tags": [
    "Prometheus",
    "node_exporter",
    "StarsL.cn"
  ],
  "templating": {
    "list": [
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_PROMETHEUS}",
        "definition": "label_values(node_uname_info, job)",
        "hide": 0,
        "includeAll": false,
        "index": -1,
        "label": "JOB",
        "multi": false,
        "name": "job",
        "options": [],
        "query": "label_values(node_uname_info, job)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 5,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_PROMETHEUS}",
        "definition": "label_values(node_uname_info{job=~\"$job\"}, nodename)",
        "hide": 0,
        "includeAll": true,
        "index": -1,
        "label": "主機名",
        "multi": false,
        "name": "hostname",
        "options": [],
        "query": "label_values(node_uname_info{job=~\"$job\"}, nodename)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 5,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allFormat": "glob",
        "allValue": null,
        "current": {},
        "datasource": "${DS_PROMETHEUS}",
        "definition": "label_values(node_uname_info{job=~\"$job\",nodename=~\"$hostname\"},instance)",
        "hide": 0,
        "includeAll": false,
        "index": -1,
        "label": "Instance",
        "multi": true,
        "multiFormat": "regex values",
        "name": "node",
        "options": [],
        "query": "label_values(node_uname_info{job=~\"$job\",nodename=~\"$hostname\"},instance)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 5,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allFormat": "glob",
        "allValue": null,
        "current": {},
        "datasource": "${DS_PROMETHEUS}",
        "definition": "label_values(node_network_info{device!~'tap.*|veth.*|br.*|docker.*|virbr.*|lo.*|cni.*'},device)",
        "hide": 0,
        "includeAll": true,
        "index": -1,
        "label": "網卡",
        "multi": true,
        "multiFormat": "regex values",
        "name": "device",
        "options": [],
        "query": "label_values(node_network_info{device!~'tap.*|veth.*|br.*|docker.*|virbr.*|lo.*|cni.*'},device)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_PROMETHEUS}",
        "definition": "query_result(topk(1,sort_desc (max(node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext.?|xfs\",mountpoint!~\".*pods.*\"}) by (mountpoint))))",
        "hide": 2,
        "includeAll": false,
        "index": -1,
        "label": "最大掛載目錄",
        "multi": false,
        "name": "maxmount",
        "options": [],
        "query": "query_result(topk(1,sort_desc (max(node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext.?|xfs\",mountpoint!~\".*pods.*\"}) by (mountpoint))))",
        "refresh": 2,
        "regex": "/.*\\\"(.*)\\\".*/",
        "skipUrlSync": false,
        "sort": 5,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {},
        "datasource": "${DS_PROMETHEUS}",
        "definition": "label_values(node_uname_info{job=~\"$job\",instance=~\"$node\"}, nodename)",
        "hide": 2,
        "includeAll": false,
        "index": -1,
        "label": "展示使用的主機名",
        "multi": false,
        "name": "show_hostname",
        "options": [],
        "query": "label_values(node_uname_info{job=~\"$job\",instance=~\"$node\"}, nodename)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 5,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-12h",
    "to": "now"
  },
  "timepicker": {
    "hidden": false,
    "now": true,
    "refresh_intervals": [
      "15s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "browser",
  "title": "1   Node Exporter for Prometheus Dashboard CN v20200628",
  "uid": "9CWBz0bik",
  "variables": {
    "list": []
  },
  "version": 91
}

 

 


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 kubernetes之監控Operator部署Prometheus（三） Prometheus Operator 監控Kubernetes 第三篇 kubernetes使用Operator部署Prometheus監控 prometheus operator（Kubernetes 集群監控） prometheus-operator監控Kubernetes 使用 prometheus-operator 監控 Kubernetes 集群 K8S從入門到放棄系列-(16)Kubernetes集群Prometheus-operator監控部署 Kubernetes 系列（六）:Kubernetes部署Prometheus監控 Kubernetes1.13.1安裝prometheus-operator監控 生產prometheus-operator 監控二進制kubernetes
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM