什么是 Operator？
operator是由CoreOS公司開發的，用來擴展kubernetes APi ，特定的應用程序控制器，它用來創建，配置和管理復雜的有狀態應用，例如數據庫，緩存和監控系統。Operator基於Kubernetes的資源和控制器概念之上構建，但同時又包含了應用程序特定的一些專業知識，比如創建一個數據庫的Operator，則必須對創建的數據庫和各種運維方式非常了解，創建operator的關鍵是 CRD（CustomResourceDefinition/自定義資源定義）的設計。
CRD 是對Kubernetes APi的擴展，Kubernetes中的每個資源都會是一個API對象的集合，例如我們在YAML文件里定義的那些
spec都是對 Kubernetes 中的資源對象的定義，所有的自定義資源可以跟 Kubernetes 中內建的資源一樣使用 kubectl 操作。
Operate是將運維人員對軟件操作的知識給代碼化，同時利用Kubernetes強大的抽象來管理大規模的軟件應用，目前CoreOS官方提供了幾種Operator的實現，其中就包括我們今天的主角：Prometheus Operator，Operator的核心實現就是基於 Kubernetes 的以下兩個概念：
- 資源：對象的狀態定義
- 控制器：觀測、分析和行動，以調節資源的分布
當然我們如果有對應的需求也完全可以自己去實現一個Operator，接下來我們就來給大家詳細介紹下Prometheus-Operator的使用方法。
介紹Prometheus-Operator
首先我們先來了解下Prometheus-Operator的架構圖：
上圖是Prometheus-Operator官方提供的架構圖，其中Operator是最核心的部分，作為一個控制器，他會去創建Prometheus、ServiceMonitor、AlertManager以及PrometheusRule4個CRD資源對象，然后會一直監控並維持這4個資源對象的狀態。
其中創建的prometheus這種資源對象就是作為Prometheus Server存在，而ServiceMonitor就是exporter的各種抽象，exporter前面我們已經學習了，是用來提供專門提供metrics數據接口的工具，Prometheus就是通過ServiceMonitor提供的metrics數據接口去 pull 數據的，當然alertmanager這種資源對象就是對應的AlertManager的抽象，而PrometheusRule是用來被Prometheus實例使用的報警規則文件。
這樣我們要在集群中監控什么數據，就變成了直接去操作 Kubernetes 集群的資源對象了，是不是方便很多了。上圖中的 Service 和 ServiceMonitor 都是 Kubernetes 的資源，一個 ServiceMonitor 可以通過 labelSelector 的方式去匹配一類 Service，Prometheus 也可以通過 labelSelector 去匹配多個ServiceMonitor。
1，service 和Endpoint 通過名稱來相對應，不管是k8s集群外部來時k8s集群內部，只要svc 下面有資源(Pod)那么他就會對應有 endpoint
2，ServerMonitor 通過lable 來匹配service的，后面詳細講解如何匹配
3，prometheus 通過
Prometheus.spec.serviceMonitorNamespaceSelector和Prometheus.spec.serviceMonitorSelector來匹配ServerMonitor
安裝Prometheus-operate
我們這里采用prometheus-operate 源碼來安裝。
官方給出的一個版本監控測試表，我們查看自己安裝的kubernetes 版本，然后選擇prometheus-operate對應的版本，防止出現不兼容情況
注意：由於Kubernetes v1.16.1和Kubernetes v1.16.5之前的兩個bug，kube-prometheus release-0.4分支只支持v1.16.5及以上版本。可以手動編輯 kube-system 命名空間中的 extension-apiserver-authentication-reader 角色，以包含 list 和 watch 權限，從而解決 Kubernetes v1.16.2 至 v1.16.4 中的第二個問題。
我這邊集群是1.15版本，所以我就拉去release-0.3版本，首先Clone 代碼
git clone https://github.com/prometheus-operator/kube-prometheus.git
git checkout release-0.3
最新的版本官方將資源https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus遷移到了獨立的 git 倉庫中：https://github.com/coreos/kube-prometheus.git
官方把所有文件都放在一起了，我們給他分開存放
mkdir kube-prom
cp -a kube-prometheus/manifests/* kube-prom/
cd kube-prom/
mkdir -p node-exporter alertmanager grafana kube-state-metrics prometheus serviceMonitor adapter
mv *-serviceMonitor* serviceMonitor/
mv setup operator/
mv grafana-* grafana/
mv kube-state-metrics-* kube-state-metrics/
mv alertmanager-* alertmanager/
mv node-exporter-* node-exporter/
mv prometheus-adapter* adapter/
mv prometheus-* prometheus/
$ ll
total 32
drwxr-xr-x 2 root root 4096 Dec 2 09:53 adapter
drwxr-xr-x 2 root root 4096 Dec 2 09:53 alertmanager
drwxr-xr-x 2 root root 4096 Dec 2 09:53 grafana
drwxr-xr-x 2 root root 4096 Dec 2 09:53 kube-state-metrics
drwxr-xr-x 2 root root 4096 Dec 2 09:53 node-exporter
drwxr-xr-x 2 root root 4096 Dec 2 09:34 operator
drwxr-xr-x 2 root root 4096 Dec 2 09:53 prometheus
drwxr-xr-x 2 root root 4096 Dec 2 09:53 serviceMonitor
$ ll operator/
total 660
-rw-r--r-- 1 root root 60 Dec 2 09:34 0namespace-namespace.yaml
-rw-r--r-- 1 root root 274629 Dec 2 09:34 prometheus-operator-0alertmanagerCustomResourceDefinition.yaml
-rw-r--r-- 1 root root 12100 Dec 2 09:34 prometheus-operator-0podmonitorCustomResourceDefinition.yaml
-rw-r--r-- 1 root root 321507 Dec 2 09:34 prometheus-operator-0prometheusCustomResourceDefinition.yaml
-rw-r--r-- 1 root root 14561 Dec 2 09:34 prometheus-operator-0prometheusruleCustomResourceDefinition.yaml
-rw-r--r-- 1 root root 17422 Dec 2 09:34 prometheus-operator-0servicemonitorCustomResourceDefinition.yaml
-rw-r--r-- 1 root root 425 Dec 2 09:34 prometheus-operator-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 1066 Dec 2 09:34 prometheus-operator-clusterRole.yaml
-rw-r--r-- 1 root root 1405 Dec 2 09:34 prometheus-operator-deployment.yaml
-rw-r--r-- 1 root root 239 Dec 2 09:34 prometheus-operator-serviceAccount.yaml
-rw-r--r-- 1 root root 420 Dec 2 09:34 prometheus-operator-service.yaml
有些版本的k8s的label為beta.kubernetes.io/os，
// 查看當前k8s集群nodes 上面的label標簽
# cat > label.sh <<-'EOF'
#!/bin/bash
kubectl get nodes -o go-template='{{printf "%-39s %-12s\n" "Node" "Label"}}
{{- range .items}}
{{- if $labels := (index .metadata.labels) }}
{{- .metadata.name }}{{ "\t" }}
{{- range $key, $value := $labels }}
{{$key}}{{ "\t" }}{{$value}}
{{- end }}
{{- "\n" }}
{{- end}}
{{- end}}'
EOF
# bash label.sh
Node Label
jx4-74.host.com
beta.kubernetes.io/arch amd64
beta.kubernetes.io/os linux
kubernetes.io/arch amd64
kubernetes.io/hostname jx4-74.host.com
kubernetes.io/os linux
node-role.kubernetes.io/master
node-role.kubernetes.io/node
jx4-75.host.com
beta.kubernetes.io/arch amd64
beta.kubernetes.io/os linux
kubernetes.io/arch amd64
kubernetes.io/hostname jx4-75.host.com
kubernetes.io/os linux
node-role.kubernetes.io/master
node-role.kubernetes.io/node
jx4-76.host.com
beta.kubernetes.io/arch amd64
beta.kubernetes.io/os linux
kubernetes.io/arch amd64
kubernetes.io/hostname jx4-76.host.com
kubernetes.io/os linux
node-role.kubernetes.io/node
如果是上面這種沒有kubernetes.io/os: linux的情況則需要修改yaml里的nodeselector字段
$ grep -A1 nodeSelector alertmanager/alertmanager-alertmanager.yaml
nodeSelector:
kubernetes.io/os: linux
需要修改下面這些文件：
adapter/prometheus-adapter-deployment.yaml
alertmanager/alertmanager-alertmanager.yaml
grafana/grafana-deployment.yaml
kube-state-metrics/kube-state-metrics-deployment.yaml
node-exporter/node-exporter-daemonset.yaml
operator/prometheus-operator-deployment.yaml
prometheus/prometheus-prometheus.yaml
里面鏡像使用的是 quay.io 的，因為我們自己有鏡像倉庫，所以我們提前先把Docker包拉去下來，然后傳到自己倉庫里去
# grep "quay.io" -r *| awk -F '[ =]' '{print $NF}' | sort |uniq
quay.io/coreos/configmap-reload:v0.0.1
quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0
quay.io/coreos/kube-rbac-proxy:v0.4.1
quay.io/coreos/kube-state-metrics:v1.8.0
quay.io/coreos/prometheus-config-reloader:v0.34.0
quay.io/coreos/prometheus-operator:v0.34.0
quay.io/prometheus/alertmanager
quay.io/prometheus/node-exporter:v0.18.1
quay.io/prometheus/prometheus
# 需要更改的文件內容如下，自行更改這里不再過多的介紹
adapter/prometheus-adapter-deployment.yaml
grafana/grafana-deployment.yaml
kube-state-metrics/kube-state-metrics-deployment.yaml
node-exporter/node-exporter-daemonset.yaml
operator/prometheus-operator-deployment.yaml
部署Operator
這個需要提前部署
kubectl apply -f operator/
確認正常之后在往后執行
# kubectl -n monitoring get pods
NAME READY STATUS RESTARTS AGE
prometheus-operator-58bc77f954-q54pw 1/1 Running 0 33h
部署整套CRD
kubectl apply -f adapter/
kubectl apply -f alertmanager/
kubectl apply -f node-exporter/
kubectl apply -f kube-state-metrics/
kubectl apply -f grafana/
kubectl apply -f prometheus/
kubectl apply -f serviceMonitor/
如果我們使用的是自己的harbor 倉庫，這一步應該很快就結束了。
# kubectl -n monitoring get all
NAME READY STATUS RESTARTS AGE
pod/alertmanager-main-0 2/2 Running 0 33h
pod/alertmanager-main-1 2/2 Running 0 33h
pod/alertmanager-main-2 2/2 Running 0 33h
pod/grafana-848969744d-6dxgb 1/1 Running 0 33h
pod/kube-state-metrics-5dd8b47cbb-nf68s 3/3 Running 0 33h
pod/node-exporter-4glkn 0/2 Pending 0 33h
pod/node-exporter-8p7c9 0/2 Pending 0 33h
pod/node-exporter-lh77n 0/2 Pending 0 33h
pod/prometheus-adapter-578c5c64b6-bzlp6 1/1 Running 0 33h
pod/prometheus-k8s-0 3/3 Running 1 33h
pod/prometheus-k8s-1 3/3 Running 1 33h
pod/prometheus-operator-58bc77f954-q54pw 1/1 Running 0 33h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-main ClusterIP 10.16.131.214 <none> 9093/TCP 33h
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 33h
service/grafana ClusterIP 10.16.36.173 <none> 3000/TCP 33h
service/kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 33h
service/node-exporter ClusterIP None <none> 9100/TCP 33h
service/prometheus-adapter ClusterIP 10.16.212.212 <none> 443/TCP 33h
service/prometheus-k8s ClusterIP 10.16.156.122 <none> 9090/TCP 33h
service/prometheus-operated ClusterIP None <none> 9090/TCP 33h
service/prometheus-operator ClusterIP None <none> 8080/TCP 33h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/node-exporter 3 3 0 3 0 kubernetes.io/os=linux 33h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/grafana 1/1 1 1 33h
deployment.apps/kube-state-metrics 1/1 1 1 33h
deployment.apps/prometheus-adapter 1/1 1 1 33h
deployment.apps/prometheus-operator 1/1 1 1 33h
NAME DESIRED CURRENT READY AGE
replicaset.apps/grafana-848969744d 1 1 1 33h
replicaset.apps/kube-state-metrics-5dd8b47cbb 1 1 1 33h
replicaset.apps/prometheus-adapter-578c5c64b6 1 1 1 33h
replicaset.apps/prometheus-operator-58bc77f954 1 1 1 33h
NAME READY AGE
statefulset.apps/alertmanager-main 3/3 33h
statefulset.apps/prometheus-k8s 2/2 33h
部署ingress 使其能通過web界面訪問
根據自己的實際情況進行更改
# cat ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kube-prometheus
namespace: monitoring
spec:
rules:
- host: prometheus.zsf.com
http:
paths:
- path: /
backend:
serviceName: prometheus-k8s
servicePort: 9090
- host: grafana.zsf.com
http:
paths:
- path: /
backend:
serviceName: grafana
servicePort: 3000
到此為止prometheus 基本部署完成，但是我們要解決其中的幾個遺漏問題
遺留問題處理
kube-controller-manager 和 kube-scheduler 無監控數據
這里要注意有一個坑,二進制部署k8s管理組件和新版本kubeadm部署的都會發現在prometheus server的頁面上發現kube-controller和kube-schedule的target為0/0也就是上圖所示
這是因為serviceMonitor是根據label去選取svc的,我們可以看到對應的serviceMonitor是選取的ns范圍是kube-system
# cat serviceMonitor/prometheus-serviceMonitorKubeScheduler.yaml
...
spec:
endpoints:
...
...
port: http-metrics
jobLabel: k8s-app
namespaceSelector:
matchNames:
- kube-system
selector:
matchLabels:
k8s-app: kube-scheduler
# cat serviceMonitor/prometheus-serviceMonitorKubeControllerManager.yaml
...
jobLabel: k8s-app
namespaceSelector:
matchNames:
- kube-system
selector:
matchLabels:
k8s-app: kube-controller-manager
而默認的 kube-system 里面沒有符合上面的svc，所以讀取不到
# kubectl -n kube-system get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
coredns ClusterIP 10.16.0.2 <none> 53/UDP,53/TCP,9153/TCP 9d
heapster ClusterIP 10.16.157.201 <none> 80/TCP 7d20h
kubelet ClusterIP None <none> 10250/TCP 2d
kubernetes-dashboard ClusterIP 10.16.232.114 <none> 443/TCP 8d
traefik-ingress-service ClusterIP 10.16.193.162 <none> 80/TCP,8080/TCP 9d
但是卻有對應的ep(沒有帶任何label)被創建
解決上述這個問題的辦法,我們手動創建一個
# cat prometheus-kubeMasterService.yaml
apiVersion: v1
kind: Service
metadata:
namespace: kube-system
name: kube-controller-manager
labels:
k8s-app: kube-controller-manager
spec:
type: ClusterIP
clusterIP: None
ports:
- name: http-metrics
port: 10252
targetPort: 10252
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
namespace: kube-system
name: kube-scheduler
labels:
k8s-app: kube-scheduler
spec:
type: ClusterIP
clusterIP: None
ports:
- name: http-metrics
port: 10251
targetPort: 10251
protocol: TCP
注意：
Service.spec.ports.name要和ServiceMonitor.spec.endpoints.port對應起來
Service.metadata.namespace要和ServiceMonitor.namespaceSelector.matchNames對應起來
Service.metadata.labels的key要和ServiceMonitor.JobLabel對應起來
Service.metadata.labels要和ServiceMonitor.selector.matchLabels`對應起來
如果是kubeadm 部署的，我們可以查看下controller-manager和scheduler的標簽，然后添加Service.spec.selector
...
spec:
selector:
component: kube-scheduler
type: ClusterIP
...
二進制的話需要我們手動填入svc對應的ep的屬性,我集群是HA的,所有有三個,僅供參考,別傻傻得照抄,另外這個ep的名字得和上面的svc的名字和屬性對應上
# cat prometheus-kubeMasterEndPoints.yaml
apiVersion: v1
kind: Endpoints
metadata:
labels:
k8s-app: kube-controller-manager
name: kube-controller-manager
namespace: kube-system
subsets:
- addresses:
- ip: 192.168.4.74
- ip: 192.168.4.75
ports:
- name: http-metrics
port: 10252
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
labels:
k8s-app: kube-scheduler
name: kube-scheduler
namespace: kube-system
subsets:
- addresses:
- ip: 192.168.4.74
- ip: 192.168.4.75
ports:
- name: http-metrics
port: 10251
protocol: TCP
然后上述那個問題就不會存在了。
監控只能看到三個Namespaces
默認serviceMonitor實際上只能選三個namespacs，默認和Kube-system和monitoring，見文件cat prometheus-roleSpecificNamespaces.yaml，需要其他的ns自行創建role，我們只需要對着抄一下就行
- apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: prometheus-k8s
namespace: monitoring
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- pods
verbs:
- get
- list
- watch
上述其實只需要更改namespace字段
Apiserver 一直處於 Down 的狀態
這個是因為我們自己的證書是自己生成的，並沒有通過第三方證書廠商認證，就想我們自己生成的證書放在 nginx 上之后我們通過 curl 的時候會提示證書不可信，這個時候我們可以使用如下操作 對證書不進行驗證，我們只需要不驗證證書就行了。
$ vim serviceMonitor/prometheus-serviceMonitorKubelet.yaml
...
tlsConfig:
insecureSkipVerify: true
jobLabel: k8s-app
...
然后重新運行之后發現正常，就不會報錯 401 了。
kubelet 不正常
在 kubelet 啟動的時候添加了下面兩個參數就正常了
--authentication-token-webhook="true" \
--authorization-mode="Webhook"
生產中的優化
prometheus 數據持久化
官方默認的部署方式不是數據持久化的，如果有對應的需求，需要我們自己來修改
我們通過 prometheus 這個 CRD 創建的 Prometheus 並沒有做數據的持久化，我們可以直接查看生成的 Prometheus Pod 的掛載情況就清楚了：
$ kubectl get pod prometheus-k8s-0 -n monitoring -o yaml
......
volumeMounts:
- mountPath: /etc/prometheus/config_out
name: config-out
readOnly: true
- mountPath: /prometheus
name: prometheus-k8s-db
......
volumes:
......
- emptyDir: {}
name: prometheus-k8s-db
......
我們可以看到 Prometheus 的數據目錄 /prometheus 實際上是通過 emptyDir 進行掛載的，我們知道 emptyDir 掛載的數據的生命周期和 Pod 生命周期一致的，所以如果 Pod 掛掉了，數據也就丟失了，這也就是為什么我們重建 Pod 后之前的數據就沒有了的原因，對應線上的監控數據肯定需要做數據的持久化的，同樣的 prometheus 這個 CRD 資源也為我們提供了數據持久化的配置方法，
$ vim operator/prometheus-operator-0prometheusCustomResourceDefinition.yaml
....
storage:
description: StorageSpec defines the configured storage for a group
Prometheus servers. If neither `emptyDir` nor `volumeClaimTemplate`
is specified, then by default an [EmptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir)
will be used.
properties:
emptyDir:
description: Represents an empty directory for a pod. Empty directory
volumes support ownership management and SELinux relabeling.
properties:
medium:
description: 'What type of storage medium should back this directory.
The default is "" which means to use the node''s default medium.
Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir'
type: string
sizeLimit: {}
type: object
volumeClaimTemplate:
description: PersistentVolumeClaim is a user's request for and claim
to a persistent volume
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this
representation of an object. Servers should convert recognized
schemas to the latest internal value, and may reject unrecognized
values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource
this object represents. Servers may infer this from the endpoint
the client submits requests to. Cannot be updated. In CamelCase.
More info: https://git.k8s.io/community
....
由於我們的 Prometheus 最終是通過 Statefulset 控制器進行部署的，所以我們這里需要通過 storageclass 來做數據持久化，首先創建一個 StorageClass 對象：
我們這里是使用 nfs 做 storageClass，存儲類的，具體配置如下：
nfs 服務安裝在 192.168.4.73，共享的目錄為/home/volume
因為 nfs 默認不支持 storageClass ，所以我們需要安裝一個外部插件，https://kubernetes.io/zh/docs/concepts/storage/storage-classes/ 那些存儲直接支持，可以查看官方文檔，我們這里面安裝nfs-client 插件，官方地址為：https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client/deploy
創建 rbac 授權
$ vim storage/prometheus-nfs-client-rbac.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
創建 nfs-client 啟動文件：
$ vim storage/prometheus-nfs-client-deployment.yml
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 192.168.4.73 #nfs server 地址
- name: NFS_PATH
value: /home/volume #nfs共享目錄
volumes:
- name: nfs-client-root
nfs:
server: 192.168.4.73
path: /home/volume
創建 storageClass 對象
$ cat storage/prometheus-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: prometheus-data-db
provisioner: fuseim.pri/ifs #對應PROVISIONER_NAME 的值
這里我們聲明Storageclass對象，其中provisioner=fuseim.pri/ifs，則是我們集群中使用NFS作為后端存儲
創建所有資源
cd storage
kubectl create -f .
然后我們在 prometheus 添加如下配置
$ vim
...
spec:
storage:
volumeClaimTemplate:
spec:
storageClassName: prometheus-data-db
resources:
requests:
storage: 10Gi
alerting:
alertmanagers:
- name: alertmanager-main
namespace: monitoring
port: web
...
然后重新執行啟動：
kubectl apply -f prometheus/prometheus-prometheus.yaml
查看結果：
$ kubectl get pvc -n monitoring
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-k8s-db-prometheus-k8s-0 Bound pvc-72a5cb90-8065-4ce8-be47-a01ced7dc152 10Gi RWO prometheus-data-db 31s
prometheus-k8s-db-prometheus-k8s-1 Bound pvc-8f4b7527-06f7-41de-9bef-39f10a6c354c 10Gi RWO prometheus-data-db 31s
$ kubectl get pv -n monitoring
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
grafana 5Gi RWO Recycle Available 2d14h
pvc-72a5cb90-8065-4ce8-be47-a01ced7dc152 10Gi RWO Delete Bound monitoring/prometheus-k8s-db-prometheus-k8s-0 prometheus-data-db 43s
pvc-8f4b7527-06f7-41de-9bef-39f10a6c354c 10Gi RWO Delete Bound monitoring/prometheus-k8s-db-prometheus-k8s-1 prometheus-data-db 43s
Grafana 數據持久化
創建 pv-pvc
# cat grafana-pv-pvc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: grafana-volume
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
nfs:
server: 192.168.4.73
path: /home/kubernetes-volume/grafana
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-volume
namespace: monitoring
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
創建 pv-pvc 並查看資源是否綁定
# kubectl apply -f grafana-pv-pvc.yaml
# kubectl get -n monitoring pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
grafana-volume Bound grafana 5Gi RWO 59s
更改 grafana 資源清單
$ vim grafana/grafana-deployment.yaml
# volumes:
# - emptyDir: {}
# name: grafana-storage
- name: grafana-storage
persistentVolumeClaim:
claimName: grafana-volume
Grafana 修改
官方的yaml grafana-dashboardDefinitions.yaml里面很多promQL的metrics名字還是老的，需要改，后續有空更新
API server
kubelet部分
operation Rate`的`kubelet_runtime_operations_total` –> `kubelet_runtime_operations
[root@k8s-m1 ~]# curl -sSk --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key https://172.16.0.4:10250/metrics | grep -P kubelet_runtime_operations
# HELP kubelet_runtime_operations Cumulative number of runtime operations by operation type.
# TYPE kubelet_runtime_operations counter
kubelet_runtime_operations{operation_type="container_status"} 94
...
operation Error Rate`的`kubelet_runtime_operations_errors_total` –> `kubelet_runtime_operations_errors
# HELP kubelet_runtime_operations_errors Cumulative number of runtime operation errors by operation type.
# TYPE kubelet_runtime_operations_errors counter
kubelet_runtime_operations_errors{operation_type="container_status"} 8
kubelet_runtime_operations_errors{operation_type="pull_image"} 13
Pod部分
下面三個的pod改成pod_name,是Network I/O
grep container_network_ grafana-dashboardDefinitions.yaml
"expr": "sort_desc(sum by (pod) (irate(container_network_receive_bytes_total{job=\"kubelet\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[4m])))",
"expr": "sort_desc(sum by (pod) (irate(container_network_transmit_bytes_total{job=\"kubelet\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\"}[4m])))",
"expr": "sum(rate(container_network_transmit_bytes_total{job=\"kubelet\", cluster=\"$cluster\", namespace=\"$namespace\", pod=~\"$statefulset.*\"}[3m])) + sum(rate(container_network_receive_bytes_total{cluster=\"$cluster\", namespace=\"$namespace\",pod=~\"$statefulset.*\"}[3m]))",
Total Restarts Per Container
參考資料
https://juejin.im/post/6865504989695967245
https://zhangguanzhang.github.io/2018/10/12/prometheus-operator/#部署官方的prometheus-operator
https://www.qikqiak.com/post/first-use-prometheus-operator/
https://www.qikqiak.com/post/prometheus-operator-monitor-etcd/