一.Prometheus简介
Prometheus是一款面向云原生应用程序的开源监控工具，作为第一个从CNCF毕业的监控工具而言，开发者对于Prometheus寄予了巨大的希望。在Kubernetes社区中，很多人认为Prometheus是容器场景中监控的第一方案，成为容器监控标准的制定者。在本文中，我们会为大家介绍如何快速部署一套Kubernetes的监控解决方案。
二.安装步骤
1./app/prometheus-operator/alertmanager.yaml文件内容，该文件主要配置告警邮件的接收人与发件人
global:
resolve_timeout: 5m
http_config: {}
smtp_hello:
'smtp.exmail.qq.com:25'
smtp_from:
'lihaichun@netschina.com'
smtp_smarthost:
'smtp.exmail.qq.com:25'
smtp_auth_username:
'lihaichun@netschina.com'
smtp_auth_password:
'XXXX'
smtp_require_tls:
false
pagerduty_url: https:
//events.pagerduty.com/v2/enqueue
hipchat_api_url: https:
//api.hipchat.com/
opsgenie_api_url: https:
//api.opsgenie.com/
wechat_api_url: https:
//qyapi.weixin.qq.com/cgi-bin/
victorops_api_url: https:
//alert.victorops.com/integrations/generic/20131114/alert/
# The root route on which each incoming alert enters.
route:
# The labels by which incoming alerts are grouped together. For example,
# multiple alerts coming in
for
cluster=A and alertname=LatencyHigh would
# be batched into a single group.
group_by: [
'alertname'
,
'cluster'
,
'service'
]
# When a
new
group of alerts is created by an incoming alert, wait at
# least
'group_wait'
to send the initial notification.
# This way ensures that you get multiple alerts
for
the same group that start
# firing shortly after another are batched together on the first
# notification.
group_wait: 30s
# When the first notification was sent, wait
'group_interval'
to send a batch
# of
new
alerts that started firing
for
that group.
group_interval: 30s
# If an alert has successfully been sent, wait
'repeat_interval'
to
# resend them.
#repeat_interval: 20s
repeat_interval: 12h
# A
default
receiver
# If an alert isn't caught by a route, send it to
default
.
receiver:
default
# All the above attributes are inherited by all child routes and can
# overwritten on each.
# The child route trees.
routes:
- match:
severity: critical
receiver: email_alert
receivers:
- name:
'default'
email_configs:
- to :
'lihaichun@zhixueyun.com,zhujun@zhixueyun.com,ouyangluping@zhixueyun.com,tangjie@zhixueyun.com'
send_resolved:
true
- name:
'email_alert'
email_configs:
- to :
'lihaichun@zhixueyun.com,zhujun@zhixueyun.com,ouyangluping@zhixueyun.com,tangjie@zhixueyun.com'
send_resolved:
true
templates: []
|
2./app/prometheus-operator/bundle.yaml的内容如下
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus-operator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus-operator
subjects:
- kind: ServiceAccount
name: prometheus-operator
namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus-operator
rules:
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
-
'*'
- apiGroups:
- monitoring.coreos.com
resources:
- alertmanagers
- prometheuses
- prometheuses/finalizers
- alertmanagers/finalizers
- servicemonitors
- prometheusrules
verbs:
-
'*'
- apiGroups:
- apps
resources:
- statefulsets
verbs:
-
'*'
- apiGroups:
-
""
resources:
- configmaps
- secrets
verbs:
-
'*'
- apiGroups:
-
""
resources:
- pods
verbs:
- list
- delete
- apiGroups:
-
""
resources:
- services
- endpoints
verbs:
- get
- create
- update
- apiGroups:
-
""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
-
""
resources:
- namespaces
verbs:
- get
- list
- watch
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
labels:
k8s-app: prometheus-operator
name: prometheus-operator
namespace: monitoring
spec:
replicas:
1
selector:
matchLabels:
k8s-app: prometheus-operator
template:
metadata:
labels:
k8s-app: prometheus-operator
spec:
containers:
- args:
- --kubelet-service=kube-system/kubelet
- --logtostderr=
true
- --config-reloader-image=quay.io/coreos/configmap-reload:v0.
0.1
- --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.
27.0
image: quay.io/coreos/prometheus-operator:v0.
27.0
name: prometheus-operator
ports:
- containerPort:
8080
name: http
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
allowPrivilegeEscalation:
false
readOnlyRootFilesystem:
true
nodeSelector:
beta.kubernetes.io/os: linux
securityContext:
runAsNonRoot:
true
runAsUser:
65534
serviceAccountName: prometheus-operator
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus-operator
namespace: monitoring
|
3./app/prometheus-operator/manifests文件夹的内容下
[root
@iZbp1at8fph52evh70atb1Z
prometheus-operator]# pwd
/app/prometheus-operator
[root
@iZbp1at8fph52evh70atb1Z
prometheus-operator]# ls
alertmanager.yaml bundle.yaml manifests
[root
@iZbp1at8fph52evh70atb1Z
prometheus-operator]# cd manifests/
[root
@iZbp1at8fph52evh70atb1Z
manifests]# ls
alertmanager-alertmanager.yaml kube-state-metrics-service.yaml prometheus-clusterRoleBinding.yaml
alertmanager-serviceAccount.yaml node-exporter-clusterRoleBinding.yaml prometheus-clusterRole.yaml
alertmanager-serviceMonitor.yaml node-exporter-clusterRole.yaml prometheus-prometheus.yaml
alertmanager-service.yaml node-exporter-daemonset.yaml prometheus-roleBindingConfig.yaml
grafana-dashboardDatasources.yaml node-exporter-serviceAccount.yaml prometheus-roleBindingSpecificNamespaces.yaml
grafana-dashboardDefinitions.yaml node-exporter-serviceMonitor.yaml prometheus-roleConfig.yaml
grafana-dashboardSources.yaml node-exporter-service.yaml prometheus-roleSpecificNamespaces.yaml
grafana-deployment.yaml prometheus-rules.yaml
grafana-serviceAccount.yaml prometheus-adapter-clusterRoleBindingDelegator.yaml prometheus-serviceAccount.yaml
grafana-service.yaml prometheus-adapter-clusterRoleBinding.yaml prometheus-serviceMonitorApiserver.yaml
kube-state-metrics-clusterRoleBinding.yaml prometheus-adapter-clusterRoleServerResources.yaml prometheus-serviceMonitorCoreDNS.yaml
kube-state-metrics-clusterRole.yaml prometheus-adapter-clusterRole.yaml prometheus-serviceMonitorKubeControllerManager.yaml
kube-state-metrics-deployment.yaml prometheus-adapter-configMap.yaml prometheus-serviceMonitorKubelet.yaml
kube-state-metrics-roleBinding.yaml prometheus-adapter-deployment.yaml prometheus-serviceMonitorKubeScheduler.yaml
kube-state-metrics-role.yaml prometheus-adapter-roleBindingAuthReader.yaml prometheus-serviceMonitor.yaml
kube-state-metrics-serviceAccount.yaml prometheus-adapter-serviceAccount.yaml prometheus-service.yaml
kube-state-metrics-serviceMonitor.yaml prometheus-adapter-service.yaml
|
4./app/prometheus-operator/manifests/prometheus-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
prometheus: k8s
role: alert-rules
name: prometheus-k8s-rules
namespace: monitoring
spec:
groups:
- name: k8s.rules
rules:
- expr: |
sum(rate(container_cpu_usage_seconds_total{job=
"kubelet"
, image!=
""
, container_name!=
""
}[5m])) by (namespace)
record: namespace:container_cpu_usage_seconds_total:sum_rate
- expr: |
sum by (namespace, pod_name, container_name) (
rate(container_cpu_usage_seconds_total{job=
"kubelet"
, image!=
""
, container_name!=
""
}[5m])
)
record: namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate
- expr: |
sum(container_memory_usage_bytes{job=
"kubelet"
, image!=
""
, container_name!=
""
}) by (namespace)
record: namespace:container_memory_usage_bytes:sum
- expr: |
sum by (namespace, label_name) (
sum(rate(container_cpu_usage_seconds_total{job=
"kubelet"
, image!=
""
, container_name!=
""
}[5m])) by (namespace, pod_name)
* on (namespace, pod_name) group_left(label_name)
label_replace(kube_pod_labels{job=
"kube-state-metrics"
},
"pod_name"
,
"$1"
,
"pod"
,
"(.*)"
)
)
record: namespace_name:container_cpu_usage_seconds_total:sum_rate
- expr: |
sum by (namespace, label_name) (
sum(container_memory_usage_bytes{job=
"kubelet"
,image!=
""
, container_name!=
""
}) by (pod_name, namespace)
* on (namespace, pod_name) group_left(label_name)
label_replace(kube_pod_labels{job=
"kube-state-metrics"
},
"pod_name"
,
"$1"
,
"pod"
,
"(.*)"
)
)
record: namespace_name:container_memory_usage_bytes:sum
- expr: |
sum by (namespace, label_name) (
sum(kube_pod_container_resource_requests_memory_bytes{job=
"kube-state-metrics"
}) by (namespace, pod)
* on (namespace, pod) group_left(label_name)
label_replace(kube_pod_labels{job=
"kube-state-metrics"
},
"pod_name"
,
"$1"
,
"pod"
,
"(.*)"
)
)
record: namespace_name:kube_pod_container_resource_requests_memory_bytes:sum
- expr: |
sum by (namespace, label_name) (
sum(kube_pod_container_resource_requests_cpu_cores{job=
"kube-state-metrics"
} and on(pod) kube_pod_status_scheduled{condition=
"true"
}) by (namespace, pod)
* on (namespace, pod) group_left(label_name)
label_replace(kube_pod_labels{job=
"kube-state-metrics"
},
"pod_name"
,
"$1"
,
"pod"
,
"(.*)"
)
)
record: namespace_name:kube_pod_container_resource_requests_cpu_cores:sum
- name: kube-scheduler.rules
rules:
- expr: |
histogram_quantile(
0.99
, sum(rate(scheduler_e2e_scheduling_latency_microseconds_bucket{job=
"kube-scheduler"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.99"
record: cluster_quantile:scheduler_e2e_scheduling_latency:histogram_quantile
- expr: |
histogram_quantile(
0.99
, sum(rate(scheduler_scheduling_algorithm_latency_microseconds_bucket{job=
"kube-scheduler"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.99"
record: cluster_quantile:scheduler_scheduling_algorithm_latency:histogram_quantile
- expr: |
histogram_quantile(
0.99
, sum(rate(scheduler_binding_latency_microseconds_bucket{job=
"kube-scheduler"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.99"
record: cluster_quantile:scheduler_binding_latency:histogram_quantile
- expr: |
histogram_quantile(
0.9
, sum(rate(scheduler_e2e_scheduling_latency_microseconds_bucket{job=
"kube-scheduler"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.9"
record: cluster_quantile:scheduler_e2e_scheduling_latency:histogram_quantile
- expr: |
histogram_quantile(
0.9
, sum(rate(scheduler_scheduling_algorithm_latency_microseconds_bucket{job=
"kube-scheduler"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.9"
record: cluster_quantile:scheduler_scheduling_algorithm_latency:histogram_quantile
- expr: |
histogram_quantile(
0.9
, sum(rate(scheduler_binding_latency_microseconds_bucket{job=
"kube-scheduler"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.9"
record: cluster_quantile:scheduler_binding_latency:histogram_quantile
- expr: |
histogram_quantile(
0.5
, sum(rate(scheduler_e2e_scheduling_latency_microseconds_bucket{job=
"kube-scheduler"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.5"
record: cluster_quantile:scheduler_e2e_scheduling_latency:histogram_quantile
- expr: |
histogram_quantile(
0.5
, sum(rate(scheduler_scheduling_algorithm_latency_microseconds_bucket{job=
"kube-scheduler"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.5"
record: cluster_quantile:scheduler_scheduling_algorithm_latency:histogram_quantile
- expr: |
histogram_quantile(
0.5
, sum(rate(scheduler_binding_latency_microseconds_bucket{job=
"kube-scheduler"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.5"
record: cluster_quantile:scheduler_binding_latency:histogram_quantile
- name: kube-apiserver.rules
rules:
- expr: |
histogram_quantile(
0.99
, sum(rate(apiserver_request_latencies_bucket{job=
"apiserver"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.99"
record: cluster_quantile:apiserver_request_latencies:histogram_quantile
- expr: |
histogram_quantile(
0.9
, sum(rate(apiserver_request_latencies_bucket{job=
"apiserver"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.9"
record: cluster_quantile:apiserver_request_latencies:histogram_quantile
- expr: |
histogram_quantile(
0.5
, sum(rate(apiserver_request_latencies_bucket{job=
"apiserver"
}[5m])) without(instance, pod)) / 1e+
06
labels:
quantile:
"0.5"
record: cluster_quantile:apiserver_request_latencies:histogram_quantile
- name: node.rules
rules:
- expr: sum(min(kube_pod_info) by (node))
record:
':kube_pod_info_node_count:'
- expr: |
max(label_replace(kube_pod_info{job=
"kube-state-metrics"
},
"pod"
,
"$1"
,
"pod"
,
"(.*)"
)) by (node, namespace, pod)
record:
'node_namespace_pod:kube_pod_info:'
- expr: |
count by (node) (sum by (node, cpu) (
node_cpu_seconds_total{job=
"node-exporter"
}
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
))
record: node:node_num_cpu:sum
- expr: |
1
- avg(rate(node_cpu_seconds_total{job=
"node-exporter"
,mode=
"idle"
}[1m]))
record: :node_cpu_utilisation:avg1m
- expr: |
1
- avg by (node) (
rate(node_cpu_seconds_total{job=
"node-exporter"
,mode=
"idle"
}[1m])
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:)
record: node:node_cpu_utilisation:avg1m
- expr: |
sum(node_load1{job=
"node-exporter"
})
/
sum(node:node_num_cpu:sum)
record:
':node_cpu_saturation_load1:'
- expr: |
sum by (node) (
node_load1{job=
"node-exporter"
}
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
/
node:node_num_cpu:sum
record:
'node:node_cpu_saturation_load1:'
- expr: |
1
-
sum(node_memory_MemFree_bytes{job=
"node-exporter"
} + node_memory_Cached_bytes{job=
"node-exporter"
} + node_memory_Buffers_bytes{job=
"node-exporter"
})
/
sum(node_memory_MemTotal_bytes{job=
"node-exporter"
})
record:
':node_memory_utilisation:'
- expr: |
sum(node_memory_MemFree_bytes{job=
"node-exporter"
} + node_memory_Cached_bytes{job=
"node-exporter"
} + node_memory_Buffers_bytes{job=
"node-exporter"
})
record: :node_memory_MemFreeCachedBuffers_bytes:sum
- expr: |
sum(node_memory_MemTotal_bytes{job=
"node-exporter"
})
record: :node_memory_MemTotal_bytes:sum
- expr: |
sum by (node) (
(node_memory_MemFree_bytes{job=
"node-exporter"
} + node_memory_Cached_bytes{job=
"node-exporter"
} + node_memory_Buffers_bytes{job=
"node-exporter"
})
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
record: node:node_memory_bytes_available:sum
- expr: |
sum by (node) (
node_memory_MemTotal_bytes{job=
"node-exporter"
}
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
record: node:node_memory_bytes_total:sum
- expr: |
(node:node_memory_bytes_total:sum - node:node_memory_bytes_available:sum)
/
scalar(sum(node:node_memory_bytes_total:sum))
record: node:node_memory_utilisation:ratio
- expr: |
1e3 * sum(
(rate(node_vmstat_pgpgin{job=
"node-exporter"
}[1m])
+ rate(node_vmstat_pgpgout{job=
"node-exporter"
}[1m]))
)
record: :node_memory_swap_io_bytes:sum_rate
- expr: |
1
-
sum by (node) (
(node_memory_MemFree_bytes{job=
"node-exporter"
} + node_memory_Cached_bytes{job=
"node-exporter"
} + node_memory_Buffers_bytes{job=
"node-exporter"
})
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
/
sum by (node) (
node_memory_MemTotal_bytes{job=
"node-exporter"
}
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
record:
'node:node_memory_utilisation:'
- expr: |
1
- (node:node_memory_bytes_available:sum / node:node_memory_bytes_total:sum)
record:
'node:node_memory_utilisation_2:'
- expr: |
1e3 * sum by (node) (
(rate(node_vmstat_pgpgin{job=
"node-exporter"
}[1m])
+ rate(node_vmstat_pgpgout{job=
"node-exporter"
}[1m]))
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
record: node:node_memory_swap_io_bytes:sum_rate
- expr: |
avg(irate(node_disk_io_time_seconds_total{job=
"node-exporter"
,device=~
"nvme.+|rbd.+|sd.+|vd.+|xvd.+"
}[1m]))
record: :node_disk_utilisation:avg_irate
- expr: |
avg by (node) (
irate(node_disk_io_time_seconds_total{job=
"node-exporter"
,device=~
"nvme.+|rbd.+|sd.+|vd.+|xvd.+"
}[1m])
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
record: node:node_disk_utilisation:avg_irate
- expr: |
avg(irate(node_disk_io_time_weighted_seconds_total{job=
"node-exporter"
,device=~
"nvme.+|rbd.+|sd.+|vd.+|xvd.+"
}[1m]) / 1e3)
record: :node_disk_saturation:avg_irate
- expr: |
avg by (node) (
irate(node_disk_io_time_weighted_seconds_total{job=
"node-exporter"
,device=~
"nvme.+|rbd.+|sd.+|vd.+|xvd.+"
}[1m]) / 1e3
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
record: node:node_disk_saturation:avg_irate
- expr: |
max by (namespace, pod, device) ((node_filesystem_size_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
}
- node_filesystem_avail_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
})
/ node_filesystem_size_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
})
record:
'node:node_filesystem_usage:'
- expr: |
max by (namespace, pod, device) (node_filesystem_avail_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
} / node_filesystem_size_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
})
record:
'node:node_filesystem_avail:'
- expr: |
sum(irate(node_network_receive_bytes_total{job=
"node-exporter"
,device=
"eth0"
}[1m])) +
sum(irate(node_network_transmit_bytes_total{job=
"node-exporter"
,device=
"eth0"
}[1m]))
record: :node_net_utilisation:sum_irate
- expr: |
sum by (node) (
(irate(node_network_receive_bytes_total{job=
"node-exporter"
,device=
"eth0"
}[1m]) +
irate(node_network_transmit_bytes_total{job=
"node-exporter"
,device=
"eth0"
}[1m]))
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
record: node:node_net_utilisation:sum_irate
- expr: |
sum(irate(node_network_receive_drop_total{job=
"node-exporter"
,device=
"eth0"
}[1m])) +
sum(irate(node_network_transmit_drop_total{job=
"node-exporter"
,device=
"eth0"
}[1m]))
record: :node_net_saturation:sum_irate
- expr: |
sum by (node) (
(irate(node_network_receive_drop_total{job=
"node-exporter"
,device=
"eth0"
}[1m]) +
irate(node_network_transmit_drop_total{job=
"node-exporter"
,device=
"eth0"
}[1m]))
* on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:
)
record: node:node_net_saturation:sum_irate
- name: kube-prometheus-node-recording.rules
rules:
- expr: sum(rate(node_cpu{mode!=
"idle"
,mode!=
"iowait"
}[3m])) BY (instance)
record: instance:node_cpu:rate:sum
- expr: sum((node_filesystem_size{mountpoint=
"/"
} - node_filesystem_free{mountpoint=
"/"
}))
BY (instance)
record: instance:node_filesystem_usage:sum
- expr: sum(rate(node_network_receive_bytes[3m])) BY (instance)
record: instance:node_network_receive_bytes:rate:sum
- expr: sum(rate(node_network_transmit_bytes[3m])) BY (instance)
record: instance:node_network_transmit_bytes:rate:sum
- expr: sum(rate(node_cpu{mode!=
"idle"
,mode!=
"iowait"
}[5m])) WITHOUT (cpu, mode)
/ ON(instance) GROUP_LEFT() count(sum(node_cpu) BY (instance, cpu)) BY (instance)
record: instance:node_cpu:ratio
- expr: sum(rate(node_cpu{mode!=
"idle"
,mode!=
"iowait"
}[5m]))
record: cluster:node_cpu:sum_rate5m
- expr: cluster:node_cpu:rate5m / count(sum(node_cpu) BY (instance, cpu))
record: cluster:node_cpu:ratio
- name: kubernetes-absent
rules:
- alert: AlertmanagerDown
annotations:
message: k8s-master-
10.80
.
154.143
Alertmanager has disappeared from Prometheus target discovery.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-alertmanagerdown
expr: |
absent(up{job=
"alertmanager-main"
} ==
1
)
for
: 1m
labels:
severity: critical
- alert: KubeAPIDown
annotations:
message: k8s-master-
10.80
.
154.143
KubeAPI has disappeared from Prometheus target discovery.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapidown
expr: |
absent(up{job=
"apiserver"
} ==
1
)
for
: 1m
labels:
severity: critical
- alert: KubeStateMetricsDown
annotations:
message: k8s-master-
10.80
.
154.143
KubeStateMetrics has disappeared from Prometheus target discovery.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatemetricsdown
expr: |
absent(up{job=
"kube-state-metrics"
} ==
1
)
for
: 1m
labels:
severity: critical
- alert: KubeletDown
annotations:
message: k8s-master-
10.80
.
154.143
Kubelet has disappeared from Prometheus target discovery.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletdown
expr: |
absent(up{job=
"kubelet"
} ==
1
)
for
: 1m
labels:
severity: critical
- alert: NodeExporterDown
annotations:
message: k8s-master-
10.80
.
154.143
NodeExporter has disappeared from Prometheus target discovery.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-nodeexporterdown
expr: |
absent(up{job=
"node-exporter"
} ==
1
)
for
: 1m
labels:
severity: critical
- alert: PrometheusDown
annotations:
message: k8s-master-
10.80
.
154.143
Prometheus has disappeared from Prometheus target discovery.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-prometheusdown
expr: |
absent(up{job=
"prometheus-k8s"
} ==
1
)
for
: 1m
labels:
severity: critical
- name: kubernetes-apps
rules:
- alert: KubePodCrashLooping
annotations:
message: k8s-master-
10.80
.
154.143
Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container
}}) is restarting {{ printf
"%.2f"
$value }} times /
5
minutes.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping
expr: |
rate(kube_pod_container_status_restarts_total{job=
"kube-state-metrics"
}[15m]) *
60
*
5
>
0
for
: 1m
labels:
severity: critical
- alert: KubePodNotReady
annotations:
message: k8s-master-
10.80
.
154.143
Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready
state
for
longer than an hour.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready
expr: |
sum by (namespace, pod) (kube_pod_status_phase{job=
"kube-state-metrics"
, phase=~
"Pending|Unknown"
}) >
0
for
: 1m
labels:
severity: critical
- alert: KubeDeploymentGenerationMismatch
annotations:
message: k8s-master-
10.80
.
154.143
Deployment generation
for
{{ $labels.namespace }}/{{ $labels.deployment
}} does not match,
this
indicates that the Deployment has failed but has
not been rolled back.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentgenerationmismatch
expr: |
kube_deployment_status_observed_generation{job=
"kube-state-metrics"
}
!=
kube_deployment_metadata_generation{job=
"kube-state-metrics"
}
for
: 1m
labels:
severity: critical
- alert: KubeDeploymentReplicasMismatch
annotations:
message: k8s-master-
10.80
.
154.143
Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not
matched the expected number of replicas
for
longer than an hour.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedeploymentreplicasmismatch
expr: |
kube_deployment_spec_replicas{job=
"kube-state-metrics"
}
!=
kube_deployment_status_replicas_available{job=
"kube-state-metrics"
}
for
: 1m
labels:
severity: critical
- alert: KubeStatefulSetReplicasMismatch
annotations:
message: k8s-master-
10.80
.
154.143
StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has
not matched the expected number of replicas
for
longer than
15
minutes.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetreplicasmismatch
expr: |
kube_statefulset_status_replicas_ready{job=
"kube-state-metrics"
}
!=
kube_statefulset_status_replicas{job=
"kube-state-metrics"
}
for
: 1m
labels:
severity: critical
- alert: KubeStatefulSetGenerationMismatch
annotations:
message: k8s-master-
10.80
.
154.143
StatefulSet generation
for
{{ $labels.namespace }}/{{ $labels.statefulset
}} does not match,
this
indicates that the StatefulSet has failed but has
not been rolled back.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetgenerationmismatch
expr: |
kube_statefulset_status_observed_generation{job=
"kube-state-metrics"
}
!=
kube_statefulset_metadata_generation{job=
"kube-state-metrics"
}
for
: 1m
labels:
severity: critical
- alert: KubeStatefulSetUpdateNotRolledOut
annotations:
message: k8s-master-
10.80
.
154.143
StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update
has not been rolled out.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetupdatenotrolledout
expr: |
max without (revision) (
kube_statefulset_status_current_revision{job=
"kube-state-metrics"
}
unless
kube_statefulset_status_update_revision{job=
"kube-state-metrics"
}
)
*
(
kube_statefulset_replicas{job=
"kube-state-metrics"
}
!=
kube_statefulset_status_replicas_updated{job=
"kube-state-metrics"
}
)
for
: 1m
labels:
severity: critical
- alert: KubeDaemonSetRolloutStuck
annotations:
message: k8s-master-
10.80
.
154.143
Only {{ $value }}% of the desired Pods of DaemonSet {{ $labels.namespace
}}/{{ $labels.daemonset }} are scheduled and ready.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetrolloutstuck
expr: |
kube_daemonset_status_number_ready{job=
"kube-state-metrics"
}
/
kube_daemonset_status_desired_number_scheduled{job=
"kube-state-metrics"
} *
100
<
100
for
: 1m
labels:
severity: critical
- alert: KubeDaemonSetNotScheduled
annotations:
message: k8s-master-
10.80
.
154.143
'{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset
}} are not scheduled.'
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetnotscheduled
expr: |
kube_daemonset_status_desired_number_scheduled{job=
"kube-state-metrics"
}
-
kube_daemonset_status_current_number_scheduled{job=
"kube-state-metrics"
} >
0
for
: 1m
labels:
severity: warning
- alert: KubeDaemonSetMisScheduled
annotations:
message: k8s-master-
10.80
.
154.143
'{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset
}} are running where they are not supposed to run.'
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubedaemonsetmisscheduled
expr: |
kube_daemonset_status_number_misscheduled{job=
"kube-state-metrics"
} >
0
for
: 1m
labels:
severity: warning
- alert: KubeCronJobRunning
annotations:
message: k8s-master-
10.80
.
154.143
CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} is taking more
than 1h to complete.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecronjobrunning
expr: |
time() - kube_cronjob_next_schedule_time{job=
"kube-state-metrics"
} >
3600
for
: 1m
labels:
severity: warning
- alert: KubeJobCompletion
annotations:
message: k8s-master-
10.80
.
154.143
Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more
than one hour to complete.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobcompletion
expr: |
kube_job_spec_completions{job=
"kube-state-metrics"
} - kube_job_status_succeeded{job=
"kube-state-metrics"
} >
0
for
: 1m
labels:
severity: warning
- alert: KubeJobFailed
annotations:
message: k8s-master-
10.80
.
154.143
Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobfailed
expr: |
kube_job_status_failed{job=
"kube-state-metrics"
} >
0
for
: 1m
labels:
severity: warning
- name: kubernetes-resources
rules:
- alert: KubeCPUOvercommit
annotations:
message: k8s-master-
10.80
.
154.143
'Cluster has overcommitted CPU resource requests for Pods and cannot tolerate node failure. '
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit
expr: |
sum(namespace_name:kube_pod_container_resource_requests_cpu_cores:sum)
/
sum(node:node_num_cpu:sum)
>
(count(node:node_num_cpu:sum)-
1
) / count(node:node_num_cpu:sum)
for
: 1m
labels:
severity: info
- alert: zxyKubeCPUOvercommit
annotations:
message: k8s-master-
10.80
.
154.143
'容器的CPU使用率大于100% ,当前值为{{ printf "%0.0f" $value }}% in namespace {{$labels.namespace }} for container {{ $labels.container }} in pod {{ $labels.pod}}.'
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit
expr: |
round(
100
* label_join(label_join(sum(rate(container_cpu_usage_seconds_total{container_name !=
"POD"
, image !=
""
}[1m])) by (pod_name, container_name, namespace) ,
"pod"
,
""
,
"pod_name"
),
"container"
,
""
,
"container_name"
)
/
ignoring(container_name, pod_name) avg(kube_pod_container_resource_limits_cpu_cores) by (pod, container, namespace))
>
100
for
: 1m
labels:
severity: critical
- alert: zxyKubeMemoryOvercommit
annotations:
message: k8s-master-
10.80
.
154.143
'容器的内存使用率大于100% ,当前值为{{ printf "%0.0f" $value }}% in namespace {{$labels.namespace }} for container {{ $labels.container }} in pod {{ $labels.pod}}.'
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit
expr: |
round(
100
* label_join(label_join(sum(container_memory_usage_bytes{container_name !=
"POD"
, image !=
""
}) by (container_name, pod_name, namespace),
"pod"
,
""
,
"pod_name"
),
"container"
,
""
,
"container_name"
)
/
ignoring(container_name, pod_name) avg(kube_pod_container_resource_limits_memory_bytes) by (container, pod, namespace))
>
100
for
: 1m
labels:
severity: critical
- alert: KubeMemOvercommit
annotations:
message: k8s-master-
10.80
.
154.143
Cluster has overcommitted memory resource requests
for
Pods and cannot
tolerate node failure.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememovercommit
expr: |
sum(namespace_name:kube_pod_container_resource_requests_memory_bytes:sum)
/
sum(node_memory_MemTotal_bytes)
>
(count(node:node_num_cpu:sum)-
1
)
/
count(node:node_num_cpu:sum)
for
: 1m
labels:
severity: warning
- alert: KubeCPUOvercommit
annotations:
message: k8s-master-
10.80
.
154.143
Cluster has overcommitted CPU resource requests
for
Namespaces.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit
expr: |
sum(kube_resourcequota{job=
"kube-state-metrics"
, type=
"hard"
, resource=
"requests.cpu"
})
/
sum(node:node_num_cpu:sum)
>
1.5
for
: 1m
labels:
severity: warning
- alert: KubeMemOvercommit
annotations:
message: k8s-master-
10.80
.
154.143
Cluster has overcommitted memory resource requests
for
Namespaces.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememovercommit
expr: |
sum(kube_resourcequota{job=
"kube-state-metrics"
, type=
"hard"
, resource=
"requests.memory"
})
/
sum(node_memory_MemTotal_bytes{job=
"node-exporter"
})
>
1.5
for
: 1m
labels:
severity: warning
- alert: KubeQuotaExceeded
annotations:
message: k8s-master-
10.80
.
154.143
Namespace {{ $labels.namespace }} is using {{ printf
"%0.0f"
$value
}}% of its {{ $labels.resource }} quota.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubequotaexceeded
expr: |
100
* kube_resourcequota{job=
"kube-state-metrics"
, type=
"used"
}
/ ignoring(instance, job, type)
(kube_resourcequota{job=
"kube-state-metrics"
, type=
"hard"
} >
0
)
>
90
for
: 1m
labels:
severity: warning
- alert: CPUThrottlingHigh
annotations:
message: k8s-master-
10.80
.
154.143
'{{ printf
"%0.0f"
$value }}% throttling of CPU in namespace {{ $labels.namespace
}}
for
container {{ $labels.container_name }} in pod {{ $labels.pod_name
}}.'
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-cputhrottlinghigh
expr: "
100
* sum(increase(container_cpu_cfs_throttled_periods_total{}[5m]))
by (container_name, pod_name, namespace) \n / \nsum(increase(container_cpu_cfs_periods_total{}[5m]))
by (container_name, pod_name, namespace)\n >
99
\n"
for
: 1m
labels:
severity: warning
- name: kubernetes-storage
rules:
- alert: KubePersistentVolumeUsageCritical
annotations:
message: k8s-master-
10.80
.
154.143
The PersistentVolume claimed by {{ $labels.persistentvolumeclaim
}} in Namespace {{ $labels.namespace }} is only {{ printf
"%0.2f"
$value
}}% free.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumeusagecritical
expr: |
100
* kubelet_volume_stats_available_bytes{job=
"kubelet"
}
/
kubelet_volume_stats_capacity_bytes{job=
"kubelet"
}
<
3
for
: 1m
labels:
severity: critical
- alert: KubePersistentVolumeFullInFourDays
annotations:
message: k8s-master-
10.80
.
154.143
Based on recent sampling, the PersistentVolume claimed by {{ $labels.persistentvolumeclaim
}} in Namespace {{ $labels.namespace }} is expected to fill up within four
days. Currently {{ printf
"%0.2f"
$value }}% is available.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumefullinfourdays
expr: |
100
* (
kubelet_volume_stats_available_bytes{job=
"kubelet"
}
/
kubelet_volume_stats_capacity_bytes{job=
"kubelet"
}
) <
15
and
predict_linear(kubelet_volume_stats_available_bytes{job=
"kubelet"
}[6h],
4
*
24
*
3600
) <
0
for
: 1m
labels:
severity: critical
- alert: KubePersistentVolumeErrors
annotations:
message: k8s-master-
10.80
.
154.143
The persistent volume {{ $labels.persistentvolume }} has status {{
$labels.phase }}.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepersistentvolumeerrors
expr: |
kube_persistentvolume_status_phase{phase=~
"Failed|Pending"
,job=
"kube-state-metrics"
} >
0
for
: 1m
labels:
severity: critical
- name: kubernetes-system
rules:
- alert: KubeNodeNotReady
annotations:
message: k8s-master-
10.80
.
154.143
'{{ $labels.node }} has been unready for more than an hour.'
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubenodenotready
expr: |
kube_node_status_condition{job=
"kube-state-metrics"
,condition=
"Ready"
,status=
"true"
} ==
0
for
: 1m
labels:
severity: warning
- alert: KubeVersionMismatch
annotations:
message: k8s-master-
10.80
.
154.143
There are {{ $value }} different versions of Kubernetes components
running.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeversionmismatch
expr: |
count(count(kubernetes_build_info{job!=
"kube-dns"
}) by (gitVersion)) >
1
for
: 1m
labels:
severity: warning
- alert: KubeClientErrors
annotations:
message: k8s-master-
10.80
.
154.143
Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance
}}
' is experiencing {{ printf "%0.0f" $value }}% errors.'
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclienterrors
expr: |
(sum(rate(rest_client_requests_total{code!~
"2..|404"
}[5m])) by (instance, job)
/
sum(rate(rest_client_requests_total[5m])) by (instance, job))
*
100
>
1
for
: 1m
labels:
severity: warning
- alert: KubeClientErrors
annotations:
message: k8s-master-
10.80
.
154.143
Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance
}}' is experiencing {{ printf
"%0.0f"
$value }} errors / second.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclienterrors
expr: |
sum(rate(ksm_scrape_error_total{job=
"kube-state-metrics"
}[5m])) by (instance, job) >
0.1
for
: 1m
labels:
severity: warning
- alert: KubeletTooManyPods
annotations:
message: k8s-master-
10.80
.
154.143
Kubelet {{ $labels.instance }} is running {{ $value }} Pods, close
to the limit of
110
.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods
expr: |
kubelet_running_pod_count{job=
"kubelet"
} >
110
*
0.9
for
: 1m
labels:
severity: warning
- alert: KubeAPILatencyHigh
annotations:
message: k8s-master-
10.80
.
154.143
The API server has a 99th percentile latency of {{ $value }} seconds
for
{{ $labels.verb }} {{ $labels.resource }}.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh
expr: |
cluster_quantile:apiserver_request_latencies:histogram_quantile{job=
"apiserver"
,quantile=
"0.99"
,subresource!=
"log"
,verb!~
"^(?:LIST|WATCH|WATCHLIST|PROXY|CONNECT)$"
} >
1
for
: 1m
labels:
severity: warning
- alert: KubeAPILatencyHigh
annotations:
message: k8s-master-
10.80
.
154.143
The API server has a 99th percentile latency of {{ $value }} seconds
for
{{ $labels.verb }} {{ $labels.resource }}.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh
expr: |
cluster_quantile:apiserver_request_latencies:histogram_quantile{job=
"apiserver"
,quantile=
"0.99"
,subresource!=
"log"
,verb!~
"^(?:LIST|WATCH|WATCHLIST|PROXY|CONNECT)$"
} >
4
for
: 1m
labels:
severity: critical
- alert: KubeAPIErrorsHigh
annotations:
message: k8s-master-
10.80
.
154.143
API server is returning errors
for
{{ $value }}% of requests.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh
expr: |
sum(rate(apiserver_request_count{job=
"apiserver"
,code=~
"^(?:5..)$"
}[5m])) without(instance, pod)
/
sum(rate(apiserver_request_count{job=
"apiserver"
}[5m])) without(instance, pod) *
100
>
10
for
: 1m
labels:
severity: critical
- alert: KubeAPIErrorsHigh
annotations:
message: k8s-master-
10.80
.
154.143
API server is returning errors
for
{{ $value }}% of requests.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh
expr: |
sum(rate(apiserver_request_count{job=
"apiserver"
,code=~
"^(?:5..)$"
}[5m])) without(instance, pod)
/
sum(rate(apiserver_request_count{job=
"apiserver"
}[5m])) without(instance, pod) *
100
>
5
for
: 1m
labels:
severity: warning
- alert: KubeClientCertificateExpiration
annotations:
message: k8s-master-
10.80
.
154.143
Kubernetes API certificate is expiring in less than
7
days.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration
expr: |
histogram_quantile(
0.01
, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=
"apiserver"
}[5m]))) <
604800
labels:
severity: warning
- alert: KubeClientCertificateExpiration
annotations:
message: k8s-master-
10.80
.
154.143
Kubernetes API certificate is expiring in less than
24
hours.
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration
expr: |
histogram_quantile(
0.01
, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job=
"apiserver"
}[5m]))) <
86400
labels:
severity: critical
- name: alertmanager.rules
rules:
- alert: AlertmanagerConfigInconsistent
annotations:
message: k8s-master-
10.80
.
154.143
The configuration of the instances of the Alertmanager cluster `{{$labels.service}}`
are out of sync.
expr: |
count_values(
"config_hash"
, alertmanager_config_hash{job=
"alertmanager-main"
}) BY (service) / ON(service) GROUP_LEFT() label_replace(prometheus_operator_spec_replicas{job=
"prometheus-operator"
,controller=
"alertmanager"
},
"service"
,
"alertmanager-$1"
,
"name"
,
"(.*)"
) !=
1
for
: 1m
labels:
severity: critical
- alert: AlertmanagerFailedReload
annotations:
message: k8s-master-
10.80
.
154.143
Reloading Alertmanager's configuration has failed
for
{{ $labels.namespace
}}/{{ $labels.pod}}.
expr: |
alertmanager_config_last_reload_successful{job=
"alertmanager-main"
} ==
0
for
: 1m
labels:
severity: warning
- alert: AlertmanagerMembersInconsistent
annotations:
message: k8s-master-
10.80
.
154.143
Alertmanager has not found all other members of the cluster.
expr: |
alertmanager_cluster_members{job=
"alertmanager-main"
}
!= on (service) GROUP_LEFT()
count by (service) (alertmanager_cluster_members{job=
"alertmanager-main"
})
for
: 1m
labels:
severity: critical
- name: general.rules
rules:
- alert: TargetDown
annotations:
message: k8s-master-
10.80
.
154.143
'{{ $value }}% of the {{ $labels.job }} targets are down.'
expr:
100
* (count(up ==
0
) BY (job) / count(up) BY (job)) >
10
for
: 1m
labels:
severity: warning
- name: kube-prometheus-node-alerting.rules
rules:
- alert: NodeDiskRunningFull
annotations:
message: k8s-master-
10.80
.
154.143
Device {{ $labels.device }} of node-exporter {{ $labels.namespace
}}/{{ $labels.pod }} will be full within the next
24
hours.
expr: |
(node:node_filesystem_usage: >
0.85
) and (predict_linear(node:node_filesystem_avail:[6h],
3600
*
24
) <
0
)
for
: 1m
labels:
severity: warning
- alert: NodeDiskRunningFull
annotations:
message: k8s-master-
10.80
.
154.143
Device {{ $labels.device }} of node-exporter {{ $labels.namespace
}}/{{ $labels.pod }} will be full within the next
2
hours.
expr: |
(node:node_filesystem_usage: >
0.85
) and (predict_linear(node:node_filesystem_avail:[30m],
3600
*
2
) <
0
)
for
: 1m
labels:
severity: critical
- name: prometheus.rules
rules:
- alert: PrometheusConfigReloadFailed
annotations:
description: Reloading Prometheus' configuration has failed
for
{{$labels.namespace}}/{{$labels.pod}}
summary: Reloading Prometheus' configuration failed
expr: |
prometheus_config_last_reload_successful{job=
"prometheus-k8s"
} ==
0
for
: 1m
labels:
severity: warning
- alert: PrometheusNotificationQueueRunningFull
annotations:
description: Prometheus' alert notification queue is running full
for
{{$labels.namespace}}/{{
$labels.pod}}
summary: Prometheus' alert notification queue is running full
expr: |
predict_linear(prometheus_notifications_queue_length{job=
"prometheus-k8s"
}[5m],
60
*
30
) > prometheus_notifications_queue_capacity{job=
"prometheus-k8s"
}
for
: 1m
labels:
severity: warning
- alert: PrometheusErrorSendingAlerts
annotations:
description: Errors
while
sending alerts from Prometheus {{$labels.namespace}}/{{
$labels.pod}} to Alertmanager {{$labels.Alertmanager}}
summary: Errors
while
sending alert from Prometheus
expr: |
rate(prometheus_notifications_errors_total{job=
"prometheus-k8s"
}[5m]) / rate(prometheus_notifications_sent_total{job=
"prometheus-k8s"
}[5m]) >
0.01
for
: 1m
labels:
severity: warning
- alert: PrometheusErrorSendingAlerts
annotations:
description: Errors
while
sending alerts from Prometheus {{$labels.namespace}}/{{
$labels.pod}} to Alertmanager {{$labels.Alertmanager}}
summary: Errors
while
sending alerts from Prometheus
expr: |
rate(prometheus_notifications_errors_total{job=
"prometheus-k8s"
}[5m]) / rate(prometheus_notifications_sent_total{job=
"prometheus-k8s"
}[5m]) >
0.03
for
: 1m
labels:
severity: critical
- alert: PrometheusNotConnectedToAlertmanagers
annotations:
description: Prometheus {{ $labels.namespace }}/{{ $labels.pod}} is not connected
to any Alertmanagers
summary: Prometheus is not connected to any Alertmanagers
expr: |
prometheus_notifications_alertmanagers_discovered{job=
"prometheus-k8s"
} <
1
for
: 1m
labels:
severity: warning
- alert: PrometheusTSDBReloadsFailing
annotations:
description: '{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}}
reload failures over the last four hours.'
summary: Prometheus has issues reloading data blocks from disk
expr: |
increase(prometheus_tsdb_reloads_failures_total{job=
"prometheus-k8s"
}[2h]) >
0
for
: 1m
labels:
severity: warning
- alert: PrometheusTSDBCompactionsFailing
annotations:
description: '{{$labels.job}} at {{$labels.instance}} had {{$value | humanize}}
compaction failures over the last four hours.'
summary: Prometheus has issues compacting sample blocks
expr: |
increase(prometheus_tsdb_compactions_failed_total{job=
"prometheus-k8s"
}[2h]) >
0
for
: 1m
labels:
severity: warning
- alert: PrometheusTSDBWALCorruptions
annotations:
description: '{{$labels.job}} at {{$labels.instance}} has a corrupted write-ahead
log (WAL).'
summary: Prometheus write-ahead log is corrupted
expr: |
tsdb_wal_corruptions_total{job=
"prometheus-k8s"
} >
0
for
: 1m
labels:
severity: warning
- alert: PrometheusNotIngestingSamples
annotations:
description: Prometheus {{ $labels.namespace }}/{{ $labels.pod}} isn't ingesting
samples.
summary: Prometheus isn't ingesting samples
expr: |
rate(prometheus_tsdb_head_samples_appended_total{job=
"prometheus-k8s"
}[5m]) <=
0
for
: 1m
labels:
severity: warning
- alert: PrometheusTargetScrapesDuplicate
annotations:
description: '{{$labels.namespace}}/{{$labels.pod}} has many samples rejected
due to duplicate timestamps but different values'
summary: Prometheus has many samples rejected
expr: |
increase(prometheus_target_scrapes_sample_duplicate_timestamp_total{job=
"prometheus-k8s"
}[5m]) >
0
for
: 1m
labels:
severity: warning
- name: prometheus-operator
rules:
- alert: PrometheusOperatorReconcileErrors
annotations:
message: k8s-master-
10.80
.
154.143
Errors
while
reconciling {{ $labels.controller }} in {{ $labels.namespace
}} Namespace.
expr: |
rate(prometheus_operator_reconcile_errors_total{job=
"prometheus-operator"
}[5m]) >
0.1
for
: 1m
labels:
severity: warning
- alert: PrometheusOperatorNodeLookupErrors
annotations:
message: k8s-master-
10.80
.
154.143
Errors
while
reconciling Prometheus in {{ $labels.namespace }} Namespace.
expr: |
rate(prometheus_operator_node_address_lookup_errors_total{job=
"prometheus-operator"
}[5m]) >
0.1
for
: 1m
labels:
severity: warning
|
for: 1m代表1分钟检查一次，如果是for: 1h代表一个小时检查一次，告警邮件也是1小时发一次
这里特意加了下面2个监控通知，目的是为了当容器的内存与cpu使用率到了85%的给出邮件通知
- alert: zxyKubeCPUOvercommit
annotations:
message:
'容器的CPU使用率大于85% ,当前值为{{ printf "%0.0f" $value }}% in namespace {{$labels.namespace }} for container {{ $labels.container }} in pod {{ $labels.pod}}.'
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit
expr: |
round(
100
* label_join(label_join(sum(rate(container_cpu_usage_seconds_total{container_name !=
"POD"
, image !=
""
}[1m])) by (pod_name, container_name, namespace) ,
"pod"
,
""
,
"pod_name"
),
"container"
,
""
,
"container_name"
)
/
ignoring(container_name, pod_name) avg(kube_pod_container_resource_limits_cpu_cores) by (pod, container, namespace))
>
85
for
: 1m
labels:
severity: critical
- alert: zxyKubeMemoryOvercommit
annotations:
message:
'容器的内存使用率大于85% ,当前值为{{ printf "%0.0f" $value }}% in namespace {{$labels.namespace }} for container {{ $labels.container }} in pod {{ $labels.pod}}.'
runbook_url: https:
//github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit
expr: |
round(
100
* label_join(label_join(sum(container_memory_usage_bytes{container_name !=
"POD"
, image !=
""
}) by (container_name, pod_name, namespace),
"pod"
,
""
,
"pod_name"
),
"container"
,
""
,
"container_name"
)
/
ignoring(container_name, pod_name) avg(kube_pod_container_resource_limits_memory_bytes) by (container, pod, namespace))
>
85
for
: 1m
labels:
severity: critical
|
大家也可以自己定义一些告警，可以参考grafana的取值定义
sum(label_replace(container_memory_usage_bytes{namespace=
"$namespace"
, pod_name=
"$pod"
, container_name!=
"POD"
, container_name!=
""
},
"container"
,
"$1"
,
"container_name"
,
"(.*)"
)) by (container)
sum(kube_pod_container_resource_requests_memory_bytes{namespace=
"$namespace"
, pod=
"$pod"
}) by (container)
sum(label_replace(container_memory_usage_bytes{namespace=
"$namespace"
, pod_name=
"$pod"
},
"container"
,
"$1"
,
"container_name"
,
"(.*)"
)) by (container) / sum(kube_pod_container_resource_requests_memory_bytes{namespace=
"$namespace"
, pod=
"$pod"
}) by (container)
sum(kube_pod_container_resource_limits_memory_bytes{namespace=
"$namespace"
, pod=
"$pod"
, container!=
""
}) by (container)
sum(label_replace(container_memory_usage_bytes{namespace=
"$namespace"
, pod_name=
"$pod"
, container_name!=
""
},
"container"
,
"$1"
,
"container_name"
,
"(.*)"
)) by (container) / sum(kube_pod_container_resource_limits_memory_bytes{namespace=
"$namespace"
, pod=
"$pod"
}) by (container)
round(
100
* label_join(label_join(sum(rate(container_cpu_usage_seconds_total{container_name !=
"POD"
, image !=
""
}[1m])) by (pod_name, container_name, namespace) ,
"pod"
,
""
,
"pod_name"
),
"container"
,
""
,
"container_name"
) / ignoring(container_name, pod_name) avg(kube_pod_container_resource_limits_cpu_cores) by (pod, container, namespace)) >
75
round(
100
* label_join(label_join(sum(container_memory_usage_bytes{container_name !=
"POD"
, image !=
""
}) by (container_name, pod_name, namespace),
"pod"
,
""
,
"pod_name"
),
"container"
,
""
,
"container_name"
) / ignoring(container_name, pod_name) avg(kube_pod_container_resource_limits_memory_bytes) by (container, pod, namespace)) >
75
|
下载地址:
https://zxytest.zhixueyun.com/installer/prometheus-operator.zip
4.启动命令
kubectl create namespace monitoring
kubectl delete secret alertmanager-main -n monitoring
kubectl create secret generic alertmanager-main --from-file=/app/prometheus-operator/alertmanager.yaml -n monitoring
#替换message的开头，使告警信息知道具体是哪个环境的，比如zxy9.zhixueyun.com
sed -i 's/message: /message: zxy9.zhixueyun.com/g' /app/prometheus-operator/prometheus-rules.yaml
#注意要先启动bundle.yaml,否则manifest下面的服务将无法启动
kubectl create -f /app/prometheus-operator/bundle.yaml
kubectl create -f /app/prometheus-operator/manifests
5.删除命令
kubectl delete secret alertmanager-main -n monitoring
kubectl delete -f /app/prometheus-operator/manifests
6.测试
[root
@iZbp1at8fph52evh70atb1Z
manifests]# kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main NodePort
10.254
.
71.140
<none>
9093
:
30093
/TCP 6m55s
alertmanager-operated ClusterIP None <none>
9093
/TCP,
6783
/TCP 6m51s
grafana NodePort
10.254
.
83.196
<none>
3000
:
30000
/TCP 6m55s
kube-state-metrics ClusterIP None <none>
8443
/TCP,
9443
/TCP 6m55s
node-exporter ClusterIP None <none>
9100
/TCP 6m55s
prometheus-adapter ClusterIP
10.254
.
92.97
<none>
443
/TCP 6m55s
prometheus-k8s NodePort
10.254
.
148.92
<none>
9090
:
30001
/TCP 6m55s
prometheus-operated ClusterIP None <none>
9090
/TCP 6m44s
prometheus-operator ClusterIP None <none>
8080
/TCP 7h48m
|
grafana的访问地址:http://120.27.159.108:30000
prometheus-k8s的访问地址:http://120.27.159.108:30001
在Prometheus的Alerts类目中可以查看当前的报警规则，红色的规则表示正在触发报警，绿色的规则表示状态正常，默认prometheus operator会自动创建一批报警规则。
告警邮件
如果需要设置报警压制，需要访问Alter Manager，alertmanager的访问地址:http://120.27.159.108:30093，点击Silence可以设置报警压制的内容。
7.alerts分析，访问http://120.27.159.108:30001/alerts
alert: KubeCPUOvercommit
sum(namespace_name:kube_pod_container_resource_requests_cpu_cores:sum) / sum(node:node_num_cpu:sum) > (count(node:node_num_cpu:sum) - 1) / count(node:node_num_cpu:sum)的值如下,代表所有namespace request cpu总核数／k8s node总核数
sum(namespace_name:kube_pod_container_resource_requests_cpu_cores:sum)的值如下 ，代表所有namespace cpu request总和
namespace_name:kube_pod_container_resource_requests_cpu_cores:sum的值如下，代表每个namespace cpu request总和
kube_pod_container_resource_requests_cpu_cores的值如下，代表每个pod容器cpu资源request值
node:node_num_cpu:sum的值如下,代表每个k8s node的cpu总核数
访问http://120.27.159.108:30001/graph，输入kube_pod_container_resource_limits_memory_bytes,可以查询每个pod的内存limit值
8.磁盘空间告警配置，使用率大于85%告警
- expr: |
max by (namespace, pod, device) ((node_filesystem_size_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
}
- node_filesystem_avail_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
})
/ node_filesystem_size_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
})
record:
'node:node_filesystem_usage:'
- expr: |
max by (namespace, pod, device) (node_filesystem_avail_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
} / node_filesystem_size_bytes{fstype=~
"ext[234]|btrfs|xfs|zfs"
})
record:
'node:node_filesystem_avail:'
- alert: NodeDiskRunningFull
annotations:
message: Device {{ $labels.device }} of node-exporter {{ $labels.namespace
}}/{{ $labels.pod }} will be full within the next
24
hours.
expr: |
(node:node_filesystem_usage: >
0.85
) and (predict_linear(node:node_filesystem_avail:[6h],
3600
*
24
) <
0
)
for
: 30m
labels:
severity: warning
- alert: NodeDiskRunningFull
annotations:
message: Device {{ $labels.device }} of node-exporter {{ $labels.namespace
}}/{{ $labels.pod }} will be full within the next
2
hours.
expr: |
(node:node_filesystem_usage: >
0.85
) and (predict_linear(node:node_filesystem_avail:[30m],
3600
*
2
) <
0
)
for
: 10m
labels:
severity: critical
|
9.如果node-exporter无法启动出现如下错误
[root
@iZbp14qk2dtp82q129jrzqZ
manifests]# kubectl logs node-exporter-9kg72 -n monitoring -c kube-rbac-proxy
I0308
06
:
29
:
35.477100
19438
main.go:
209
] Generating self signed cert as no cert is provided
log: exiting because of error: log: cannot create log: open /tmp/kube-rbac-proxy.iZbp1hkg813np4ep5cuakvZ.unknownuser.log.INFO.
20190308
-
062935.19438
: permission denied
|
则需要修改node-exporter-daemonset.yaml,
runAsNonRoot: false
runAsUser: 0
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
labels:
app: node-exporter
name: node-exporter
namespace: monitoring
spec:
selector:
matchLabels:
app: node-exporter
template:
metadata:
labels:
app: node-exporter
spec:
containers:
- args:
- --web.listen-address=
127.0
.
0.1
:
9100
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/)
- --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
image: quay.io/prometheus/node-exporter:v0.
16.0
name: node-exporter
resources:
limits:
cpu: 250m
memory: 180Mi
requests:
cpu: 102m
memory: 180Mi
volumeMounts:
- mountPath: /host/proc
name: proc
readOnly:
false
- mountPath: /host/sys
name: sys
readOnly:
false
- mountPath: /host/root
mountPropagation: HostToContainer
name: root
readOnly:
true
- args:
- --secure-listen-address=$(IP):
9100
- --upstream=http:
//127.0.0.1:9100/
env:
- name: IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: quay.io/coreos/kube-rbac-proxy:v0.
4.0
name: kube-rbac-proxy
ports:
- containerPort:
9100
hostPort:
9100
name: https
resources:
limits:
cpu: 20m
memory: 40Mi
requests:
cpu: 10m
memory: 20Mi
hostNetwork:
true
hostPID:
true
nodeSelector:
beta.kubernetes.io/os: linux
securityContext:
runAsNonRoot:
false
runAsUser:
0
serviceAccountName: node-exporter
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
volumes:
- hostPath:
path: /proc
name: proc
- hostPath:
path: /sys
name: sys
- hostPath:
path: /
name: root
|
10.alertmanager-alertmanager.yaml,prometheus-prometheus.yaml最好通过nodeName: k8s_master_ip都限制在master节点
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
prometheus: k8s
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- name: alertmanager-main
namespace: monitoring
port: web
baseImage: quay.io/prometheus/prometheus
nodeName:
10.80
.
154.143
#nodeSelector:
#beta.kubernetes.io/os: linux
replicas:
2
resources:
requests:
memory: 600Mi
ruleSelector:
matchLabels:
prometheus: k8s
role: alert-rules
securityContext:
fsGroup:
2000
runAsNonRoot:
true
runAsUser:
1000
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.
5.0
|
11.注意kind: ReplicationController要改成kind: Deployment，ReplicationController中container_memory_usage_bytes{container_name!="POD",image!=""}获取到的pod占用内存不准确，Deployment就准确
比如当kind为ReplicationController时，kubectl top po 获取到该pod的占用内存值与prometheus监测的值对应不上
参考文档:https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus