目錄
一、監控簡介
監控就是實時的幫助我們來監控或者探測我們部署的服務是否正常運行。
常用的監控
- zabbix
- 組件比較全，缺點：性能不高
- 當zabbix監控大型集群的時候，怎么優化
- 當數據庫中一個表中數據超過2000w的時候，數據庫的性能急劇下降
- 阿里雲雲監控
- 騰訊雲藍鯨監控
- 普羅米修斯（prometheus）
- 性能比較高，底層使用（時序數據庫）
- 原生支持監控容器
普羅米修斯（prometheus）
普羅米修斯監控分為兩種情況：
1、攜帶metrics接口的服務（kubernetes、ETCD、Docker）
2、不攜帶metrics接口的服務（Nginx、mysql、Linux主機），針對於不攜帶metrics接口的服務，我們需要安裝一個exporter插件。
二、部署prometheus
1.prometheus主機上部署軟件
# 下載
[root@prometheus opt]# wget https://github.com/prometheus/prometheus/releases/download/v2.27.1/prometheus-2.27.1.linux-amd64.tar.gz
或者 wget https://github.com/prometheus/prometheus/releases/download/v2.25.0/prometheus-2.25.0.linux-amd64.tar.gz
# 解壓
[root@prometheus opt]# tar -xf prometheus-2.27.1.linux-amd64.tar.gz -C /usr/local/
[root@prometheus ~]# cd /usr/local/
# 建立超鏈接
[root@prometheus local]# ln -s /usr/local/prometheus-2.27.1.linux-amd64 /usr/local/prometheus
[root@prometheus local]# ll
lrwxrwxrwx 1 root root 30 Jun 2 09:49 prometheus -> prometheus-2.25.0.linux-amd64/
# 創建環境變量
[root@prometheus local]# cat >> /etc/profile <<EOF
export PROMETHEUS_HOME=/usr/local/prometheus
PATH=$PATH:$PROMETHEUS_HOME
export PATH
EOF
#重載環境變量文件
[root@prometheus ~]# source /etc/profile
# 測試安裝成功
[root@prometheus ~]# prometheus --version
prometheus, version 2.25.0 (branch: HEAD, revision: a6be548dbc17780d562a39c0e4bd0bd4c00ad6e2)
build user: root@615f028225c9
build date: 20210217-14:17:24
go version: go1.15.8
platform: linux/amd64
#查看配置文件（prometheus.yml）
[root@prometheus ~]# cd /usr/local/prometheus
[root@prometheus prometheus]# ll
total 167984
drwxr-xr-x 2 3434 3434 38 Feb 18 00:11 console_libraries
drwxr-xr-x 2 3434 3434 173 Feb 18 00:11 consoles
-rw-r--r-- 1 3434 3434 11357 Feb 18 00:11 LICENSE
-rw-r--r-- 1 3434 3434 3420 Feb 18 00:11 NOTICE
-rwxr-xr-x 1 3434 3434 91044140 Feb 17 22:19 prometheus
-rw-r--r-- 1 3434 3434 1170 Jun 2 18:16 prometheus.yml
-rwxr-xr-x 1 3434 3434 80948693 Feb 17 22:21 promtool
#啟動prometheus
[root@prometheus ~]# prometheus --config.file=/usr/local/prometheus/prometheus.yml
#訪問查看是否監控
http://192.168.15.71:9090/targets
#查看接口數據
http://192.168.15.71:9090/metrics
#創建promethets的systemd啟動文件"
cat >>/usr/lib/systemd/system/prometheus.service <<EOF
[Unit]
Description=https://prometheus.io
[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
[Install]
WantedBy=multi-user.target
EOF
#.啟動promethets"
systemctl daemon-reload
systemctl enable --now prometheus.service
2.使用普羅米修斯監控Linux主機 （web集群）
# 下載
[root@web01 opt]# wget https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz
#解壓
[root@web01 opt]# tar -xf node_exporter-1.1.2.linux-amd64.tar.gz -C /usr/local/
[root@prometheus ~]# cd /usr/local/
#建立軟連接
[root@web01 local]# ln -s /usr/local/node_exporter-1.1.2.linux-amd64/ /usr/local/node_exporter
#添加環境變量
[root@web01 local]# vim /etc/profile
... ...
export NODE_EXPORTER=/usr/local/node_exporter
export PATH=$PATH:$NODE_EXPORTER
#重載配置文件
[root@web01 ~]# source /etc/profile
#啟動prometheus監控（9100端口）
[root@web01 ~]# node_exporter
level=info ts=2021-06-02T07:01:08.509Z caller=node_exporter.go:195 msg="Listening on" address=:9100
#創建systemd服務
cat > /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=This is prometheus node exporter
After=node_exporter.service
[Service]
Type=simple
ExecStart=/usr/local/node_exporter/node_exporter
ExecReload=/bin/kill -HUP
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
#.啟動node_exporter服務"
systemctl daemon-reload
systemctl enable --now node_exporter.service
參數：
--web.listen-address=":9100" #修改默認端口，防止沖突
--web.telemetry-path="/metrics" #獲取metric信息的url，默認是/metrics，若需要修改則通過此參數
--log.level="info" #設置日志級別
--log.format="logger:stderr" #設置打印日志的格式，若有自動化日志提取工具可以使用這個參數規范日志打印的格式
3.將Linux主機的exporter插件添加至Prometheus軟件服務
[root@prometheus ~]# vim /usr/local/prometheus/prometheus.yml
- job_name: 'Linux Node' # 添加監控項的名字
static_configs:
- targets:
- "172.16.1.7:9100" # 監控主機的ip地址和端口號
- "172.16.1.8:9100"
- "172.16.1.9:9100"
#重載prometheus
[root@prometheus ~]# prometheus --config.file=/usr/local/prometheus/prometheus.yml
#訪問查看是否監控
http://192.168.15.71:9090/targets
4.使用Garafana展示數據
Garafana是業內做數據展示挺好的一款產品
[root@prometheus opt]# rz -E rafana-7.3.6-1.x86_64.rpm
[root@prometheus opt]# yum install -y grafana-7.3.6-1.x86_64.rpm
[root@prometheus opt]# systemctl start grafana-server.service
[root@prometheus opt]# netstat -tnlp
tcp6 0 0 :::3000 :::* LISTEN 10137/grafana-serve
#訪問
HTTPS://192.168.15.71:3000
賬號和密碼默認是admin
監控Linux主機單個狀態信息
監控Linux主機整體的狀態信息
Prometheus各種展示頁面狀態碼： https://grafana.com/grafana/dashboards?dataSource=prometheus
5.prometheus監控數據庫
Prometheus監控Mysql數據庫官網教程：https://github.com/prometheus/mysqld_exporter
#數據庫的版本要求：
- mysql >= 5.6
- mariadb >= 10.2
#上傳解壓包
[root@db01 opt]# rz -E
-rw-r--r-- 1 root root 7121565 Apr 22 21:33 mysqld_exporter-0.12.1.linux-amd64.tar.gz
[root@db01 opt]# tar -xf mysqld_exporter-0.12.1.linux-amd64.tar.gz -C /usr/local/
[root@db01 opt]# cd /usr/local/
[root@db01 local]# ll
drwxr-xr-x 2 3434 3434 58 Jul 29 2019 mysqld_exporter-0.12.1.linux-amd64
#做軟連接
[root@db01 local]# ln -s /usr/local/mysqld_exporter-0.12.1.linux-amd64/ /usr/local/mysqld_exporter
#創建用戶並授權以及查看用戶權限
[root@db01 ~]# mysql -uroot -p111
MariaDB [(none)]> create user 'exporter'@'%' identified by '123';
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> grant process, replication client, select on *.* to 'exporter'@'%';
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [mysql]> select host,user from user;
+------------+----------+
| host | user |
+------------+----------+
| % | exporter |
| 127.0.0.1 | root |
| 172.16.1.% | www |
| ::1 | root |
| db01 | |
| db01 | root |
| localhost | |
| localhost | root |
+------------+----------+
8 rows in set (0.00 sec)
#添加文件
[root@db01 mysqld_exporter]# vim .my.cnf
[client]
host=172.16.1.51
user=exporter
password=123
#監控成功（9104端口）
[root@db01 mysqld_exporter]# ./mysqld_exporter --config.my-cnf="/usr/local/mysqld_exporter/.my.cnf"
INFO[0000] Starting mysqld_exporter (version=0.12.1, branch=HEAD, revision=48667bf7c3b438b5e93b259f3d17b70a7c9aff96) source="mysqld_exporter.go:257"
INFO[0000] Build context (go=go1.12.7, user=root@0b3e56a7bc0a, date=20190729-12:35:58) source="mysqld_exporter.go:258"
INFO[0000] Enabled scrapers:
INFO[0000] Listening on :9104 source="mysqld_exporter.go:283"
6.將Mysql主機的exporter插件添加至Prometheus軟件服務
[root@prometheus ~]# vim /usr/local/prometheus/prometheus.yml
- job_name: 'Mysql server'
static_configs:
- targets:
- "172.16.1.51:9104"
#重載prometheus
[root@prometheus ~]# prometheus --config.file=/usr/local/prometheus/prometheus.yml
#創建systemdqldmysqld_exporter.service務"
cat >> /usr/lib/systemd/system/mysqld_exporter.service <<EOF
[Unit]
Description=Prometheus
[Service]
Environment=DATA_SOURCE_NAME=root:123@(172.16.1.18:3306)/
ExecStart=/usr/local/mysqld_exporter/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter/.my.cnf --web.listen-address=:9104
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
#啟動node_exporter服務"
systemctl daemon-reload
systemctl enable --now mysqld_exporter.service
# 普羅米修斯是通過mysql_exporter來找mariadb進行獲取數據的，所以要給它授權（用戶名可自定義，與.my.cnf對應）
# (注意:授權ip為localhost，因為不是prometheus服務器來直接找mariadb 獲取數據，而是prometheus服務器找mysql_exporter,mysql_exporter 再找mariadb。所以這個localhost是指的mysql_exporter的IP)
7.Mysql主機狀態添加至Garafana展示
https://grafana.com/grafana/dashboards
8.redis主機監控
[root@jenkins ~]# git clone https://github.com/oliver006/redis_exporter.git
[root@jenkins ~]# ll
drwxr-xr-x 7 root root 239 Aug 31 11:23 redis_exporter
[root@jenkins ~]# cd redis_exporter/
[root@jenkins redis_exporter]# ls
contrib exporter go.sum main.go README.md
docker go.mod LICENSE Makefile release-github-binaries.sh
[root@jenkins redis_exporter]# export GOPROXY=https://mirrors.aliyun.com/goproxy/ #golang阿里雲代理添加至環境變量
[root@jenkins redis_exporter]# go build . #編譯
[root@jenkins redis_exporter]# ./redis_exporter --version
INFO[0000] Redis Metrics Exporter <<< filled in by build >>> build date: <<< filled in by build >>> sha1: <<< filled in by build >>> Go: go1.15.14 GOOS: linux GOARCH: amd64
[root@jenkins redis_exporter]# ./redis_exporter -redis.addr "redis://192.168.15.14:6379" #監控redis主機
[root@jenkins redis_exporter]# mv redis_exporter /usr/local/bin/
[root@jenkins redis_exporter]# cd /usr/lib/systemd/system
[root@jenkins system]# vim redis-exporter.service
[Unit]
Description=This is prometheus redis exporter
After=node_exporter.service
[Service]
Type=simple
ExecStart=/usr/local/bin/redis_exporter -redis.addr "redis://192.168.15.14:6379"
ExecReload=/bin/kill -HUP
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
[root@jenkins system]# systemctl daemon-reload
[root@jenkins system]# systemctl start redis-exporter.service
[root@jenkins system]# netstat -lnutp
... ...
tcp6 0 0 :::9121 :::* LISTEN 40279/redis_exporte
#去prometheus主機上添加配置
[root@prometheus ~]# vim /usr/local/prometheus/prometheus.yml
- job_name: "RedisDB"
static_configs:
- targets:
- "192.168.15.15:9121"
[root@prometheus ~]# systemctl restart prometheus
攜帶metircs接口的服務流程
1.將外部服務，接入到集群中，需要用到EndPoints
2.使用EndPoints關聯Service
3.使用Service注冊一個ServiceMoniter
4.ServiceMoniter自動注入到k8s集群之中
5.從而實現容器化的Prometheus自動發現
注意：監控ETCD需要證書，掛載到prometheus-k8s容器中
不攜帶metircs接口的服務流程
1.在外部或者容器中，部署exporter，實現一個metircs接口服務
2.將metircs接口服務，接入到集群中，需要用到EndPoints
3.使用EndPoints關聯Service
4.使用Service注冊一個ServiceMoniter
5.ServiceMoniter自動注入到k8s集群之中
6.從而實現容器化的Prometheus自動發現