Prometheus安裝部署及監控linux主機

       Prometheus（普羅米修斯）是一套開源的監控&報警&時間序列數據庫的組合，起始是由SoundCloud公司開發的.Prometheus基本原理是通過HTTP協議周期性抓取被監控組件的狀態，這樣做的好處是任意組件只要提供HTTP接口就可以接入監控系統，不需要任何SDK或者其他的集成過程。這樣做非常適合虛擬化環境比如VM或者Docker 。

Prometheus官網下載地址： https://prometheus.io/download/

grafana官網下載值：https://grafana.com/grafana/download/

本章部署軟件及版本如下：

https://github.com/prometheus/prometheus/releases/download/v2.26.0/prometheus-2.26.0.linux-amd64.tar.gz
https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz
https://github.com/prometheus/alertmanager/releases/download/v0.21.0/alertmanager-0.21.0.linux-amd64.tar.gz
https://dl.grafana.com/oss/release/grafana-7.1.4-1.x86_64.rpm

一、安裝Prometheus

###########################################
#創建用戶組
# groupadd prometheus
# useradd -g prometheus -m -s /sbin/nologin prometheus

# 解壓配置
# tar -zxvf prometheus-2.26.0.linux-amd64.tar.gz
# mv prometheus-2.26.0.linux-amd64 /usr/local/prometheus
# sed -i 's/localhost/10.0.2.118/g' /usr/local/prometheus/prometheus.yml

# 創建數據目錄
$ mkdir /usr/local/prometheus/data/

# 權限修改
# chown -R prometheus.prometheus /usr/local/prometheus

# 檢查並加載配置文件
 [root@localhost prometheus]# ./promtool check config prometheus.yml

# 創建systemd服務

cat <<EOF > /usr/lib/systemd/system/prometheus.service
[Unit]
Description=prometheus
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/usr/local/prometheus/data/ --web.enable-lifecycle --storage.tsdb.retention.time=30d
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

# --config.file: 指定配置文件
# --web.enable-lifecycle: 支持通過http請求重載配置
# --storage.tsdb.path: 指定數據存儲目錄（默認當前目錄的的data目錄，若不存在則新建）
# --storage.tsdb.retention.time: 指定數據保留時間（默認15d）

# 啟動服務
# systemctl daemon-reload
# systemctl start prometheus
# systemctl status prometheus && systemctl enable prometheus

# 確認端口已經被監聽
# ss -lnput | grep 9090
tcp    LISTEN     0      128    [::]:9090               [::]:*                   users:(("prometheus",pid=71025,fd=8))

瀏覽器訪問：

 Prometheus管理接口:

# curl  http://10.0.2.118:9090/-/healthy
Prometheus is Healthy.
# curl  http://10.0.2.118:9090/-/ready
Prometheus is Ready.

# curl -XPOST http://10.0.2.118:9090/-/reload
# 通過web接口重載，啟動時需增加選項  --web.enable-lifecycle

# curl -XPUT http://10.0.2.118:9090/-/quit
# 停止Prometheus
# 同樣需啟動時增加--web.enable-lifecycle選項

二、安裝grafana

# wget https://dl.grafana.com/oss/release/grafana-7.1.4-1.x86_64.rpm
# yum -y install grafana-7.1.4-1.x86_64.rpm
# systemctl daemon-reload 
# systemctl enable grafana-server && systemctl start grafana-server
# ss -lnput | grep 3000
tcp    LISTEN     0      1024     :::3000                 :::*                   users:(("grafana-server",pid=8843,fd=8))

三、grafana添加Prometheus數據源
訪問grafana的3000端口進行登錄，默認用戶名和密碼為: admin/admin。第一次登陸修改更改密碼！

Configuration -> Data Sources -> Prometheus

四、安裝node_exporter
node_exporter用於收集主機運行信息，比如CPU、內存、磁盤等資源使用情況。

各種監控exporter，可以去官網下載

#創建用戶組
# groupadd prometheus
# useradd -g prometheus -m -s /sbin/nologin prometheus

#安裝並解壓
# tar -zxvf node_exporter-1.1.2.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# mv node_exporter-1.1.2.linux-amd64/ node_exporter

# 權限修改
# chown -R prometheus.prometheus /usr/local/node_exporter


# 創建systemd服務
cat > /usr/lib/systemd/system/node_exporter.service <<EOF

[Unit]
Description=node_export
Documentation=https://github.com/prometheus/node_exporter
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/node_exporter/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

# 啟動服務
# systemctl daemon-reload
# systemctl enable node_exporter && systemctl start node_exporter
# systemctl status node_exporter
# ss -lnput | grep 9100
tcp    LISTEN     0      128    [::]:9100               [::]:*                   users:(("node_exporter",pid=78546,fd=3))

五、調整Prometheus配置獲取主機及應用監控數據,添加116、117兩台主機作為agent端.

116/117主機安裝agent客戶端node_exporter,方法同上.

#創建用戶組
# groupadd prometheus
# useradd -g prometheus -m -s /sbin/nologin prometheus

#安裝並解壓
# tar -zxvf node_exporter-1.1.2.linux-amd64.tar.gz -C /usr/local/
# cd /usr/local/
# mv node_exporter-1.1.2.linux-amd64/ node_exporter

# 權限修改
# chown -R prometheus.prometheus /usr/local/node_exporter


# 創建systemd服務
cat > /usr/lib/systemd/system/node_exporter.service <<EOF

[Unit]
Description=node_export
Documentation=https://github.com/prometheus/node_exporter
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/node_exporter/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

# 啟動服務
# systemctl daemon-reload
# systemctl enable node_exporter && systemctl start node_exporter
# systemctl status node_exporter
# ss -lnput | grep 9100
tcp    LISTEN     0      128    [::]:9100               [::]:*                   users:(("node_exporter",pid=78546,fd=3))

調整Prometheus配置，並重啟prometheus服務.

# vim  prometheus.yml
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
    - targets: ['10.0.2.118:9090']

#增加以下配置，其中job_name為自定義
  - job_name: 'kafaka'
    static_configs:
    - targets: ['10.0.2.116:9100','10.0.2.117:9100','10.0.2.118:9100']

確定Prometheus已采集到數據

六、驗證

使用Grafana添加圖形，便於展示！（監控Linux主機推薦使用11074模板）！

 grafana模板地址：點擊查看

Grafana插件擴展之餅圖安裝
有些時候導入的插件里面有餅圖類型，但是Grafana餅圖插件默認並沒有安裝，所以需要自行安裝！（強烈推薦在線安裝，避免版本的問題）！

官網：https://grafana.com/grafana/plugins/grafana-piechart-panel

# grafana-cli plugins install grafana-piechart-panel
# systemctl restart grafana-server         # 安裝完成后重啟生效