Ceph Dashboard介紹
Ceph 的監控可視化界面方案很多----grafana、Kraken。但是從Luminous開始，Ceph 提供了原生的Dashboard功能，通過Dashboard可以獲取Ceph集群的各種基本狀態信息。
(nautilus版) 需要安裝 ceph-mgr-dashboard
ceph nautilus dashboard 已經具有一些管理功能了，可以在WEB界面創建一些內容
1. Install the appropriate package from the below link
# ceph -v
ceph version 14.2.9 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)
# yum install ceph-mgr-dashboard
Note: it will ask for few dependencies, install with yum/apt package manager and then execute the above command.
注：所有mgr節點都安裝上ceph-mgr-dashboard包，
2. Enable the ceph mgr dashboard
# ceph mgr module enable dashboard --force
# ceph mgr module ls | grep dashboard
3. Create a user for Dashboard
Example: [ceph dashboard ac-user-create (username) (password) administrator]
# ceph dashboard ac-user-create admin admin administrator
{"username": "admin", "lastUpdate": 1581346477, "name": null, "roles": ["administrator"], "password": "$2b$12$55BR7an1KU9Rp3pBcc7HiOWNmI45kNzZlAJVK1c/Tk7krnGEEfY4S", "email": null}
------------------------------------------------------------------------
注：新點的版本比如14.2.19,創建用戶的命令已經更改，操作如下
echo '123456' > password.txt
ceph dashboard ac-user-create admin administrator -i password.txt
{"username": "admin", "lastUpdate": 1619334864, "name": null, "roles": ["administrator"], "password": "$2b$12$3XoNkgAD19mGLgjX28s4Z.9os8kMJoRglLuMWejXVINGxNiMNQA2W", "email": null}
------------------------------------------------------------------------
[root@ceph-host-01 mgr-dashboard]# ceph mgr services
{
"dashboard": "
https://ceph-host-01:8443/"
}
Note: Here you can access with IP address of ceph-mgr node, instead of hostname.
5. Make sure firewall port is open
# firewall-cmd --add-port=8443/tcp --permanent
# firewall-cmd --reload
6. Open the dashboard url in any browser
7. Enter the username: cent and password: password
配置好之后就可以登錄，頁面如下：
注：當mgr主機重啟后，
https://10.30.1.221:8443無法打開頁面來了，解決辦法是在admin節點重新把那個mgr節點加入進來，或者查看到現在的active的mgr節點是哪個，如果active的mgr節點有安裝ceph-mgr-dashboard，則直接使用active的mgr節點IP登陸就好
[root@ceph-host-01 ceph-cluster]# ceph -s
cluster:
id: 272905d2-fd66-4ef6-a772-9cd73a274683
health: HEALTH_WARN
application not enabled on 1 pool(s)
8 daemons have recently crashed
services:
mon: 3 daemons, quorum ceph-host-01,ceph-host-02,ceph-host-03 (age 14m)
mgr: ceph-host-03(active, since 15m), standbys: ceph-host-02, ceph-host-01
mds: nova:1 {0=ceph-host-02=up:active} 1 up:standby
osd: 20 osds: 20 up (since 13m), 20 in (since 19h)
data:
pools: 6 pools, 512 pgs
objects: 1.10k objects, 7.6 GiB
usage: 35 GiB used, 1.5 TiB / 1.5 TiB avail
pgs: 512 active+clean
[root@ceph-host-01 ceph-cluster]# ceph-deploy mgr create ceph-host-01
總結ceph nautilus開啟dashboard的步驟
----------------------------------------------------------------------------
mgr節點操作：
yum -y install ceph-mgr-dashboard
pip uninstall urllib3
yum install python-urllib3 -y
ceph管理節點操作：
ceph mgr module enable dashboard
ceph config set mgr mgr/dashboard/server_addr 0.0.0.0
ceph config set mgr mgr/dashboard/server_port 7000
ceph config set mgr mgr/dashboard/ssl false
ceph dashboard ac-user-create admin password administrator
修改默認配置命令#
指定集群dashboard的訪問端口
# ceph config set mgr mgr/dashboard/server_port 7000
指定集群 dashboard的訪問IP
# ceph config set mgr mgr/dashboard/server_addr 0.0.0.0
關閉ssl
# ceph config set mgr mgr/dashboard/ssl false
開啟Object Gateway管理功能#
1、創建rgw用戶
radosgw-admin user create --uid=dashboard --display-name='dashboard rgw' --system
# radosgw-admin user info --uid=dashboard
2、提供Dashboard證書
# ceph dashboard set-rgw-api-access-key $access_key
# ceph dashboard set-rgw-api-secret-key $secret_key
3、配置rgw主機名和端口
# ceph dashboard set-rgw-api-host 0.0.0.0
# ceph dashboard set-rgw-api-port 7480
4、刷新web頁面
-----------------------------------------------------------------------------------
使用 Prometheus 監控 Ceph
ceph的可視化方案很多，本篇介紹的是比較簡單的一種方式，並且對包都進行了二次封裝，所以能夠在極短的時間內構建出一個可視化的監控系統
本系統組件如下：
* ceph-nautilus版本
* ceph_exporter的nautilus版本
* prometheus的2.20.1版本
* grafana的grafana-7.1.3版本
* Ceph grafana的插件- Clusterby Cristian Calin
適配的系統為centos7
資源如下：
https://github.com/digitalocean/ceph_exporter（國內源
https://github.com.cnpmjs.org/digitalocean/ceph_exporter）
https://nodejs.org/dist/v10.13.0/node-v10.13.0-linux-x64.tar.xz
http://static.zybuluo.com/zphj1987/1nu2k4cpcery94q2re3u6s1t/ceph-cluster_rev1.json
注：一些包的來源地址
grafana下載源
https://github.com/grafana/grafana/tags
https://grafana.com/grafana/download/8.0.2
prometheus下載源
https://github.com/prometheus/prometheus/tags
ceph_exporter下載源
https://github.com/digitalocean/ceph_exporter/tags
以上資源均可以直接用wget進行下載，然后直接安裝
監控的架構介紹
通過ceph_exporter抓取的ceph相關的數據並且在本地監聽端口9128端口
prometheus抓取ceph_exporter的9128的端口的數據存儲在本地的/usr/local/prometheus/目錄下面
grafana抓取prometheus的數據進行渲染成web頁面
頁面的模板就是使用的grafana的ceph模板插件
那么我們就根據上面的架構去一步步的把系統配置起來
配置監控系統
安裝ceph_exporter
方法1：
ceph mgr module enable prometheus
端口 9283
方法2：
1）安裝需要的軟件golang
#yum install golang git librados2-devel librbd1-devel -y
2）設置go的環境變量
# /etc/profile.d/go.sh
export GOROOT=/usr/lib/golang
export GOBIN=$GOROOT/bin
export GOPATH=/home/golang
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
#source /etc/profile.d/go.sh
3）安裝ceph_exporter
#cd /home/golang
#go get -u
github.com/digitalocean/ceph_exporter （由於被牆我們只能放棄這個辦法而改用下面的命令）
# mkdir -pv src/
github.com/digitalocean/
# cd src/
github.com/digitalocean/ceph_exporter/
# go build
4）運行ceph_exporter
#cd /usr/lib/golang/bin/
#nohup ./ceph_exporter &
5）檢驗結果
#curl 127.0.0.1:9128
可以看到端口起來了就是安裝成功了，這個ceph_exporter建議是安裝在管理節點上，也就是能夠執行出ceph -s的節點上面的
安裝prometheus
admin節點上部署prometheus
tar -xvf prometheus-2.24.1.linux-amd64.tar.gz
cd prometheus-2.24.1.linux-amd64/
cp prometheus /usr/bin/
cp promtool /usr/bin/
# vim /lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus: the monitoring system
Documentation=http://prometheus.io/docs/
[Service]
ExecStart=/usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --web.listen-address=0.0.0.0:9090 --web.external-url=
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
mkdir /etc/prometheus/
cp -R consoles console_libraries prometheus.yml /etc/prometheus/
mkdir /var/lib/prometheus/
systemctl daemon-reload
systemctl enable prometheus.service
systemctl start prometheus.service
vim /etc/prometheus/prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: 'ceph'
static_configs:
- targets: ['10.2.36.196:9283']
systemctl restart prometheus.service
這個地方默認是認為prometheus和ceph_exporter在一台機器上面，所以配置文件的/etc/ceph/prometheus.yml里面的targets寫的是127.0.0.1，根據需要修改成ceph_exporter的ip地址即可
prometheus的默認監聽端口為9090，到這個時候直接去web 上面就可以看到prometheus的抓取的數據了
到這里是數據到prometheus的已經完成了，下面就去做跟grafana相關的配置了
安裝grafana
yum install grafana-8.0.2-1.x86_64.rpm
systemctl enable grafana-server.service
systemctl start grafana-server.service
grafana默認監聽的3000的端口
默認登陸的用戶名密碼為admin admin,登陸成功后會強制修改密碼
配置grafana
首先增加數據源
這里如果能上網就直接輸入id 917 ，如果不能上網就把上面的ceph-cluster_rev1.json文件弄到本地去，導入進去即可
把ceph-cluster_rev1.json內容粘貼進去就行
到這里就完成了配置了
--------------------------------------------------------------------------------------------------------------------------------------------------
擴展：編譯N版ceph rpm包支持中文dashboard
一、前言
N版ceph dashboard已經支持中文了，但是用默認是英語，對於有中文需求的人可以參考以下步驟，重新編譯ceph RPM包，支持dashboard中文顯示。
二，環境准備
需要一個大內存和多核的主機可以是虛擬機，磁盤空間要足夠大（80GB），安裝centos7.6
三、下載ceph源碼包
用git clone方式下載ceph源碼需要下載ceph的子模塊，這個過程要連接國外的網站，下載速度非常的慢，因此下載ceph的tar源碼包，下載地址：https://download.ceph.com/tarballs/。
四、編譯dashboard angular
ceph的dashboard使用的angular框架，默認情況下只編譯了英文包，因此需要自己重新編譯自己需要的語言包
1、解壓
下載了ceph 源碼包后，解壓
tar -xvf ceph_14.2.19.orig.tar.gz
2、創建臨時文件
TEMP_DIR=`mktemp -d`
3、創建python虛擬機環境
cd ceph-14.2.19/
./src/tools/setup-virtualenv.sh --python=python2.7 $TEMP_DIR
4、安裝包
$TEMP_DIR/bin/pip install nodeenv
$TEMP_DIR/bin/nodeenv -p --node=10.13.0
5、進入dashboard目錄
cd ./src/pybind/mgr/dashboard/frontend
6、激活python虛擬環境
. $TEMP_DIR/bin/activate
7、安裝js相關包
npm ci
8、編譯angular
npx npm-run-all --print-label --parallel --max-parallel 2 "build:* - - -- --prod" #編譯支持的所有語言也可以用:zh-CN編譯中文
------------------------------------------------------------------------
報錯：Error: Cannot find module 'node-sass'
解決辦法：
npm install -g cnpm --registry=https://registry.npm.taobao.org
cnpm install node-sass
報錯：Error:Node Sass version 6.0.0 is incompatible with ^4.0.0
卸載 node-sass
npm uninstall node-sass
然后安裝最新版本（5.0之前）
npm install node-sass@4.14.1
---------------------------------------------------------------------------
編譯成功后查看dist目錄下是否有對應的語言文件
編譯成功后退出python虛擬環境
deactivate
9、重新打ceph tar包
tar czvf ceph_14.2.19.tar.gz ceph-14.2.19/
五、編譯ceph rpm包
1、安裝ceph編譯依賴包
執行ceph源碼目錄下的./install-deps.sh
2、安裝gcc 7.3版本
yum install centos-release-scl scl-utils-build
yum list all --enablerepo='centos-sclo-rh' 列出scl有哪些源可以用
yum install devtoolset-7-gcc.x86_64 devtoolset-7-gcc-c++.x86_64 devtoolset-7-gcc-gdb-plugin.x86_64 安裝gcc-7.3編譯工具鏈
scl enable devtoolset-7 bash 切換到gcc-7.3編譯環境
3、設置編譯環境
yum install rpm-build rpmdevtools 安裝rpm包編譯工具
rpmdev-setuptree 設置編譯環境
cp ceph_14.2.19.tar.gz rpmbuild/SOURCES 將ceph 源碼壓縮包拷貝到rpmbuild目錄下
tar --strip-components=1 -C ~/rpmbuild/SPECS/ --no-anchored -xvzf ~/rpmbuild/SOURCES/ceph_14.2.19.tar.gz "ceph.spec" "ceph.spec" 提取ceph.sepc文件
修改文件rpmbuild/SPECS/ceph.spec 編輯Source0: http://ceph.com/download/ceph-<version>行，
將ceph-<version>改為rpmbuild/SOURCES下的文件名
4、編譯ceph rpm包
執行 rpmbuild -ba ~/rpmbuild/SPECS/ceph.spec編譯ceph rpm包
注：編譯的過程會報大量錯，主要是缺失一些包導致的
解決辦法：
把所有報錯中提到的缺失的包都安裝上
yum install -y gperf cmake3 cryptsetup fuse-devel devtoolset-8-gcc-c++ gperftools-develleveldb-devellibaio-devel libblkid-devel libcurl-devel libcap-ng-devel libudev-devel libnl3-devel liboath-devel libtool libxml2-devel ncurses-devel snappy-devel valgrind-devel xfsprogs-devel xmlstarlet yasm librabbitmq-devel librdkafka-devel boost-random nss-devel keyutils-libs-devel libibverbs-devel librdmacm-devel openldap-devel openssl-devel CUnit-devel redhat-lsb-core python2-Cython python36-Cython python-prettytable python-sphinx lz4-devel lttng-ust-devel libbabeltrace-devel expat-devel junit gperftools-devel
查看結果
# find /root/rpmbuild/ -name "*.rpm"
/root/rpmbuild/SRPMS/ceph-14.2.19-0.el7.src.rpm
/root/rpmbuild/RPMS/x86_64/python3-rbd-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-test-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librados2-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-mon-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python3-rgw-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-mgr-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librgw-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librbd1-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-radosgw-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libcephfs_jni1-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-rgw-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-common-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-osd-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/rados-objclass-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librgw2-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-rados-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/cephfs-java-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/rbd-mirror-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-rbd-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-cephfs-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/rbd-nbd-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-base-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python3-rados-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-ceph-argparse-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python3-ceph-argparse-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-mds-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libcephfs2-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python-ceph-compat-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/rbd-fuse-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/python3-cephfs-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libcephfs-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-selinux-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-fuse-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libradosstriper1-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libcephfs_jni-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librbd-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libradospp-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/librados-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/libradosstriper-devel-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-resource-agents-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/x86_64/ceph-debuginfo-14.2.19-0.el7.x86_64.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-rook-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-diskprediction-local-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-ssh-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-diskprediction-cloud-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-k8sevents-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-mgr-dashboard-14.2.19-0.el7.noarch.rpm
/root/rpmbuild/RPMS/noarch/ceph-grafana-dashboards-14.2.19-0.el7.noarch.rpm
六、遇到的問題
1、編譯rpm包過程中可能會報需要python3
yum install python36u -y
yum install python36u-devel -y
2、編譯rpm包過程中出現以下錯誤
在ceph.spec文件中加入
%define _binaries_in_noarch_packages_terminate_build 0
七、安裝方法
rpm -Uvh ceph-mgr-dashboard-14.2.19-0.el7.noarch.rpm --force
我們再次打開dashboard后發現支持中文了
作者：Dexter_Wang 工作崗位：某互聯網公司資深雲計算與存儲工程師 聯系郵箱：993852246@qq.com