一、Ceph對象存儲（RGW）的簡介和安裝
1.1、對象存儲的簡介：
- 通過對象存儲，將數據存儲為對象，每個對象除了包含數據，還包含數據自身的元數據
- 對象通過Object ID來檢索，無法通過普通文件系統操作來直接訪問對象，只能通過API來訪問，或者第三方客戶端（實際上也是對API的封裝）
- 對象存儲中的對象不整理到目錄樹中，而是存儲在扁平的命名空間中，Amazon S3將這個扁平命名空間稱為bucket。而swift則將其稱為容器
- 無論是bucket還是容器，都不能嵌套
- bucket需要被授權才能訪問到，一個帳戶可以對多個bucket授權，而權限可以不同
- 對象存儲的優點：易擴展、快速檢索
1.2、Rados網關介紹：
- RADOS網關也稱為Ceph對象網關、RADOSGW、RGW，是一種服務，使客戶端能夠利用標准對象存儲API來訪問Ceph集群。它支持S3和Swift API
- rgw運行於librados之上，事實上就是一個稱之為Civetweb的web服務器來響應api請求
- 客戶端使用標准api與rgw通信，而rgw則使用librados與ceph集群通信
- rgw客戶端通過s3或者swift api使用rgw用戶進行身份驗證。然后rgw網關代表用戶利用cephx與ceph存儲進行身份驗證
更多詳細介紹請查看官網：https://docs.ceph.com/en/latest/radosgw/
本次實驗的主鍵情況如下：
1.3、安裝Ceph的對象網關主鍵並驗證（本次打算再node02和node03上安裝RGW），具體詳細過程如下：
1 一、安裝RGW: 2 3 node02：上的操作 4 root@node02:~# apt install radosgw 5 Reading package lists... Done 6 Building dependency tree 7 Reading state information... Done 8 radosgw is already the newest version (16.2.5-1focal). 9 0 upgraded, 0 newly installed, 0 to remove and 73 not upgraded. 10 root@node02:~# 11 12 13 14 node03:上的操作 15 root@node03:~# apt install radosgw 16 Reading package lists... Done 17 Building dependency tree 18 Reading state information... Done 19 radosgw is already the newest version (16.2.5-1focal). 20 0 upgraded, 0 newly installed, 0 to remove and 73 not upgraded. 21 root@node03:~# 22 23 24 25 ceph-deploy(node01):上的操作 26 root@node01:~# ceph -s 27 cluster: 28 id: 9138c3cf-f529-4be6-ba84-97fcab59844b 29 health: HEALTH_OK 30 31 services: 32 mon: 3 daemons, quorum node01,node02,node03 (age 42s) 33 mgr: node01(active, since 35s), standbys: node02, node03 34 osd: 6 osds: 6 up (since 37s), 6 in (since 2w) 35 36 data: 37 pools: 1 pools, 1 pgs 38 objects: 0 objects, 0 B 39 usage: 33 MiB used, 240 GiB / 240 GiB avail 40 pgs: 1 active+clean 41 42 root@node01:~# 43 root@node01:~/ceph-deploy# 44 root@node01:~/ceph-deploy# ceph-deploy rgw create node02 45 [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf 46 [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create node02 47 [ceph_deploy.cli][INFO ] ceph-deploy options: 48 [ceph_deploy.cli][INFO ] username : None 49 [ceph_deploy.cli][INFO ] verbose : False 50 [ceph_deploy.cli][INFO ] rgw : [('node02', 'rgw.node02')] 51 [ceph_deploy.cli][INFO ] overwrite_conf : False 52 [ceph_deploy.cli][INFO ] subcommand : create 53 [ceph_deploy.cli][INFO ] quiet : False 54 [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fa24fea4d20> 55 [ceph_deploy.cli][INFO ] cluster : ceph 56 [ceph_deploy.cli][INFO ] func : <function rgw at 0x7fa24ff101d0> 57 [ceph_deploy.cli][INFO ] ceph_conf : None 58 [ceph_deploy.cli][INFO ] default_release : False 59 [ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts node02:rgw.node02 60 [node02][DEBUG ] connected to host: node02 61 [node02][DEBUG ] detect platform information from remote host 62 [node02][DEBUG ] detect machine type 63 [ceph_deploy.rgw][INFO ] Distro info: Ubuntu 20.04 focal 64 [ceph_deploy.rgw][DEBUG ] remote host will use systemd 65 [ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to node02 66 [node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf 67 [node02][DEBUG ] create path recursively if it doesn't exist 68 [node02][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.node02 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.node02/keyring 69 [node02][INFO ] Running command: systemctl enable ceph-radosgw@rgw.node02 70 [node02][INFO ] Running command: systemctl start ceph-radosgw@rgw.node02 71 [node02][INFO ] Running command: systemctl enable ceph.target 72 [ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host node02 and default port 7480 73 root@node01:~/ceph-deploy# ceph-deploy rgw create node03 74 [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf 75 [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create node03 76 [ceph_deploy.cli][INFO ] ceph-deploy options: 77 [ceph_deploy.cli][INFO ] username : None 78 [ceph_deploy.cli][INFO ] verbose : False 79 [ceph_deploy.cli][INFO ] rgw : [('node03', 'rgw.node03')] 80 [ceph_deploy.cli][INFO ] overwrite_conf : False 81 [ceph_deploy.cli][INFO ] subcommand : create 82 [ceph_deploy.cli][INFO ] quiet : False 83 [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f6dee079d20> 84 [ceph_deploy.cli][INFO ] cluster : ceph 85 [ceph_deploy.cli][INFO ] func : <function rgw at 0x7f6dee0e51d0> 86 [ceph_deploy.cli][INFO ] ceph_conf : None 87 [ceph_deploy.cli][INFO ] default_release : False 88 [ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts node03:rgw.node03 89 [node03][DEBUG ] connected to host: node03 90 [node03][DEBUG ] detect platform information from remote host 91 [node03][DEBUG ] detect machine type 92 [ceph_deploy.rgw][INFO ] Distro info: Ubuntu 20.04 focal 93 [ceph_deploy.rgw][DEBUG ] remote host will use systemd 94 [ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to node03 95 [node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf 96 [node03][WARNIN] rgw keyring does not exist yet, creating one 97 [node03][DEBUG ] create a keyring file 98 [node03][DEBUG ] create path recursively if it doesn't exist 99 [node03][INFO ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.node03 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.node03/keyring 100 [node03][INFO ] Running command: systemctl enable ceph-radosgw@rgw.node03 101 [node03][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.node03.service → /lib/systemd/system/ceph-radosgw@.service. 102 [node03][INFO ] Running command: systemctl start ceph-radosgw@rgw.node03 103 [node03][INFO ] Running command: systemctl enable ceph.target 104 [ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host node03 and default port 7480 105 root@node01:~/ceph-deploy# 106 root@node01:~/ceph-deploy# 107 root@node01:~/ceph-deploy# 108 root@node01:~/ceph-deploy# 109 root@node01:~/ceph-deploy# 110 root@node01:~/ceph-deploy# curl http://node02:7480 111 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>root@node01:~/ceph-deploy# curl http://node03:7480 112 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>root@node01:~/ceph-deploy# 113 root@node01:~/ceph-deploy# 114 root@node01:~/ceph-deploy# ceph -s 115 cluster: 116 id: 9138c3cf-f529-4be6-ba84-97fcab59844b 117 health: HEALTH_OK 118 119 services: 120 mon: 3 daemons, quorum node01,node02,node03 (age 12m) 121 mgr: node01(active, since 12m), standbys: node02, node03 122 osd: 6 osds: 6 up (since 12m), 6 in (since 2w) 123 rgw: 3 daemons active (3 hosts, 1 zones) 124 125 data: 126 pools: 5 pools, 105 pgs 127 objects: 189 objects, 4.9 KiB 128 usage: 90 MiB used, 240 GiB / 240 GiB avail 129 pgs: 105 active+clean 130 131 root@node01:~/ceph-deploy# 132 root@node01:~/ceph-deploy# 133 root@node01:~/ceph-deploy# ceph osd lspools 134 1 device_health_metrics 135 2 .rgw.root 136 3 default.rgw.log 137 4 default.rgw.control 138 5 default.rgw.meta 139 root@node01:~/ceph-deploy#
主要步驟如下：
1 # node02：上的操作 2 root@node02:~# apt install radosgw 3 4 # node03:上的操作 5 root@node03:~# apt install radosgw 6 7 # ceph-deploy(node01):上的操作 —— 將node02、node03分別加入RGW服務 8 root@node01:~/ceph-deploy# ceph-deploy rgw create node02 9 root@node01:~/ceph-deploy# ceph-deploy rgw create node03 10 11 # 驗證node02、node03上的服務是否正常： 12 root@node01:~/ceph-deploy# curl http://node02:7480 13 root@node01:~/ceph-deploy# curl http://node03:7480 14 15 # 由於RGW是無狀態的服務，可以使用Nginx或HAProxy和Keepalived做個高可用，這里提供一下HAProxy的相關主要的配置內容： 16 listen ceph-rgw 17 bind 192.168.11.200:80 18 mode tcp 19 server rgw1 192.168.11.220:7480 check inter 3s fall 3 rise 5 20 server rgw2 192.168.11.230:7480 check inter 3s fall 3 rise 5
二、Ceph對象存儲：創建（RGW）用戶
話不多說，直接看下面的詳細過程：
1 root@node01:~/ceph-deploy# radosgw-admin user list 2 [] 3 root@node01:~/ceph-deploy# radosgw-admin user create --uid='jerry' --display-name='hello jerry' 4 { 5 "user_id": "jerry", 6 "display_name": "hello jerry", 7 "email": "", 8 "suspended": 0, 9 "max_buckets": 1000, 10 "subusers": [], 11 "keys": [ 12 { 13 "user": "jerry", 14 "access_key": "84UFQ26AJJ95KTT1KGGH", 15 "secret_key": "tSmIeDKqCD7AneP5Wof0NFKKg0PhAxslsDVj3C3n" 16 } 17 ], 18 "swift_keys": [], 19 "caps": [], 20 "op_mask": "read, write, delete", 21 "default_placement": "", 22 "default_storage_class": "", 23 "placement_tags": [], 24 "bucket_quota": { 25 "enabled": false, 26 "check_on_raw": false, 27 "max_size": -1, 28 "max_size_kb": 0, 29 "max_objects": -1 30 }, 31 "user_quota": { 32 "enabled": false, 33 "check_on_raw": false, 34 "max_size": -1, 35 "max_size_kb": 0, 36 "max_objects": -1 37 }, 38 "temp_url_keys": [], 39 "type": "rgw", 40 "mfa_ids": [] 41 } 42 43 root@node01:~/ceph-deploy# radosgw-admin user list 44 [ 45 "jerry" 46 ] 47 root@node01:~/ceph-deploy# radosgw-admin user info --uid=jerry 48 { 49 "user_id": "jerry", 50 "display_name": "hello jerry", 51 "email": "", 52 "suspended": 0, 53 "max_buckets": 1000, 54 "subusers": [], 55 "keys": [ 56 { 57 "user": "jerry", 58 "access_key": "84UFQ26AJJ95KTT1KGGH", 59 "secret_key": "tSmIeDKqCD7AneP5Wof0NFKKg0PhAxslsDVj3C3n" 60 } 61 ], 62 "swift_keys": [], 63 "caps": [], 64 "op_mask": "read, write, delete", 65 "default_placement": "", 66 "default_storage_class": "", 67 "placement_tags": [], 68 "bucket_quota": { 69 "enabled": false, 70 "check_on_raw": false, 71 "max_size": -1, 72 "max_size_kb": 0, 73 "max_objects": -1 74 }, 75 "user_quota": { 76 "enabled": false, 77 "check_on_raw": false, 78 "max_size": -1, 79 "max_size_kb": 0, 80 "max_objects": -1 81 }, 82 "temp_url_keys": [], 83 "type": "rgw", 84 "mfa_ids": [] 85 } 86 87 root@node01:~/ceph-deploy#
主要步驟如下：
# 查看所有用戶： root@node01:~/ceph-deploy# radosgw-admin user list # 創建用戶——例如這里的Jerry： root@node01:~/ceph-deploy# radosgw-admin user create --uid='jerry' --display-name='hello jerry' # 查看用戶Jerry的詳細信息： root@node01:~/ceph-deploy# radosgw-admin user info --uid=jerry # 至於其它關於radosgw-admin命令詳細的信息，例如：修改、刪除、停用、啟動、配額等情況，請查看如下鏈接，我也不復制貼過來了： https://blog.csdn.net/qq_40017427/article/details/106358210
三、Ceph對象存儲：客戶端工具s3cmd安裝配置：
3.1、安裝配置s3cmd工具，具體過程如下：
1 root@node01:~/ceph-deploy# apt install s3cmd 2 Reading package lists... Done 3 Building dependency tree 4 Reading state information... Done 5 The following additional packages will be installed: 6 python3-magic 7 The following NEW packages will be installed: 8 python3-magic s3cmd 9 0 upgraded, 2 newly installed, 0 to remove and 73 not upgraded. 10 Need to get 121 kB of archives. 11 After this operation, 540 kB of additional disk space will be used. 12 Do you want to continue? [Y/n] y 13 Get:1 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 python3-magic all 2:0.4.15-3 [9,376 B] 14 Get:2 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/universe amd64 s3cmd all 2.0.2-1ubuntu1 [111 kB] 15 Fetched 121 kB in 1s (169 kB/s) 16 Selecting previously unselected package python3-magic. 17 (Reading database ... 74528 files and directories currently installed.) 18 Preparing to unpack .../python3-magic_2%3a0.4.15-3_all.deb ... 19 Unpacking python3-magic (2:0.4.15-3) ... 20 Selecting previously unselected package s3cmd. 21 Preparing to unpack .../s3cmd_2.0.2-1ubuntu1_all.deb ... 22 Unpacking s3cmd (2.0.2-1ubuntu1) ... 23 Setting up python3-magic (2:0.4.15-3) ... 24 Setting up s3cmd (2.0.2-1ubuntu1) ... 25 Processing triggers for man-db (2.9.1-1) ... 26 root@node01:~/ceph-deploy# 27 root@node01:~/ceph-deploy# 28 root@node01:~/ceph-deploy# s3cmd --configure 29 30 Enter new values or accept defaults in brackets with Enter. 31 Refer to user manual for detailed description of all options. 32 33 Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. 34 Access Key: 84UFQ26AJJ95KTT1KGGH 35 Secret Key: tSmIeDKqCD7AneP5Wof0NFKKg0PhAxslsDVj3C3n 36 Default Region [US]: 37 38 Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. 39 S3 Endpoint [s3.amazonaws.com]: 192.168.11.220:7480 40 41 Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used 42 if the target S3 system supports dns based buckets. 43 DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: 192.168.11.220:7480/%(bucket)s 44 45 Encryption password is used to protect your files from reading 46 by unauthorized persons while in transfer to S3 47 Encryption password: 48 Path to GPG program [/usr/bin/gpg]: 49 50 When using secure HTTPS protocol all communication with Amazon S3 51 servers is protected from 3rd party eavesdropping. This method is 52 slower than plain HTTP, and can only be proxied with Python 2.7 or newer 53 Use HTTPS protocol [Yes]: No 54 55 On some networks all internet access must go through a HTTP proxy. 56 Try setting it here if you can't connect to S3 directly 57 HTTP Proxy server name: 58 59 New settings: 60 Access Key: 84UFQ26AJJ95KTT1KGGH 61 Secret Key: tSmIeDKqCD7AneP5Wof0NFKKg0PhAxslsDVj3C3n 62 Default Region: US 63 S3 Endpoint: 192.168.11.220:7480 64 DNS-style bucket+hostname:port template for accessing a bucket: 192.168.11.220:7480/%(bucket)s 65 Encryption password: 66 Path to GPG program: /usr/bin/gpg 67 Use HTTPS protocol: False 68 HTTP Proxy server name: 69 HTTP Proxy server port: 0 70 71 Test access with supplied credentials? [Y/n] y 72 Please wait, attempting to list all buckets... 73 Success. Your access key and secret key worked fine :-) 74 75 Now verifying that encryption works... 76 Not configured. Never mind. 77 78 Save settings? [y/N] y 79 Configuration saved to '/root/.s3cfg' 80 root@node01:~/ceph-deploy#
主要步驟如下：
# 安裝s3cmd客戶端工具： root@node01:~/ceph-deploy# apt install s3cmd -y # 配置s3cmd客戶端工具（按照提示一步步走下去即可）： root@node01:~/ceph-deploy# s3cmd --configure
3.2、使用s3cmd工具創建bucket，上傳、下載文件，具體過程如下：
1 root@node01:~# s3cmd la 2 root@node01:~# s3cmd ls 3 root@node01:~# s3cmd mb s3://mybucket 4 ERROR: S3 error: 403 (SignatureDoesNotMatch) 5 root@node01:~# vim .s 6 .s3cfg .ssh/ 7 root@node01:~# sed -i '/signature_v2/s/False/True/g' /root/.s3cfg #由於直接使用的是IP地址而為使用域名：修改版本啟用V2 8 root@node01:~# s3cmd mb s3://mybucket 9 Bucket 's3://mybucket/' created 10 root@node01:~# 11 root@node01:~# s3cmd ls 12 2021-09-01 14:32 s3://mybucket 13 root@node01:~# 14 root@node01:~# s3cmd put /etc/passwd s3://mybucket/ 15 upload: '/etc/passwd' -> 's3://mybucket/passwd' [1 of 1] 16 1899 of 1899 100% in 2s 922.58 B/s done 17 root@node01:~# s3cmd ls 18 2021-09-01 14:32 s3://mybucket 19 root@node01:~# s3cmd ls s3://mybucket 20 2021-09-01 14:32 1899 s3://mybucket/passwd 21 root@node01:~# ls /opt/ 22 root@node01:~# s3cmd get s3://mybucket/passwd /opt/ 23 download: 's3://mybucket/passwd' -> '/opt/passwd' [1 of 1] 24 1899 of 1899 100% in 0s 159.77 kB/s done 25 root@node01:~# ls /opt/ 26 passwd 27 root@node01:~# s3cmd info s3://mybucket/passwd 28 s3://mybucket/passwd (object): 29 File size: 1899 30 Last mod: Wed, 01 Sep 2021 14:32:59 GMT 31 MIME type: text/plain 32 Storage: STANDARD 33 MD5 sum: 0bf05629c11b52b6566b3bfff218c054 34 SSE: none 35 Policy: none 36 CORS: none 37 ACL: hello jerry: FULL_CONTROL 38 x-amz-meta-s3cmd-attrs: atime:1630504988/ctime:1629203718/gid:0/gname:root/md5:0bf05629c11b52b6566b3bfff218c054/mode:33188/mtime:1629203718/uid:0/uname:root 39 root@node01:~#
主要步驟如下：
1 # 創建bucket： 2 root@node01:~# s3cmd mb s3://mybucket 3 4 # 上傳文件： 5 root@node01:~# s3cmd put /etc/passwd s3://mybucket/ 6 7 # 下載文件： 8 root@node01:~# s3cmd get s3://mybucket/passwd /opt/ 9 10 # 查看： 11 root@node01:~# s3cmd ls 12 root@node01:~# s3cmd la 13 root@node01:~# s3cmd ls s3://mybucket 14 15 # 過程遇到的一個小問題（ERROR: S3 error: 403 (SignatureDoesNotMatch)）—— 由於直接使用的是IP地址而為使用域名：修改版本啟用V2，如下方法解決一下： 16 root@node01:~# sed -i '/signature_v2/s/False/True/g' /root/.s3cfg
四、Ceph Dashboard啟用
Ceph Dashboard是通過一個 web 界面，對已經運行的 ceph 集群進行狀態查看及功能配置等 功能，而且Ceph-Dashboard還是用 Python 開發的一個 Ceph 的監控面板，用來監控 Ceph 的運行狀態。同時提供 REST API 來訪問狀態數據。它具有如下優缺點：
優點：
- 易部署
- 輕量級
- 靈活（可以自定義開發功能）
缺點：
- 功能相對簡單
4.1、安裝並啟用 dashboard 插件：
新版本需要安裝 dashboard 安保，而且必須安裝在 mgr 節點，否則報錯；由於我這里三台都是mgr（同時活躍節點就是node01——即deploy機器）
1 root@node01:~/ceph-deploy# ceph -s 2 cluster: 3 id: 9138c3cf-f529-4be6-ba84-97fcab59844b 4 health: HEALTH_OK 5 6 services: 7 mon: 3 daemons, quorum node01,node02,node03 (age 7m) 8 mgr: node01(active, since 3s), standbys: node02, node03 9 osd: 6 osds: 6 up (since 7m), 6 in (since 2w) 10 rgw: 3 daemons active (3 hosts, 1 zones) 11 12 data: 13 pools: 7 pools, 145 pgs 14 objects: 206 objects, 7.6 KiB 15 usage: 104 MiB used, 240 GiB / 240 GiB avail 16 pgs: 145 active+clean 17 18 root@node01:~/ceph-deploy# 19 root@node01:~/ceph-deploy# apt install ceph-mgr-dashboard 20 Reading package lists... Done 21 Building dependency tree 22 Reading state information... Done 23 The following additional packages will be installed: 24 javascript-common libjs-sphinxdoc libjs-underscore python3-repoze.lru python3-routes 25 Suggested packages: 26 apache2 | lighttpd | httpd 27 The following NEW packages will be installed: 28 ceph-mgr-dashboard javascript-common libjs-sphinxdoc libjs-underscore python3-repoze.lru python3-routes 29 0 upgraded, 6 newly installed, 0 to remove and 73 not upgraded. 30 Need to get 2,660 kB of archives. 31 After this operation, 16.6 MB of additional disk space will be used. 32 Do you want to continue? [Y/n] y 33 Get:1 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 34 Get:2 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 35 Get:3 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 36 Get:4 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 37 Get:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific focal/main amd64 ceph-mgr-dashboard all 16.2.5-1focal [2,357 kB] 38 Get:6 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 javascript-common all 11 [6,066 B] 39 Fetched 2,660 kB in 1s (3,200 kB/s) 40 Selecting previously unselected package python3-repoze.lru. 41 (Reading database ... 74577 files and directories currently installed.) 42 Preparing to unpack .../0-python3-repoze.lru_0.7-2_all.deb ... 43 Unpacking python3-repoze.lru (0.7-2) ... 44 Selecting previously unselected package libjs-underscore. 45 Preparing to unpack .../1-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 46 Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 47 Selecting previously unselected package libjs-sphinxdoc. 48 Preparing to unpack .../2-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 49 Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 50 Selecting previously unselected package python3-routes. 51 Preparing to unpack .../3-python3-routes_2.4.1-2_all.deb ... 52 Unpacking python3-routes (2.4.1-2) ... 53 Selecting previously unselected package ceph-mgr-dashboard. 54 Preparing to unpack .../4-ceph-mgr-dashboard_16.2.5-1focal_all.deb ... 55 Unpacking ceph-mgr-dashboard (16.2.5-1focal) ... 56 Selecting previously unselected package javascript-common. 57 Preparing to unpack .../5-javascript-common_11_all.deb ... 58 Unpacking javascript-common (11) ... 59 Setting up javascript-common (11) ... 60 Setting up python3-repoze.lru (0.7-2) ... 61 Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 62 Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 63 Setting up python3-routes (2.4.1-2) ... 64 Setting up ceph-mgr-dashboard (16.2.5-1focal) ... 65 Processing triggers for man-db (2.9.1-1) ... 66 root@node01:~/ceph-deploy# 67 root@node01:~/ceph-deploy# ceph -s 68 cluster: 69 id: 9138c3cf-f529-4be6-ba84-97fcab59844b 70 health: HEALTH_OK 71 72 services: 73 mon: 3 daemons, quorum node01,node02,node03 (age 7m) 74 mgr: node01(active, since 3s), standbys: node02, node03 75 osd: 6 osds: 6 up (since 7m), 6 in (since 2w) 76 rgw: 3 daemons active (3 hosts, 1 zones) 77 78 data: 79 pools: 7 pools, 145 pgs 80 objects: 206 objects, 7.6 KiB 81 usage: 104 MiB used, 240 GiB / 240 GiB avail 82 pgs: 145 active+clean 83 84 root@node01:~/ceph-deploy# 85 root@node01:~/ceph-deploy# 86 root@node01:~/ceph-deploy# ceph mgr module enable dashboard 87 root@node01:~/ceph-deploy# ceph mgr module ls | grep dashboard 88 "dashboard", 89 root@node01:~/ceph-deploy# 90 root@node01:~/ceph-deploy# 91 root@node01:~/ceph-deploy# 92 root@node01:~/ceph-deploy# ceph config set mgr mgr/dashboard/ssl false 93 root@node01:~/ceph-deploy# 94 root@node01:~/ceph-deploy# ceph config set mgr mgr/dashboard/node01/server_addr 192.168.11.210 95 root@node01:~/ceph-deploy# echo $? 96 0 97 root@node01:~/ceph-deploy# ceph config set mgr mgr/dashboard/node01/server_port 2000 98 root@node01:~/ceph-deploy# lsof -i:2000 99 root@node01:~/ceph-deploy# systemctl restart ceph-mgr 100 ceph-mgr@node01.service ceph-mgr.target 101 root@node01:~/ceph-deploy# systemctl restart ceph-mgr@node01.service 102 root@node01:~/ceph-deploy# lsof -i:2000 103 root@node01:~/ceph-deploy# lsof -i:2000 104 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME 105 ceph-mgr 5485 ceph 29u IPv4 59814 0t0 TCP node01:cisco-sccp (LISTEN) 106 root@node01:~/ceph-deploy# lsof -i:2000 107 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME 108 ceph-mgr 5485 ceph 29u IPv4 59814 0t0 TCP node01:cisco-sccp (LISTEN) 109 root@node01:~/ceph-deploy# ss -ntl 110 State Recv-Q Send-Q Local Address:Port Peer Address:Port Process 111 LISTEN 0 5 192.168.11.210:2000 0.0.0.0:* 112 LISTEN 0 512 192.168.22.210:6800 0.0.0.0:* 113 LISTEN 0 512 192.168.11.210:6800 0.0.0.0:* 114 LISTEN 0 512 192.168.22.210:6801 0.0.0.0:* 115 LISTEN 0 512 192.168.11.210:6801 0.0.0.0:* 116 LISTEN 0 512 192.168.22.210:6802 0.0.0.0:* 117 LISTEN 0 512 192.168.11.210:6802 0.0.0.0:* 118 LISTEN 0 512 192.168.22.210:6803 0.0.0.0:* 119 LISTEN 0 512 192.168.11.210:6803 0.0.0.0:* 120 LISTEN 0 512 192.168.22.210:6804 0.0.0.0:* 121 LISTEN 0 512 192.168.11.210:6804 0.0.0.0:* 122 LISTEN 0 512 192.168.22.210:6805 0.0.0.0:* 123 LISTEN 0 512 192.168.11.210:6805 0.0.0.0:* 124 LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* 125 LISTEN 0 512 192.168.22.210:6806 0.0.0.0:* 126 LISTEN 0 512 192.168.11.210:6806 0.0.0.0:* 127 LISTEN 0 128 0.0.0.0:22 0.0.0.0:* 128 LISTEN 0 512 192.168.22.210:6807 0.0.0.0:* 129 LISTEN 0 512 192.168.11.210:6807 0.0.0.0:* 130 LISTEN 0 512 192.168.11.210:6808 0.0.0.0:* 131 LISTEN 0 4096 0.0.0.0:7480 0.0.0.0:* 132 LISTEN 0 512 192.168.11.210:6809 0.0.0.0:* 133 LISTEN 0 128 127.0.0.1:6010 0.0.0.0:* 134 LISTEN 0 512 192.168.11.210:3300 0.0.0.0:* 135 LISTEN 0 512 192.168.11.210:6789 0.0.0.0:* 136 LISTEN 0 128 [::]:22 [::]:* 137 LISTEN 0 4096 [::]:7480 [::]:* 138 LISTEN 0 128 [::1]:6010 [::]:* 139 root@node01:~/ceph-deploy# 140 root@node01:~/ceph-deploy# 141 root@node01:~/ceph-deploy# echo '123456789'> passwd.txt 142 root@node01:~/ceph-deploy# ceph dashboard set-login-credentials tom -i passwd.txt 143 ****************************************************************** 144 *** WARNING: this command is deprecated. *** 145 *** Please use the ac-user-* related commands to manage users. *** 146 ****************************************************************** 147 Username and password updated 148 root@node01:~/ceph-deploy#
主要步驟如下：
1 # 重點：在活躍的mgr節點上安裝（當然你也可以用上次課的方式——全部活躍） 2 root@node01:~/ceph-deploy# apt install ceph-mgr-dashboard -y 3 4 # 啟用Dashboard模塊： 5 root@node01:~/ceph-deploy# ceph mgr module enable dashboard 6 7 # 注：模塊啟用后還不能直接訪問，需要配置關閉 SSL 或啟用 SSL 及指定監聽地址 8 root@node01:~/ceph-deploy# ceph config set mgr mgr/dashboard/ssl false 9 10 # 設置Dashboard的地址端口並檢查確認（如果：沒有看到端口起來，重啟一下服務再檢查）： 11 root@node01:~/ceph-deploy# ceph config set mgr mgr/dashboard/node01/server_addr 192.168.11.210 12 root@node01:~/ceph-deploy# ceph config set mgr mgr/dashboard/node01/server_port 2000 13 root@node01:~/ceph-deploy# lsof -i:2000 14 root@node01:~/ceph-deploy# systemctl restart ceph-mgr 15 16 # 設置dashboard賬戶密碼： 17 root@node01:~/ceph-deploy# echo '123456789'> passwd.txt 18 root@node01:~/ceph-deploy# ceph dashboard set-login-credentials tom -i passwd.txt
使用瀏覽器登陸驗證：
4.2、再來個雞肋功能：Dashboard啟動用SSL：
具體詳細過程如下：
1 root@node01:~/ceph-deploy# ceph dashboard create-self-signed-cert 2 Self-signed certificate created 3 root@node01:~/ceph-deploy# 4 root@node01:~/ceph-deploy# ceph config set mgr mgr/dashboard/ssl true 5 root@node01:~/ceph-deploy# 6 root@node01:~/ceph-deploy# ceph mgr services 7 { 8 "dashboard": "http://192.168.11.210:2000/" 9 } 10 root@node01:~/ceph-deploy# ceph mgr services 11 { 12 "dashboard": "http://192.168.11.210:2000/" 13 } 14 root@node01:~/ceph-deploy# ceph mgr services 15 { 16 "dashboard": "http://192.168.11.210:2000/" 17 } 18 root@node01:~/ceph-deploy# ceph mgr services 19 { 20 "dashboard": "http://192.168.11.210:2000/" 21 } 22 root@node01:~/ceph-deploy# ceph mgr services 23 { 24 "dashboard": "http://192.168.11.210:2000/" 25 } 26 root@node01:~/ceph-deploy# ceph mgr services 27 { 28 "dashboard": "http://192.168.11.210:2000/" 29 } 30 root@node01:~/ceph-deploy# ceph mgr services 31 { 32 "dashboard": "http://192.168.11.210:2000/" 33 } 34 root@node01:~/ceph-deploy# systemctl restart ceph-mgr@node01.service 35 root@node01:~/ceph-deploy# ceph mgr services 36 { 37 "dashboard": "http://192.168.11.210:2000/" 38 } 39 root@node01:~/ceph-deploy# ceph mgr services 40 { 41 "dashboard": "http://192.168.11.210:2000/" 42 } 43 root@node01:~/ceph-deploy# ceph mgr services 44 {} 45 root@node01:~/ceph-deploy# ceph mgr services 46 {} 47 root@node01:~/ceph-deploy# ceph mgr services 48 { 49 "dashboard": "https://192.168.11.210:8443/" 50 } 51 root@node01:~/ceph-deploy#
主要就兩步即可：
1 # 自簽證書： 2 root@node01:~/ceph-deploy# ceph dashboard create-self-signed-cert 3 4 # 啟用SSL功能 5 root@node01:~/ceph-deploy# ceph config set mgr mgr/dashboard/ssl true 6 7 # 查看狀態（不行重啟一下呢）： 8 root@node01:~/ceph-deploy# ceph mgr services
好了，Dashboard就這樣吧！
五、使用 Prometheus 監控 Ceph
5.1、先安裝部署Prometheus服務器端(具體詳細過程如下)：
1 root@node01:/usr/local/src# ls 2 root@node01:/usr/local/src# rz -E 3 rz waiting to receive. 4 root@node01:/usr/local/src# ls 5 prometheus-2.23.0.linux-amd64.tar.gz 6 root@node01:/usr/local/src# tar zxf prometheus-2.23.0.linux-amd64.tar.gz 7 root@node01:/usr/local/src# ls 8 prometheus-2.23.0.linux-amd64 prometheus-2.23.0.linux-amd64.tar.gz 9 root@node01:/usr/local/src# mv prometheus-2.23.0.linux-amd64 /usr/local/ 10 root@node01:/usr/local/src# cd /usr/local/ 11 root@node01:/usr/local# ln -s prometheus-2.23.0.linux-amd64 prometheus 12 root@node01:/usr/local# ll 13 total 44 14 drwxr-xr-x 11 root root 4096 Sep 2 12:08 ./ 15 drwxr-xr-x 14 root root 4096 Feb 2 2021 ../ 16 drwxr-xr-x 2 root root 4096 Feb 2 2021 bin/ 17 drwxr-xr-x 2 root root 4096 Feb 2 2021 etc/ 18 drwxr-xr-x 2 root root 4096 Feb 2 2021 games/ 19 drwxr-xr-x 2 root root 4096 Feb 2 2021 include/ 20 drwxr-xr-x 4 root root 4096 Aug 17 20:24 lib/ 21 lrwxrwxrwx 1 root root 9 Feb 2 2021 man -> share/man/ 22 lrwxrwxrwx 1 root root 29 Sep 2 12:08 prometheus -> prometheus-2.23.0.linux-amd64/ 23 drwxr-xr-x 4 3434 3434 4096 Nov 26 2020 prometheus-2.23.0.linux-amd64/ 24 drwxr-xr-x 2 root root 4096 Feb 2 2021 sbin/ 25 drwxr-xr-x 4 root root 4096 Feb 2 2021 share/ 26 drwxr-xr-x 2 root root 4096 Sep 2 12:08 src/ 27 root@node01:/usr/local# cd prometheus 28 root@node01:/usr/local/prometheus# pwd 29 /usr/local/prometheus 30 root@node01:/usr/local/prometheus# vim /etc/systemd/system/prometheus.service 31 root@node01:/usr/local/prometheus# cat /etc/systemd/system/prometheus.service 32 [Unit] 33 Description=Prometheus Server 34 Documentation=https://prometheus.io/docs/introduction/overview/ 35 After=network.target 36 [Service] 37 Restart=on-failure 38 WorkingDirectory=//usr/local/prometheus/ 39 ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml 40 [Install] 41 WantedBy=multi-user.target 42 root@node01:/usr/local/prometheus# systemctl daemon-reload 43 root@node01:/usr/local/prometheus# systemctl enable prometheus.service --now 44 Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /etc/systemd/system/prometheus.service. 45 root@node01:/usr/local/prometheus# systemctl status prometheus.service 46 ● prometheus.service - Prometheus Server 47 Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled) 48 Active: active (running) since Thu 2021-09-02 12:11:20 CST; 19s ago 49 Docs: https://prometheus.io/docs/introduction/overview/ 50 Main PID: 11155 (prometheus) 51 Tasks: 7 (limit: 2245) 52 Memory: 20.3M 53 CGroup: /system.slice/prometheus.service 54 └─11155 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml 55 56 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.865Z caller=head.go:645 component=tsdb msg="Replaying on-disk memory mappable chunks if any" 57 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.865Z caller=head.go:659 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.692µs 58 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.865Z caller=head.go:665 component=tsdb msg="Replaying WAL, this may take a while" 59 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.865Z caller=head.go:717 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 60 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.865Z caller=head.go:722 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=22.351µs wal_replay_duration=327.424µs t> 61 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.866Z caller=main.go:742 fs_type=EXT4_SUPER_MAGIC 62 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.868Z caller=main.go:745 msg="TSDB started" 63 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.868Z caller=main.go:871 msg="Loading configuration file" filename=/usr/local/prometheus/prometheus.yml 64 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.868Z caller=main.go:902 msg="Completed loading of configuration file" filename=/usr/local/prometheus/prometheus.yml totalDuration=585> 65 Sep 02 12:11:20 node01 prometheus[11155]: level=info ts=2021-09-02T04:11:20.868Z caller=main.go:694 msg="Server is ready to receive web requests." 66 67 root@node01:/usr/local/prometheus# systemctl restart prometheus.service 68 root@node01:/usr/local/prometheus#
主要步驟如下：
1 # 上傳二進制文件到/usr/local/src目錄下，並解壓到/usr/local目錄下並創建軟鏈接；編寫systemd文件，最后啟動： 2 root@node01:/usr/local/src# tar zxf prometheus-2.23.0.linux-amd64.tar.gz 3 root@node01:/usr/local# ln -s prometheus-2.23.0.linux-amd64 prometheus
4 root@node01:/usr/local/prometheus# cat /etc/systemd/system/prometheus.service 5 [Unit] 6 Description=Prometheus Server 7 Documentation=https://prometheus.io/docs/introduction/overview/ 8 After=network.target 9 [Service] 10 Restart=on-failure 11 WorkingDirectory=//usr/local/prometheus/ 12 ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml 13 [Install] 14 WantedBy=multi-user.target
15 root@node01:/usr/local/prometheus# systemctl daemon-reload 16 root@node01:/usr/local/prometheus# systemctl enable prometheus.service --now 17 Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /etc/systemd/system/prometheus.service. 18 root@node01:/usr/local/prometheus# systemctl status prometheus.service
登陸瀏覽器界面驗證一下：
5.2、安裝部署Prometheus的node_exporter（三台node都裝）
具體詳細過程如下：
1 root@node03:~# tar -zxvf node_exporter-1.0.1.linux-amd64.tar.gz -C /usr/local/ 2 node_exporter-1.0.1.linux-amd64/ 3 node_exporter-1.0.1.linux-amd64/NOTICE 4 node_exporter-1.0.1.linux-amd64/node_exporter 5 node_exporter-1.0.1.linux-amd64/LICENSE 6 root@node03:~# ln -sv /usr/local/node_exporter-1.0.1.linux-amd64 /usr/local/node_exporter 7 '/usr/local/node_exporter' -> '/usr/local/node_exporter-1.0.1.linux-amd64' 8 root@node03:~# ll /usr/local/ 9 total 44 10 drwxr-xr-x 11 root root 4096 Sep 2 12:44 ./ 11 drwxr-xr-x 14 root root 4096 Feb 2 2021 ../ 12 drwxr-xr-x 2 root root 4096 Feb 2 2021 bin/ 13 drwxr-xr-x 2 root root 4096 Feb 2 2021 etc/ 14 drwxr-xr-x 2 root root 4096 Feb 2 2021 games/ 15 drwxr-xr-x 2 root root 4096 Feb 2 2021 include/ 16 drwxr-xr-x 4 root root 4096 Aug 17 20:25 lib/ 17 lrwxrwxrwx 1 root root 9 Feb 2 2021 man -> share/man/ 18 lrwxrwxrwx 1 root root 42 Sep 2 12:44 node_exporter -> /usr/local/node_exporter-1.0.1.linux-amd64/ 19 drwxr-xr-x 2 3434 3434 4096 Jun 16 2020 node_exporter-1.0.1.linux-amd64/ 20 drwxr-xr-x 2 root root 4096 Feb 2 2021 sbin/ 21 drwxr-xr-x 4 root root 4096 Feb 2 2021 share/ 22 drwxr-xr-x 2 root root 4096 Feb 2 2021 src/ 23 root@node03:~# vim /etc/systemd/system/node-exporter.service 24 root@node03:~# systemctl daemon-reload 25 root@node03:~# systemctl start node-exporter.service 26 root@node03:~# systemctl enable node-exporter.service 27 Created symlink /etc/systemd/system/multi-user.target.wants/node-exporter.service → /etc/systemd/system/node-exporter.service. 28 root@node03:~# ss -ntl|grep 9100 29 LISTEN 0 4096 *:9100 *:* 30 root@node03:~# 31 root@node03:~# cat /etc/systemd/system/node-exporter.service 32 [Unit] 33 Description=Prometheus Node Exporter 34 After=network.target 35 36 [Service] 37 ExecStart=/usr/local/node_exporter/node_exporter 38 39 [Install] 40 WantedBy=multi-user.target 41 root@node03:~#
主要步驟如下（這邊只記錄了node03的過程，node01和node02步驟過程一樣）：
1 # 解壓到/usr/local目錄下： 2 root@node03:~# tar -zxvf node_exporter-1.0.1.linux-amd64.tar.gz -C /usr/local/ 3 4 # 創建軟連接： 5 root@node03:~# ln -sv /usr/local/node_exporter-1.0.1.linux-amd64 /usr/local/node_exporter 6 7 # 創建systemd文件並重新加載啟動： 8 root@node03:~# vim /etc/systemd/system/node-exporter.service 9 root@node03:~# systemctl daemon-reload 10 root@node03:~# systemctl start node-exporter.service 11 root@node03:~# systemctl enable node-exporter.service 12 13 ################ systemd 文件內容 ############### 14 root@node03:~# cat /etc/systemd/system/node-exporter.service 15 [Unit] 16 Description=Prometheus Node Exporter 17 After=network.target 18 19 [Service] 20 ExecStart=/usr/local/node_exporter/node_exporter 21 22 [Install] 23 WantedBy=multi-user.target 24 root@node03:~#
登陸瀏覽器界面驗證一下：
5.3、Prometheus服務器端添加node節點：
修改配置文件，重啟服務並登陸瀏覽器查看；具體步驟如下：
1 root@node01:/usr/local/prometheus# tail -n3 /usr/local/prometheus/prometheus.yml 2 - job_name: 'Ceph-node' 3 static_configs: 4 - targets: ['192.168.11.210:9100','192.168.11.220:9100','192.168.11.230:9100'] 5 root@node01:/usr/local/prometheus# systemctl restart prometheus.service
################################################# 邪惡的分割線 —— 中文躺了一下，寶寶玩電腦 —— 集群都不知道為什么起不來了：回復快照重新開始吧 #################################################
5.3、通過 Prometheus監控Ceph：
由於Ceph manager 內部的模塊中包含了 prometheus 的監控模塊,並監聽在每個 manager 節點的 9283 端口，該端口用於將采集到的信息通過 http 接口向 prometheus 提供數據。所以下面先啟用相關功能：
1 # 啟用功能： 2 root@node01:/usr/local# ceph mgr module enable prometheus 3 4 # 查看端口是否起來： 5 root@node01:/usr/local# ss -ntlp|grep 9283 6 LISTEN 0 5 192.168.11.210:9283 0.0.0.0:* users:(("ceph-mgr",pid=918,fd=27)) 7 root@node01:/usr/local#
瀏覽器查看一下：
修改Prometheus配置文件並重啟：
1 root@node01:/usr/local/prometheus# vim prometheus.yml 2 root@node01:/usr/local/prometheus# systemctl restart prometheus.service 3 root@node01:/usr/local/prometheus# 4 root@node01:/usr/local/prometheus# 5 root@node01:/usr/local/prometheus# 6 root@node01:/usr/local/prometheus# 7 root@node01:/usr/local/prometheus# 8 root@node01:/usr/local/prometheus# 9 root@node01:/usr/local/prometheus# tail prometheus.yml 10 static_configs: 11 - targets: ['localhost:9090'] 12 13 - job_name: 'prometheus-ceph-cluster' 14 15 # metrics_path defaults to '/metrics' 16 # scheme defaults to 'http'. 17 18 static_configs: 19 - targets: ['192.168.11.210:9283'] 20 root@node01:/usr/local/prometheus#
瀏覽器查看驗證：
5.4、通過 Grafana 展示監控數據：
先安裝部署Grafana，具體過程如下：
1 root@node01:~# apt-get install -y adduser libfontconfig1 2 Reading package lists... Done 3 Building dependency tree 4 Reading state information... Done 5 adduser is already the newest version (3.118ubuntu2). 6 adduser set to manually installed. 7 The following additional packages will be installed: 8 fontconfig-config fonts-dejavu-core 9 The following NEW packages will be installed: 10 fontconfig-config fonts-dejavu-core libfontconfig1 11 0 upgraded, 3 newly installed, 0 to remove and 85 not upgraded. 12 Need to get 1,184 kB of archives. 13 After this operation, 3,525 kB of additional disk space will be used. 14 Get:1 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 fonts-dejavu-core all 2.37-1 [1,041 kB] 15 Get:2 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 fontconfig-config all 2.13.1-2ubuntu3 [28.8 kB] 16 Get:3 https://mirrors.tuna.tsinghua.edu.cn/ubuntu focal/main amd64 libfontconfig1 amd64 2.13.1-2ubuntu3 [114 kB] 17 Fetched 1,184 kB in 1s (1,482 kB/s) 18 Selecting previously unselected package fonts-dejavu-core. 19 (Reading database ... 74549 files and directories currently installed.) 20 Preparing to unpack .../fonts-dejavu-core_2.37-1_all.deb ... 21 Unpacking fonts-dejavu-core (2.37-1) ... 22 Selecting previously unselected package fontconfig-config. 23 Preparing to unpack .../fontconfig-config_2.13.1-2ubuntu3_all.deb ... 24 Unpacking fontconfig-config (2.13.1-2ubuntu3) ... 25 Selecting previously unselected package libfontconfig1:amd64. 26 Preparing to unpack .../libfontconfig1_2.13.1-2ubuntu3_amd64.deb ... 27 Unpacking libfontconfig1:amd64 (2.13.1-2ubuntu3) ... 28 Setting up fonts-dejavu-core (2.37-1) ... 29 Setting up fontconfig-config (2.13.1-2ubuntu3) ... 30 Setting up libfontconfig1:amd64 (2.13.1-2ubuntu3) ... 31 Processing triggers for man-db (2.9.1-1) ... 32 Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 33 root@node01:~# wget https://dl.grafana.com/oss/release/grafana_7.5.1_amd64.deb 34 --2021-09-02 16:17:39-- https://dl.grafana.com/oss/release/grafana_7.5.1_amd64.deb 35 Resolving dl.grafana.com (dl.grafana.com)... 199.232.198.217, 199.232.194.217, 2a04:4e42:4d::729, ... 36 Connecting to dl.grafana.com (dl.grafana.com)|199.232.198.217|:443... connected. 37 HTTP request sent, awaiting response... 200 OK 38 Length: 55597840 (53M) [application/vnd.debian.binary-package] 39 Saving to: ‘grafana_7.5.1_amd64.deb’ 40 41 grafana_7.5.1_amd64.deb 100%[====================================================================================================================>] 53.02M 7.98MB/s in 8.9s 42 43 2021-09-02 16:17:49 (5.93 MB/s) - ‘grafana_7.5.1_amd64.deb’ saved [55597840/55597840] 44 45 root@node01:~# ls 46 ceph-deploy grafana_7.5.1_amd64.deb snap 47 root@node01:~# dpkg -i grafana_7.5.1_amd64.deb 48 Selecting previously unselected package grafana. 49 (Reading database ... 74661 files and directories currently installed.) 50 Preparing to unpack grafana_7.5.1_amd64.deb ... 51 Unpacking grafana (7.5.1) ... 52 Setting up grafana (7.5.1) ... 53 Adding system user `grafana' (UID 113) ... 54 Adding new user `grafana' (UID 113) with group `grafana' ... 55 Not creating home directory `/usr/share/grafana'. 56 ### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd 57 sudo /bin/systemctl daemon-reload 58 sudo /bin/systemctl enable grafana-server 59 ### You can start grafana-server by executing 60 sudo /bin/systemctl start grafana-server 61 Processing triggers for systemd (245.4-4ubuntu3.11) ... 62 root@node01:~# systemctl status gra 63 grafana-server.service graphical.target 64 root@node01:~# systemctl status grafana-server.service 65 ● grafana-server.service - Grafana instance 66 Loaded: loaded (/lib/systemd/system/grafana-server.service; disabled; vendor preset: enabled) 67 Active: inactive (dead) 68 Docs: http://docs.grafana.org 69 root@node01:~# systemctl enable grafana-server.service --now 70 Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install. 71 Executing: /lib/systemd/systemd-sysv-install enable grafana-server 72 Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /lib/systemd/system/grafana-server.service. 73 root@node01:~# systemctl status grafana-server.service 74 ● grafana-server.service - Grafana instance 75 Loaded: loaded (/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled) 76 Active: active (running) since Thu 2021-09-02 16:18:40 CST; 1s ago 77 Docs: http://docs.grafana.org 78 Main PID: 12671 (grafana-server) 79 Tasks: 8 (limit: 2245) 80 Memory: 23.4M 81 CGroup: /system.slice/grafana-server.service 82 └─12671 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var> 83 84 Sep 02 16:18:40 node01 grafana-server[12671]: {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-09-02T16:18:40.822136+08:00","queryType":"grafana_api"} 85 Sep 02 16:18:40 node01 grafana-server[12671]: {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-09-02T16:18:40.822143+08:00","queryType":"arrow"} 86 Sep 02 16:18:40 node01 grafana-server[12671]: {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-09-02T16:18:40.822150+08:00","queryType":"annotations"} 87 Sep 02 16:18:40 node01 grafana-server[12671]: {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-09-02T16:18:40.822156+08:00","queryType":"table_static"} 88 Sep 02 16:18:40 node01 grafana-server[12671]: {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-09-02T16:18:40.822162+08:00","queryType":"random_walk_with_error"} 89 Sep 02 16:18:40 node01 grafana-server[12671]: {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-09-02T16:18:40.822167+08:00","queryType":"server_error_500"} 90 Sep 02 16:18:40 node01 grafana-server[12671]: {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-09-02T16:18:40.822173+08:00","queryType":"logs"} 91 Sep 02 16:18:40 node01 grafana-server[12671]: {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-09-02T16:18:40.822178+08:00","queryType":"node_graph"} 92 Sep 02 16:18:40 node01 grafana-server[12671]: {"@level":"debug","@message":"datasource: registering query type fallback handler","@timestamp":"2021-09-02T16:18:40.822185+08:00"} 93 Sep 02 16:18:40 node01 grafana-server[12671]: t=2021-09-02T16:18:40+0800 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket= 94 95 root@node01:~# ss -ntlp|grep 3000 96 LISTEN 0 4096 *:3000 *:* users:(("grafana-server",pid=12671,fd=8)) 97 root@node01:~#
主要步驟如下：
1 root@node01:~# apt-get install -y adduser libfontconfig1 2 root@node01:~# wget https://dl.grafana.com/oss/release/grafana_7.5.1_amd64.deb 3 root@node01:~# dpkg -i grafana_7.5.1_amd64.deb 4 root@node01:~# systemctl enable grafana-server.service --now
下面是圖形化配置：
至於其它模板：可以登陸官網查詢：https://grafana.com/grafana/dashboards
最終結果如下圖：
今天就到這里吧，至於Docker相關基礎：查看我以前的截圖筆記吧（當然我的博客園里面也有相關文章）：
文檔：馬哥Docker
鏈接：http://note.youdao.com/noteshare?id=c91e30129443ebb3381814ea8cf33c62
其實這里有個小哥整理了一下：https://blog.csdn.net/zisefeizhu/article/details/83472190