k8s 安裝 GlusterFS 分布式文件系統（一）
本節動態存儲主要介紹GFS的使用。
一、准備工作
為了保證 Pod 能夠正常使用 GFS 作為后端存儲，需要每台運行 Pod 的節點上提前安裝 GFS 的客戶端工具。
所有節點安裝 GFS 客戶端：
[root@k8s-master01 ~]#yum install glusterfs glusterfs-fuse -y Loaded plugins: fastestmirror Determining fastest mirrors * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com aliyun-docker-ce | 3.5 kB 00:00:00 base | 3.6 kB 00:00:00 epel | 4.7 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/2): epel/x86_64/updateinfo | 1.0 MB 00:00:05 (2/2): epel/x86_64/primary_db | 6.9 MB 00:00:25 Resolving Dependencies --> Running transaction check ---> Package glusterfs.x86_64 0:6.0-49.1.el7 will be installed --> Processing Dependency: glusterfs-libs(x86-64) = 6.0-49.1.el7 for package: glusterfs-6.0-49.1.el7.x86_64 --> Processing Dependency: libglusterfs.so.0()(64bit) for package: glusterfs-6.0-49.1.el7.x86_64 --> Processing Dependency: libgfxdr.so.0()(64bit) for package: glusterfs-6.0-49.1.el7.x86_64 --> Processing Dependency: libgfrpc.so.0()(64bit) for package: glusterfs-6.0-49.1.el7.x86_64 ---> Package glusterfs-fuse.x86_64 0:6.0-49.1.el7 will be installed --> Processing Dependency: glusterfs-client-xlators(x86-64) = 6.0-49.1.el7 for package: glusterfs-fuse-6.0-49.1.el7.x86_64 --> Processing Dependency: attr for package: glusterfs-fuse-6.0-49.1.el7.x86_64 --> Running transaction check ---> Package attr.x86_64 0:2.4.46-13.el7 will be installed ---> Package glusterfs-client-xlators.x86_64 0:6.0-49.1.el7 will be installed ---> Package glusterfs-libs.x86_64 0:6.0-49.1.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ===================================================================================================================== Package Arch Version Repository Size ===================================================================================================================== Installing: glusterfs x86_64 6.0-49.1.el7 updates 622 k glusterfs-fuse x86_64 6.0-49.1.el7 updates 130 k Installing for dependencies: attr x86_64 2.4.46-13.el7 base 66 k glusterfs-client-xlators x86_64 6.0-49.1.el7 updates 839 k glusterfs-libs x86_64 6.0-49.1.el7 updates 398 k Transaction Summary ===================================================================================================================== Install 2 Packages (+3 Dependent packages) Total download size: 2.0 M Installed size: 9.0 M Downloading packages: (1/5): attr-2.4.46-13.el7.x86_64.rpm | 66 kB 00:00:00 (2/5): glusterfs-client-xlators-6.0-49.1.el7.x86_64.rpm | 839 kB 00:00:02 (3/5): glusterfs-fuse-6.0-49.1.el7.x86_64.rpm | 130 kB 00:00:00 (4/5): glusterfs-6.0-49.1.el7.x86_64.rpm | 622 kB 00:00:03 (5/5): glusterfs-libs-6.0-49.1.el7.x86_64.rpm | 398 kB 00:00:01 --------------------------------------------------------------------------------------------------------------------- Total 435 kB/s | 2.0 MB 00:00:04 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : glusterfs-libs-6.0-49.1.el7.x86_64 1/5 Installing : glusterfs-6.0-49.1.el7.x86_64 2/5 Installing : glusterfs-client-xlators-6.0-49.1.el7.x86_64 3/5 Installing : attr-2.4.46-13.el7.x86_64 4/5 Installing : glusterfs-fuse-6.0-49.1.el7.x86_64 5/5 Verifying : attr-2.4.46-13.el7.x86_64 1/5 Verifying : glusterfs-fuse-6.0-49.1.el7.x86_64 2/5 Verifying : glusterfs-6.0-49.1.el7.x86_64 3/5 Verifying : glusterfs-client-xlators-6.0-49.1.el7.x86_64 4/5 Verifying : glusterfs-libs-6.0-49.1.el7.x86_64 5/5 Installed: glusterfs.x86_64 0:6.0-49.1.el7 glusterfs-fuse.x86_64 0:6.0-49.1.el7 Dependency Installed: attr.x86_64 0:2.4.46-13.el7 glusterfs-client-xlators.x86_64 0:6.0-49.1.el7 glusterfs-libs.x86_64 0:6.0-49.1.el7 Complete! [root@k8s-master01 ~]#
給需要作為 GFS 節點提供存儲的節點打上標簽：
[root@k8s-master01 ~]#kubectl label node k8s-master01 storagenode=glusterfs node/k8s-master01 labeled [root@k8s-master01 ~]#kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master01 Ready matser 18d v1.20.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master01,kubernetes.io/os=linux,node-role.kubernetes.io/matser=,node.kubernetes.io/node=,storagenode=glusterfs k8s-master02 Ready matser 18d v1.20.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master02,kubernetes.io/os=linux,node-role.kubernetes.io/matser=,node.kubernetes.io/node= k8s-master03 Ready matser 18d v1.20.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master03,kubernetes.io/os=linux,node-role.kubernetes.io/matser=,node.kubernetes.io/node= [root@k8s-master01 ~]#kubectl label node k8s-master02 storagenode=glusterfs node/k8s-master02 labeled [root@k8s-master01 ~]#kubectl label node k8s-master03 storagenode=glusterfs node/k8s-master03 labeled [root@k8s-master01 ~]#kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master01 Ready matser 18d v1.20.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master01,kubernetes.io/os=linux,node-role.kubernetes.io/matser=,node.kubernetes.io/node=,storagenode=glusterfs k8s-master02 Ready matser 18d v1.20.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master02,kubernetes.io/os=linux,node-role.kubernetes.io/matser=,node.kubernetes.io/node=,storagenode=glusterfs k8s-master03 Ready matser 18d v1.20.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master03,kubernetes.io/os=linux,node-role.kubernetes.io/matser=,node.kubernetes.io/node=,storagenode=glusterfs [root@k8s-master01 ~]#
所有節點加載對應模塊：
[root@k8s-master01 ~]# modprobe dm_snapshot [root@k8s-master01 ~]# modprobe dm_mirror [root@k8s-master01 ~]# modprobe dm_thin_pool
cat >/etc/sysconfig/modules/glusterfs.modules <<EOF #!/bin/bash for kernel_module in dm_snapshot dm_mirror dm_thin_pool;do /sbin/modinfo -F filename ${kernel_module} > /dev/null 2>&1 if [ $? -eq 0 ]; then /sbin/modprobe ${kernel_module} fi done; EOF
[root@kube-node1 ~]# chmod +x /etc/sysconfig/modules/glusterfs.modules
檢查 modprobe 是否加載成功
[root@k8s-master01 ~]#lsmod | egrep '(dm_snapshot|dm_mirror|dm_thin_pool)' dm_thin_pool 69632 0 dm_persistent_data 73728 1 dm_thin_pool dm_bio_prison 20480 1 dm_thin_pool dm_snapshot 40960 0 dm_bufio 28672 2 dm_persistent_data,dm_snapshot dm_mirror 24576 0 dm_region_hash 20480 1 dm_mirror dm_log 20480 2 dm_region_hash,dm_mirror dm_mod 126976 13 dm_thin_pool,dm_log,dm_snapshot,dm_mirror,dm_bufio
二、部署GlusterFS
這里采用容器化方式部署 GFS 集群，也可以使用傳統方式部署。生產環境下建議使用獨立於 kubernetes 集群之外進行部署，之后建立對應的 EndPoints 資源即可。
本次部署采用 DaemonSet 方式，同時保證已經打上標簽的節點上運行了一個 GFS 服務，並且均有提供存儲的磁盤。
Heketi provides a RESTful management interface which can be used to manage the life cycle of GlusterFS volumes. With Heketi, cloud services like OpenStack Manila, Kubernetes, and OpenShift can dynamically provision GlusterFS volumes with any of the supported durability types. Heketi will automatically determine the location for bricks across the cluster, making sure to place bricks and its replicas across different failure domains. Heketi also supports any number of GlusterFS clusters, allowing cloud services to provide network file storage without being limited to a single GlusterFS cluster.
[root@k8s-master01 GFS]#wget https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-client-v7.0.0.linux.amd64.tar.gz --2021-06-29 16:45:53-- https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-client-v7.0.0.linux.amd64.tar.gz Resolving github.com (github.com)... 13.250.177.223 Connecting to github.com (github.com)|13.250.177.223|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-releases.githubusercontent.com/37446835/88bdaaa2-68bf-11e8-8915-37b7ef02cfc9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210629T084555Z&X-Amz-Expires=300&X-Amz-Signature=30369a37c801c4e5d2ee74e8eff1cf4e80b710ecb7f7236549830233f0b438a4&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=37446835&response-content-disposition=attachment%3B%20filename%3Dheketi-client-v7.0.0.linux.amd64.tar.gz&response-content-type=application%2Foctet-stream [following] --2021-06-29 16:45:54-- https://github-releases.githubusercontent.com/37446835/88bdaaa2-68bf-11e8-8915-37b7ef02cfc9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210629T084555Z&X-Amz-Expires=300&X-Amz-Signature=30369a37c801c4e5d2ee74e8eff1cf4e80b710ecb7f7236549830233f0b438a4&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=37446835&response-content-disposition=attachment%3B%20filename%3Dheketi-client-v7.0.0.linux.amd64.tar.gz&response-content-type=application%2Foctet-stream Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.110.154, 185.199.108.154, 185.199.111.154, ... Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.110.154|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10520043 (10M) [application/octet-stream] Saving to: ‘heketi-client-v7.0.0.linux.amd64.tar.gz.1’ 100%[===========================================================================>] 10,520,043 3.13MB/s in 3.6s 2021-06-29 16:45:59 (2.79 MB/s) - ‘heketi-client-v7.0.0.linux.amd64.tar.gz.1’ saved [10520043/10520043] [root@k8s-master01 GFS]#
解壓 heketi-client-v7.0.0.linux.amd64.tar.gz：
[root@k8s-master01 GFS]#tar -xf heketi-client-v7.0.0.linux.amd64.tar.gz
[root@k8s-master01 GFS]#cd heketi-client/share/heketi/kubernetes/
[root@k8s-master01 kubernetes]#ll
total 40
-rw-rw-r-- 1 1000 1000 5222 Jun 5 2018 glusterfs-daemonset.json
-rw-rw-r-- 1 1000 1000 3513 Jun 5 2018 heketi-bootstrap.json
-rw-rw-r-- 1 1000 1000 4113 Jun 5 2018 heketi-deployment.json
-rw-rw-r-- 1 1000 1000 1109 Jun 5 2018 heketi.json
-rw-rw-r-- 1 1000 1000 111 Jun 5 2018 heketi-service-account.json
-rwxrwxr-x 1 1000 1000 584 Jun 5 2018 heketi-start.sh
-rw-rw-r-- 1 1000 1000 977 Jun 5 2018 README.md
-rw-rw-r-- 1 1000 1000 1827 Jun 5 2018 topology-sample.json
[root@k8s-master01 kubernetes]#
創建集群：
[root@k8s-master01 kubernetes]#kubectl apply -f glusterfs-daemonset.json
遇到的報錯后將 glusterfs-daemonset.json 在線工具轉成 glusterfs-daemonset.yaml，修改一些報錯后，最終如下
kind: DaemonSet apiVersion: apps/v1 metadata: name: glusterfs labels: glusterfs: deployment annotations: description: GlusterFS Daemon Set tags: glusterfs spec: selector: matchLabels: glusterfs-node: daemonset template: metadata: name: glusterfs labels: glusterfs-node: daemonset spec: nodeSelector: storagenode: glusterfs hostNetwork: true containers: - image: 'gluster/gluster-centos:latest' imagePullPolicy: IfNotPresent name: glusterfs volumeMounts: - name: glusterfs-heketi mountPath: /var/lib/heketi - name: glusterfs-run mountPath: /run - name: glusterfs-lvm mountPath: /run/lvm - name: glusterfs-etc mountPath: /etc/glusterfs - name: glusterfs-logs mountPath: /var/log/glusterfs - name: glusterfs-config mountPath: /var/lib/glusterd - name: glusterfs-dev mountPath: /dev - name: glusterfs-cgroup mountPath: /sys/fs/cgroup securityContext: capabilities: {} privileged: true readinessProbe: timeoutSeconds: 3 initialDelaySeconds: 60 exec: command: - /bin/bash - '-c' - systemctl status glusterd.service livenessProbe: timeoutSeconds: 3 initialDelaySeconds: 60 exec: command: - /bin/bash - '-c' - systemctl status glusterd.service volumes: - name: glusterfs-heketi hostPath: path: /var/lib/heketi - name: glusterfs-run - name: glusterfs-lvm hostPath: path: /run/lvm - name: glusterfs-etc hostPath: path: /etc/glusterfs - name: glusterfs-logs hostPath: path: /var/log/glusterfs - name: glusterfs-config hostPath: path: /var/lib/glusterd - name: glusterfs-dev hostPath: path: /dev - name: glusterfs-cgroup hostPath: path: /sys/fs/cgroup
再次創建集群：
[root@k8s-master01 kubernetes]#kubectl apply -f glusterfs-daemonset.json daemonset.apps/glusterfs configured [root@k8s-master01 kubernetes]#kubectl get daemonset --all-namespaces NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE devops glusterfs 3 3 3 3 3 storagenode=glusterfs 4h54m kube-system calico-node 3 3 3 3 3 kubernetes.io/os=linux 19d
注意1：此處采用的是默認的掛載方式，可使用其他磁盤作為 GFS 的工作目錄。
注意2：此處創建的 Namespace 為 devops，如果你們創建的話，默認的 Namespace 是default，我的修改過，你們可按需修改。
注意3：可使用 gluster/gluster-centos:gluster4u0_centos7 鏡像。
三、部署Heketi服務
Heketi 是一個提供 RESTful API 管理 GFS 卷的框架，能夠在 kubernetes、Openshift、OpenStack 等雲平台上實現動態存儲資源供應，支持 GFS 多集群管理，便於管理員對 GFS 進行操作，在 Kubernetes 集群中，Pod 將存儲的請求發送至 Heketi，然后 Heketi 控制 GFS 集群創建對應的存儲卷。
查看 Heketi 的 ServiceAccount 對象：
[root@k8s-master01 kubernetes]#cat heketi-service-account.json { "apiVersion": "v1", "kind": "ServiceAccount", "metadata": { "name": "heketi-service-account" } }
創建 Heketi 的 ServiceAccount 對象：
[root@k8s-master01 kubernetes]#kubectl apply -f heketi-service-account.json serviceaccount/heketi-service-account created [root@k8s-master01 kubernetes]#kubectl get sa NAME SECRETS AGE default 1 14d heketi-service-account 1 2s jenkins 1 14d [root@k8s-master01 kubernetes]#
創建 Heketi 對應的權限和 Secret：
[root@k8s-master01 kubernetes]#kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=get,list,watch,create,edit --serviceaccount=devops:heketi-service-account clusterrolebinding.rbac.authorization.k8s.io/heketi-gluster-admin created [root@k8s-master01 kubernetes]#
[root@k8s-master01 kubernetes]#kubectl create secret generic heketi-config-secret --from-file=./heketi.json secret/heketi-config-secret created
在線工具轉換：
kind: List apiVersion: v1 items: - kind: Service apiVersion: v1 metadata: name: deploy-heketi labels: glusterfs: heketi-service deploy-heketi: support annotations: description: Exposes Heketi Service spec: selector: name: deploy-heketi ports: - name: deploy-heketi port: 8080 targetPort: 8080 - kind: Deployment apiVersion: apps/v1 metadata: name: deploy-heketi labels: glusterfs: heketi-deployment deploy-heketi: deployment annotations: description: Defines how to deploy Heketi spec: replicas: 1 selector: matchLabels: glusterfs: heketi-pod deploy-heketi: pod template: metadata: name: deploy-heketi labels: name: deploy-heketi glusterfs: heketi-pod deploy-heketi: pod spec: serviceAccountName: heketi-service-account containers: - image: 'heketi/heketi:dev' imagePullPolicy: Always name: deploy-heketi env: - name: HEKETI_EXECUTOR value: kubernetes - name: HEKETI_DB_PATH value: /var/lib/heketi/heketi.db - name: HEKETI_FSTAB value: /var/lib/heketi/fstab - name: HEKETI_SNAPSHOT_LIMIT value: '14' - name: HEKETI_KUBE_GLUSTER_DAEMONSET value: 'y' ports: - containerPort: 8080 volumeMounts: - name: db mountPath: /var/lib/heketi - name: config mountPath: /etc/heketi readinessProbe: timeoutSeconds: 3 initialDelaySeconds: 3 httpGet: path: /hello port: 8080 livenessProbe: timeoutSeconds: 3 initialDelaySeconds: 30 httpGet: path: /hello port: 8080 volumes: - name: db - name: config secret: secretName: heketi-config-secret
查看當前svc，deploy資源：
[root@k8s-master01 kubernetes]#kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE jenkins NodePort 10.111.57.164 <none> 80:32433/TCP,50000:30752/TCP 12d postgres NodePort 10.99.208.124 <none> 5432:31692/TCP 6d1h sonarqube NodePort 10.102.29.13 <none> 9000:30003/TCP 4d23h [root@k8s-master01 kubernetes]#kubectl get pods NAME READY STATUS RESTARTS AGE glusterfs-2l5jf 1/1 Running 0 38m glusterfs-4l88m 1/1 Running 0 38m glusterfs-6fswc 1/1 Running 0 37m jenkins-0 1/1 Running 6 8d postgres-57f59c66fd-bfg7n 1/1 Running 4 5d23h sonarqube-649955d9b-7hgnz 1/1 Running 3 4d23h [root@k8s-master01 kubernetes]#
再初始化部署 Heketi：
[root@k8s-master01 kubernetes]#kubectl create -f heketi-bootstrap.yaml service/deploy-heketi created deployment.apps/deploy-heketi created [root@k8s-master01 kubernetes]#kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE deploy-heketi ClusterIP 10.102.18.156 <none> 8080/TCP 4s jenkins NodePort 10.111.57.164 <none> 80:32433/TCP,50000:30752/TCP 12d postgres NodePort 10.99.208.124 <none> 5432:31692/TCP 6d1h sonarqube NodePort 10.102.29.13 <none> 9000:30003/TCP 4d23h [root@k8s-master01 kubernetes]#kubectl get pods NAME READY STATUS RESTARTS AGE deploy-heketi-6565469fdf-wcnjc 0/1 ContainerCreating 0 6s glusterfs-2l5jf 1/1 Running 0 42m glusterfs-4l88m 1/1 Running 0 42m glusterfs-6fswc 1/1 Running 0 42m jenkins-0 1/1 Running 6 8d postgres-57f59c66fd-bfg7n 1/1 Running 4 5d23h sonarqube-649955d9b-7hgnz 1/1 Running 3 4d23h [root@k8s-master01 kubernetes]#kubectl rollout status deployments/deploy-heketi Waiting for deployment "deploy-heketi" rollout to finish: 0 of 1 updated replicas are available... deployment "deploy-heketi" successfully rolled out [root@k8s-master01 kubernetes]#kubectl get pods NAME READY STATUS RESTARTS AGE deploy-heketi-6565469fdf-wcnjc 1/1 Running 0 55s glusterfs-2l5jf 1/1 Running 0 43m glusterfs-4l88m 1/1 Running 0 43m glusterfs-6fswc 1/1 Running 0 43m jenkins-0 1/1 Running 6 8d postgres-57f59c66fd-bfg7n 1/1 Running 4 5d23h sonarqube-649955d9b-7hgnz 1/1 Running 3 4d23h [root@k8s-master01 kubernetes]#
四、創建GFS集群
本節使用 Heketi 創建 GFS 集群，其管理方式更加簡單和高效。
復制 heketi-cli 至 /usr/local/bin/：
[root@k8s-master01 bin]#pwd /root/GFS/heketi-client/bin [root@k8s-master01 bin]#ll total 29784 -rwxr-xr-x 1 root root 30498281 Apr 7 21:38 heketi-cli [root@k8s-master01 bin]#cp heketi-cli /usr/local/bin/ [root@k8s-master01 bin]#ls -l /usr/local/bin/ total 582732 -rwxr-xr-x 1 root root 10376657 Apr 17 03:17 cfssl -rwxr-xr-x 1 root root 2277873 Apr 17 03:17 cfssljson -rwxr-xr-x 1 root root 23847904 Aug 25 2020 etcd -rwxr-xr-x 1 root root 17620576 Aug 25 2020 etcdctl -rwxr-xr-x 1 root root 30498281 Jun 29 13:24 heketi-cli -rwxr-xr-x 1 root root 45109248 Jun 17 00:09 helm -rwxr-xr-x 1 root root 118128640 Dec 9 2020 kube-apiserver -rwxr-xr-x 1 root root 112308224 Dec 9 2020 kube-controller-manager -rwxr-xr-x 1 root root 40230912 Dec 9 2020 kubectl -rwxr-xr-x 1 root root 113974120 Dec 9 2020 kubelet -rwxr-xr-x 1 root root 39485440 Dec 9 2020 kube-proxy -rwxr-xr-x 1 root root 42848256 Dec 9 2020 kube-scheduler [root@k8s-master01 bin]#
同步 heketi-cli 至其他node /usr/local/bin/：
[root@k8s-master01 bin]#rsync -avzpP heketi-cli root@192.168.153.42:/usr/local/bin/ sending incremental file list heketi-cli 30,498,281 100% 22.95MB/s 0:00:01 (xfr#1, to-chk=0/1) sent 12,258,006 bytes received 35 bytes 4,903,216.40 bytes/sec total size is 30,498,281 speedup is 2.49 [root@k8s-master01 bin]#rsync -avzpP heketi-cli root@192.168.153.43:/usr/local/bin/ sending incremental file list heketi-cli 30,498,281 100% 21.81MB/s 0:00:01 (xfr#1, to-chk=0/1) sent 12,258,006 bytes received 35 bytes 4,903,216.40 bytes/sec total size is 30,498,281 speedup is 2.49 [root@k8s-master01 bin]#rsync -avzpP heketi-cli root@192.168.153.44:/usr/local/bin/ The authenticity of host '192.168.153.44 (192.168.153.44)' can't be established. ECDSA key fingerprint is SHA256:AqR5ZL4OLkrfdBddeQVMjgrUGyAGLw1C7mTCQXAy7xE. ECDSA key fingerprint is MD5:18:1c:bd:c3:e6:0c:24:b9:1e:09:e7:1a:25:ee:e8:e0. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.153.44' (ECDSA) to the list of known hosts. root@192.168.153.44's password: bash: rsync: command not found rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: remote command not found (code 127) at io.c(226) [sender=3.1.2] [root@k8s-master01 bin]#rsync -avzpP heketi-cli root@192.168.153.44:/usr/local/bin/ root@192.168.153.44's password: sending incremental file list heketi-cli 30,498,281 100% 22.56MB/s 0:00:01 (xfr#1, to-chk=0/1) sent 12,258,006 bytes received 35 bytes 2,724,009.11 bytes/sec total size is 30,498,281 speedup is 2.49 [root@k8s-master01 bin]#
heketi-cli 版本號：
[root@k8s-master01 ~]#heketi-cli -v heketi-cli v7.0.0
修改 topology-sample，manage 為 GFS 管理服務的節點（Node）主機名，storage 為節點的 IP 地址，devices 為節點上裸設備，也就是用於提供存儲的磁盤最好使用裸設備：
[root@k8s-master01 kubernetes]#cat topology-sample.json { "clusters": [ { "nodes": [ { "node": { "hostnames": { "manage": [ "k8s-master01" ], "storage": [ "192.168.153.41" ] }, "zone": 1 }, "devices": [ { "name": "/dev/sdb", "destroydata": false } ] }, { "node": { "hostnames": { "manage": [ "k8s-master02" ], "storage": [ "192.168.153.42" ] }, "zone": 1 }, "devices": [ { "name": "/dev/sdb", "destroydata": false } ] }, { "node": { "hostnames": { "manage": [ "k8s-master03" ], "storage": [ "192.168.153.43" ] }, "zone": 1 }, "devices": [ { "name": "/dev/sdb", "destroydata": false } ] } ] } ] }
查看當前 Heketi 的 ClusterIP：
[root@k8s-master01 kubernetes]#kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE deploy-heketi ClusterIP 10.102.18.156 <none> 8080/TCP 5m42s jenkins NodePort 10.111.57.164 <none> 80:32433/TCP,50000:30752/TCP 12d postgres NodePort 10.99.208.124 <none> 5432:31692/TCP 6d1h sonarqube NodePort 10.102.29.13 <none> 9000:30003/TCP 4d23h [root@k8s-master01 kubernetes]#curl 10.102.18.156:8080/hello Hello from Heketi[root@k8s-master01 kubernetes]# [root@k8s-master01 kubernetes]# [root@k8s-master01 kubernetes]#export HEKETI_CLI_SERVER="http://10.102.18.156:8080" [root@k8s-master01 kubernetes]#export |grep HEKETI declare -x HEKETI_CLI_SERVER="http://10.102.18.156:8080"
使用 Heketi 創建 GFS 集群：
[root@k8s-master01 kubernetes]#heketi-cli topology load --json=topology-sample.json
Error: Unable to get topology information: Invalid JWT token: Token missing iss claim
這是因為新版本的 heketi 在創建 gfs 集群時需要帶上參數，聲明用戶名及密碼，相應值在 heketi.json 文件中配置，即：
[root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology load --json=topology-sample.json Creating cluster ... ID: 8e17d5f80328a9e8c7d141ab4034e2e6 Allowing file volumes on cluster. Allowing block volumes on cluster. Creating node k8s-master01 ... Unable to create node: New Node doesn't have glusterd running Creating node k8s-master02 ... Unable to create node: New Node doesn't have glusterd running Creating node k8s-master03 ... Unable to create node: New Node doesn't have glusterd running
不報之前的錯誤，出現新的錯誤 Unable to create node: New Node doesn't have glusterd running，查看 deployment 的 pod 日志：
[root@k8s-master01 kubernetes]#kubectl get pods NAME READY STATUS RESTARTS AGE deploy-heketi-6565469fdf-wcnjc 1/1 Running 0 12m glusterfs-2l5jf 1/1 Running 0 54m glusterfs-4l88m 1/1 Running 0 54m glusterfs-6fswc 1/1 Running 0 54m jenkins-0 1/1 Running 6 8d postgres-57f59c66fd-bfg7n 1/1 Running 4 6d sonarqube-649955d9b-7hgnz 1/1 Running 3 4d23h
日志顯示 Failed to get list of pods：
[root@k8s-master01 kubernetes]#kubectl logs -f deploy-heketi-6565469fdf-wcnjc [heketi] ERROR 2021/06/29 09:10:57 heketi/apps/glusterfs/app_node.go:108:glusterfs.(*App).NodeAdd: New Node doesn't have glusterd running
[negroni] 2021-06-29T09:10:57Z | 400 | 3.867841ms | 10.102.18.156:8080 | POST /nodes [cmdexec] INFO 2021/06/29 09:10:57 Check Glusterd service status in node k8s-master03 [negroni] 2021-06-29T09:10:57Z | 400 | 4.219108ms | 10.102.18.156:8080 | POST /nodes [kubeexec] ERROR 2021/06/29 09:10:57 heketi/pkg/remoteexec/kube/target.go:134:kube.TargetDaemonSet.GetTargetPod: pods is forbidden: User "system:serviceaccount:devops:heketi-service-account" cannot list resource "pods" in API group "" in the namespace "devops" [kubeexec] ERROR 2021/06/29 09:10:57 heketi/pkg/remoteexec/kube/target.go:135:kube.TargetDaemonSet.GetTargetPod: Failed to get list of pods [cmdexec] ERROR 2021/06/29 09:10:57 heketi/executors/cmdexec/peer.go:80:cmdexec.(*CmdExecutor).GlusterdCheck: Failed to get list of pods [heketi] ERROR 2021/06/29 09:10:57 heketi/apps/glusterfs/app_node.go:107:glusterfs.(*App).NodeAdd: Failed to get list of pods [heketi] ERROR 2021/06/29 09:10:57 heketi/apps/glusterfs/app_node.go:108:glusterfs.(*App).NodeAdd: New Node doesn't have glusterd running
[negroni] 2021-06-29T09:10:57Z | 200 | 353.242µs | 10.102.18.156:8080 | GET /clusters/8e17d5f80328a9e8c7d141ab4034e2e6 [heketi] INFO 2021/06/29 09:10:57 Deleted cluster [8e17d5f80328a9e8c7d141ab4034e2e6] [negroni] 2021-06-29T09:10:57Z | 200 | 3.360667ms | 10.102.18.156:8080 | DELETE /clusters/8e17d5f80328a9e8c7d141ab4034e2e6 [heketi] INFO 2021/06/29 09:12:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:12:03 Cleaned 0 nodes from health cache [heketi] INFO 2021/06/29 09:14:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:14:03 Cleaned 0 nodes from health cache
解決辦法: 創建 role 並綁定到 ServiceAccount，
[root@k8s-master01 kubernetes]#kubectl create clusterrole foo --verb=get,list,watch,create --resource=pods,pods/status,pods/exec
clusterrole.rbac.authorization.k8s.io/foo created
再次執行gluster添加命令，觀察日志：
[heketi] INFO 2021/06/29 09:12:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:12:03 Cleaned 0 nodes from health cache [heketi] INFO 2021/06/29 09:14:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:14:03 Cleaned 0 nodes from health cache [heketi] INFO 2021/06/29 09:16:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:16:03 Cleaned 0 nodes from health cache [heketi] INFO 2021/06/29 09:18:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:18:03 Cleaned 0 nodes from health cache [heketi] INFO 2021/06/29 09:20:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:20:03 Cleaned 0 nodes from health cache
再使用 Heketi 創建 GFS 集群：
[root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology load --json=topology-sample.json Creating cluster ... ID: ba8f4a8a29e5c436d0c84c45ad9e00d3 Allowing file volumes on cluster. Allowing block volumes on cluster. Creating node k8s-master01 ... Unable to create node: New Node doesn't have glusterd running Creating node k8s-master02 ... Unable to create node: New Node doesn't have glusterd running Creating node k8s-master03 ... Unable to create node: New Node doesn't have glusterd running [root@k8s-master01 kubernetes]#
再打開一個窗口，查看日志：
[heketi] INFO 2021/06/29 09:42:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:42:03 Cleaned 0 nodes from health cache [heketi] INFO 2021/06/29 09:44:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:44:03 Cleaned 0 nodes from health cache [heketi] INFO 2021/06/29 09:46:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:46:03 Cleaned 0 nodes from health cache [heketi] INFO 2021/06/29 09:48:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:48:03 Cleaned 0 nodes from health cache [heketi] INFO 2021/06/29 09:50:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:50:03 Cleaned 0 nodes from health cache [negroni] 2021-06-29T09:51:42Z | 200 | 102.535µs | 10.102.18.156:8080 | GET /clusters [negroni] 2021-06-29T09:51:42Z | 201 | 3.432335ms | 10.102.18.156:8080 | POST /clusters [cmdexec] INFO 2021/06/29 09:51:42 Check Glusterd service status in node k8s-master01 [kubeexec] ERROR 2021/06/29 09:51:42 heketi/pkg/remoteexec/kube/target.go:134:kube.TargetDaemonSet.GetTargetPod: pods is forbidden: User "system:serviceaccount:devops:heketi-service-account" cannot list resource "pods" in API group "" in the namespace "devops" [kubeexec] ERROR 2021/06/29 09:51:42 heketi/pkg/remoteexec/kube/target.go:135:kube.TargetDaemonSet.GetTargetPod: Failed to get list of pods [cmdexec] ERROR 2021/06/29 09:51:42 heketi/executors/cmdexec/peer.go:80:cmdexec.(*CmdExecutor).GlusterdCheck: Failed to get list of pods [heketi] ERROR 2021/06/29 09:51:42 heketi/apps/glusterfs/app_node.go:107:glusterfs.(*App).NodeAdd: Failed to get list of pods [heketi] ERROR 2021/06/29 09:51:42 heketi/apps/glusterfs/app_node.go:108:glusterfs.(*App).NodeAdd: New Node doesn't have glusterd running [negroni] 2021-06-29T09:51:42Z | 400 | 34.875543ms | 10.102.18.156:8080 | POST /nodes [cmdexec] INFO 2021/06/29 09:51:42 Check Glusterd service status in node k8s-master02 [kubeexec] ERROR 2021/06/29 09:51:42 heketi/pkg/remoteexec/kube/target.go:134:kube.TargetDaemonSet.GetTargetPod: pods is forbidden: User "system:serviceaccount:devops:heketi-service-account" cannot list resource "pods" in API group "" in the namespace "devops" [kubeexec] ERROR 2021/06/29 09:51:42 heketi/pkg/remoteexec/kube/target.go:135:kube.TargetDaemonSet.GetTargetPod: Failed to get list of pods [cmdexec] ERROR 2021/06/29 09:51:42 heketi/executors/cmdexec/peer.go:80:cmdexec.(*CmdExecutor).GlusterdCheck: Failed to get list of pods [heketi] ERROR 2021/06/29 09:51:42 heketi/apps/glusterfs/app_node.go:107:glusterfs.(*App).NodeAdd: Failed to get list of pods [heketi] ERROR 2021/06/29 09:51:42 heketi/apps/glusterfs/app_node.go:108:glusterfs.(*App).NodeAdd: New Node doesn't have glusterd running [negroni] 2021-06-29T09:51:42Z | 400 | 5.317761ms | 10.102.18.156:8080 | POST /nodes [cmdexec] INFO 2021/06/29 09:51:42 Check Glusterd service status in node k8s-master03 [kubeexec] ERROR 2021/06/29 09:51:42 heketi/pkg/remoteexec/kube/target.go:134:kube.TargetDaemonSet.GetTargetPod: pods is forbidden: User "system:serviceaccount:devops:heketi-service-account" cannot list resource "pods" in API group "" in the namespace "devops" [kubeexec] ERROR 2021/06/29 09:51:42 heketi/pkg/remoteexec/kube/target.go:135:kube.TargetDaemonSet.GetTargetPod: Failed to get list of pods [cmdexec] ERROR 2021/06/29 09:51:42 heketi/executors/cmdexec/peer.go:80:cmdexec.(*CmdExecutor).GlusterdCheck: Failed to get list of pods [heketi] ERROR 2021/06/29 09:51:42 heketi/apps/glusterfs/app_node.go:107:glusterfs.(*App).NodeAdd: Failed to get list of pods [heketi] ERROR 2021/06/29 09:51:42 heketi/apps/glusterfs/app_node.go:108:glusterfs.(*App).NodeAdd: New Node doesn't have glusterd running [negroni] 2021-06-29T09:51:42Z | 400 | 4.586467ms | 10.102.18.156:8080 | POST /nodes [negroni] 2021-06-29T09:51:42Z | 200 | 237.734µs | 10.102.18.156:8080 | GET /clusters/ba8f4a8a29e5c436d0c84c45ad9e00d3 [heketi] INFO 2021/06/29 09:51:42 Deleted cluster [ba8f4a8a29e5c436d0c84c45ad9e00d3] [negroni] 2021-06-29T09:51:42Z | 200 | 895.405µs | 10.102.18.156:8080 | DELETE /clusters/ba8f4a8a29e5c436d0c84c45ad9e00d3 [heketi] INFO 2021/06/29 09:52:03 Starting Node Health Status refresh [heketi] INFO 2021/06/29 09:52:03 Cleaned 0 nodes from health cache
終於找到為啥了：
[kubeexec] ERROR 2021/06/29 09:51:42 heketi/pkg/remoteexec/kube/target.go:134:kube.TargetDaemonSet.GetTargetPod: pods is forbidden: User "system:serviceaccount:devops:heketi-service-account" cannot list resource "pods" in API group "" in the namespace "devops"
刪除clusterrolebinding，重新創建新的clusterrolebinding后，成功！
[root@k8s-master01 kubernetes]#kubectl delete clusterrolebinding heketi-gluster-admin clusterrolebinding.rbac.authorization.k8s.io "heketi-gluster-admin" deleted [root@k8s-master01 kubernetes]#kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=cluster-admin --serviceaccount=devops:heketi-service-account clusterrolebinding.rbac.authorization.k8s.io/heketi-gluster-admin created [root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology load --json=topology-sample.json Creating cluster ... ID: c934f76dfae0fc21e0d8820c5e2ee401 Allowing file volumes on cluster. Allowing block volumes on cluster. Creating node k8s-master01 ... ID: aaf700d47bfa7d2c0bd2a08e66a0d1f3 Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Device /dev/sdb excluded by a filter. Creating node k8s-master02 ... ID: 04b711a1eb44601f8d6b5c002b28aaf9 Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Device /dev/sdb excluded by a filter. Creating node k8s-master03 ... ID: cca811a225c58034b3d79fc2c2d01be4 Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Device /dev/sdb excluded by a filter. [root@k8s-master01 kubernetes]#
第二天打開電腦，開機后發現 Heketi 配置失效了，原來是沒有配置持久卷，就把上面部分步驟重新做一遍，在到執行 Heketi 創建 GFS 集群時，又報錯了，命令執行報錯如下：
[root@k8s-master01 kubernetes]#kubectl logs -f deploy-heketi-6565469fdf-n2wnh -n devops^C [root@k8s-master01 kubernetes]#kubectl create clusterrole foo --verb=get,list,watch,create --resource=pods,pods/status,pods/exec Error from server (AlreadyExists): clusterroles.rbac.authorization.k8s.io "foo" already exists [root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology load --json=topology-sample.json Found node k8s-master01 on cluster 221eddbc9d9ec714e6de6c19f5e86e09 Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Device /dev/sdb excluded by a filter. Found node k8s-master02 on cluster 221eddbc9d9ec714e6de6c19f5e86e09 Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Device /dev/sdb excluded by a filter. Found node k8s-master03 on cluster 221eddbc9d9ec714e6de6c19f5e86e09 Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Device /dev/sdb excluded by a filter. [root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology load --json=topology-sample.json Found node k8s-master01 on cluster 221eddbc9d9ec714e6de6c19f5e86e09 Adding device /dev/sdb ... OK Found node k8s-master02 on cluster 221eddbc9d9ec714e6de6c19f5e86e09 Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Device /dev/sdb excluded by a filter. Found node k8s-master03 on cluster 221eddbc9d9ec714e6de6c19f5e86e09 Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Device /dev/sdb excluded by a filter.
解決辦法如下，先查看pod的日志
[root@k8s-master01 kubernetes]#kubectl logs -f deploy-heketi-6565469fdf-n2wnh [negroni] 2021-07-01T02:46:30Z | 200 | 77.337µs | 10.102.18.156:8080 | GET /clusters [negroni] 2021-07-01T02:46:30Z | 200 | 210.78µs | 10.102.18.156:8080 | GET /clusters/221eddbc9d9ec714e6de6c19f5e86e09 [negroni] 2021-07-01T02:46:30Z | 200 | 446.48µs | 10.102.18.156:8080 | GET /nodes/1e515e038850e2f725125cd55a19d278 [negroni] 2021-07-01T02:46:30Z | 200 | 256.658µs | 10.102.18.156:8080 | GET /nodes/4154491e2eb27e7017f9b8dab8046076 [negroni] 2021-07-01T02:46:30Z | 200 | 244.247µs | 10.102.18.156:8080 | GET /nodes/45d66ab47e0e299b25c66a57c667b1de [negroni] 2021-07-01T02:46:30Z | 200 | 334.914µs | 10.102.18.156:8080 | GET /clusters/221eddbc9d9ec714e6de6c19f5e86e09 [negroni] 2021-07-01T02:46:30Z | 200 | 460.879µs | 10.102.18.156:8080 | GET /clusters/221eddbc9d9ec714e6de6c19f5e86e09 [heketi] INFO 2021/07/01 02:46:30 Adding device /dev/sdb to node 4154491e2eb27e7017f9b8dab8046076 [negroni] 2021-07-01T02:46:30Z | 202 | 4.574525ms | 10.102.18.156:8080 | POST /devices [asynchttp] INFO 2021/07/01 02:46:30 Started job 0c89db58f2ffcf410c0777d2f20a08b3 [negroni] 2021-07-01T02:46:30Z | 200 | 74.084µs | 10.102.18.156:8080 | GET /queue/0c89db58f2ffcf410c0777d2f20a08b3 [kubeexec] DEBUG 2021/07/01 02:46:30 heketi/pkg/remoteexec/log/commandlog.go:34:log.(*CommandLogger).Before: Will run command [/usr/sbin/lvm pvcreate -qq --metadatasize=128M --dataalignment=256K '/dev/sdb'] on [pod:glusterfs-d2glt c:glusterfs ns:devops (from host:k8s-master02 selector:glusterfs-node)] [kubeexec] DEBUG 2021/07/01 02:46:30 heketi/pkg/remoteexec/kube/exec.go:72:kube.ExecCommands: Current kube connection count: 0 [kubeexec] ERROR 2021/07/01 02:46:30 heketi/pkg/remoteexec/log/commandlog.go:56:log.(*CommandLogger).Error: Failed to run command [/usr/sbin/lvm pvcreate -qq --metadatasize=128M --dataalignment=256K '/dev/sdb'] on [pod:glusterfs-d2glt c:glusterfs ns:devops (from host:k8s-master02 selector:glusterfs-node)]: Err[command terminated with exit code 5]: Stdout []: Stderr [WARNING: dos signature detected on /dev/sdb at offset 510. Wipe it? [y/n]: [n] Aborted wiping of dos. 1 existing signature left on the device. ] [kubeexec] DEBUG 2021/07/01 02:46:30 heketi/pkg/remoteexec/log/commandlog.go:34:log.(*CommandLogger).Before: Will run command [/usr/sbin/lvm pvs -o pv_name,pv_uuid,vg_name --reportformat=json /dev/sdb] on [pod:glusterfs-d2glt c:glusterfs ns:devops (from host:k8s-master02 selector:glusterfs-node)] [kubeexec] DEBUG 2021/07/01 02:46:30 heketi/pkg/remoteexec/kube/exec.go:72:kube.ExecCommands: Current kube connection count: 0 [asynchttp] INFO 2021/07/01 02:46:30 Completed job 0c89db58f2ffcf410c0777d2f20a08b3 in 343.470109ms [kubeexec] ERROR 2021/07/01 02:46:30 heketi/pkg/remoteexec/log/commandlog.go:56:log.(*CommandLogger).Error: Failed to run command [/usr/sbin/lvm pvs -o pv_name,pv_uuid,vg_name --reportformat=json /dev/sdb] on [pod:glusterfs-d2glt c:glusterfs ns:devops (from host:k8s-master02 selector:glusterfs-node)]: Err[command terminated with exit code 5]: Stdout [ { "report": [ { "pv": [ ] } ] } ]: Stderr [ Failed to find physical volume "/dev/sdb". ] [negroni] 2021-07-01T02:46:31Z | 500 | 75.41µs | 10.102.18.156:8080 | GET /queue/0c89db58f2ffcf410c0777d2f20a08b3 [negroni] 2021-07-01T02:46:31Z | 200 | 200.176µs | 10.102.18.156:8080 | GET /clusters/221eddbc9d9ec714e6de6c19f5e86e09 [heketi] INFO 2021/07/01 02:46:31 Adding device /dev/sdb to node 45d66ab47e0e299b25c66a57c667b1de [negroni] 2021-07-01T02:46:31Z | 202 | 1.013933ms | 10.102.18.156:8080 | POST /devices [asynchttp] INFO 2021/07/01 02:46:31 Started job eee9aed41f9be12d74592b3f1d9212ef [negroni] 2021-07-01T02:46:31Z | 200 | 73.998µs | 10.102.18.156:8080 | GET /queue/eee9aed41f9be12d74592b3f1d9212ef [kubeexec] DEBUG 2021/07/01 02:46:31 heketi/pkg/remoteexec/log/commandlog.go:34:log.(*CommandLogger).Before: Will run command [/usr/sbin/lvm pvcreate -qq --metadatasize=128M --dataalignment=256K '/dev/sdb'] on [pod:glusterfs-ttv65 c:glusterfs ns:devops (from host:k8s-master03 selector:glusterfs-node)] [kubeexec] DEBUG 2021/07/01 02:46:31 heketi/pkg/remoteexec/kube/exec.go:72:kube.ExecCommands: Current kube connection count: 0 [kubeexec] ERROR 2021/07/01 02:46:31 heketi/pkg/remoteexec/log/commandlog.go:56:log.(*CommandLogger).Error: Failed to run command [/usr/sbin/lvm pvcreate -qq --metadatasize=128M --dataalignment=256K '/dev/sdb'] on [pod:glusterfs-ttv65 c:glusterfs ns:devops (from host:k8s-master03 selector:glusterfs-node)]: Err[command terminated with exit code 5]: Stdout []: Stderr [WARNING: dos signature detected on /dev/sdb at offset 510. Wipe it? [y/n]: [n] Aborted wiping of dos. 1 existing signature left on the device. ] [kubeexec] DEBUG 2021/07/01 02:46:31 heketi/pkg/remoteexec/log/commandlog.go:34:log.(*CommandLogger).Before: Will run command [/usr/sbin/lvm pvs -o pv_name,pv_uuid,vg_name --reportformat=json /dev/sdb] on [pod:glusterfs-ttv65 c:glusterfs ns:devops (from host:k8s-master03 selector:glusterfs-node)] [kubeexec] DEBUG 2021/07/01 02:46:31 heketi/pkg/remoteexec/kube/exec.go:72:kube.ExecCommands: Current kube connection count: 0 [kubeexec] ERROR 2021/07/01 02:46:31 heketi/pkg/remoteexec/log/commandlog.go:56:log.(*CommandLogger).Error: Failed to run command [/usr/sbin/lvm pvs -o pv_name,pv_uuid,vg_name --reportformat=json /dev/sdb] on [pod:glusterfs-ttv65 c:glusterfs ns:devops (from host:k8s-master03 selector:glusterfs-node)]: Err[command terminated with exit code 5]: Stdout [ { "report": [ { "pv": [ ] } ] } ]: Stderr [ Failed to find physical volume "/dev/sdb".
發現：
[kubeexec] ERROR 2021/07/01 02:46:30 heketi/pkg/remoteexec/log/commandlog.go:56:log.(*CommandLogger).Error: Failed to run command [/usr/sbin/lvm pvcreate -qq --metadatasize=128M --dataalignment=256K '/dev/sdb'] on [pod:glusterfs-d2glt c:glusterfs ns:devops (from host:k8s-master02 selector:glusterfs-node)]: Err[command terminated with exit code 5]: Stdout []: Stderr [WARNING: dos signature detected on /dev/sdb at offset 510. Wipe it? [y/n]: [n] Aborted wiping of dos. 1 existing signature left on the device. ]
先umount /dev/sdb，再執行：
[root@k8s-master01 ~]#parted /dev/sdb GNU Parted 3.1 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel msdos Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue? Yes/No? yes (parted) quit Information: You may need to update /etc/fstab. [root@k8s-master01 ~]#pvcreate /dev/sdb WARNING: dos signature detected on /dev/sdb at offset 510. Wipe it? [y/n]: y Wiping dos signature on /dev/sdb. Physical volume "/dev/sdb" successfully created. [root@k8s-master01 ~]#
再執行 Heketi 創建 GFS 集群，再次成功！
[root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology load --json=topology-sample.json Found node k8s-master01 on cluster 221eddbc9d9ec714e6de6c19f5e86e09 Found device /dev/sdb Found node k8s-master02 on cluster 221eddbc9d9ec714e6de6c19f5e86e09 Adding device /dev/sdb ... OK Found node k8s-master03 on cluster 221eddbc9d9ec714e6de6c19f5e86e09 Adding device /dev/sdb ... OK [root@k8s-master01 kubernetes]#
好久沒有更新了，今天打算繼續更新，重新打開虛擬機，遇到報錯。檢查步驟，執行 heketi-cli topology load --json=topology-sample.json 時報錯如下：
[root@k8s-master01 kubernetes]#heketi-cli topology load --json=topology-sample.json Error: Unable to get topology information: Invalid JWT token: Token missing iss claim [root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology load --json=topology-sample.json Found node k8s-master01 on cluster 181637c237552c5367734c1956a6222d Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Can't open /dev/sdb exclusively. Mounted filesystem? Can't open /dev/sdb exclusively. Mounted filesystem? Found node k8s-master02 on cluster 181637c237552c5367734c1956a6222d Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (already initialized or contains data?): Can't open /dev/sdb exclusively. Mounted filesystem? Can't open /dev/sdb exclusively. Mounted filesystem? Found node k8s-master03 on cluster 181637c237552c5367734c1956a6222d Adding device /dev/sdb ... Unable to add device: Initializing device /dev/sdb failed (aleady contains Physical Volume B9EZvW-KxuV-SQ0l-5Hau-ODL0-S2ve-Rd8d7i): Can't initialize physical volume "/dev/sdb" of volume group "vg_936bddeece0f76fec700998c5520c6eb" without -ff /dev/sdb: physical volume not initialized.
解決辦法：
1、重新掛載 /dev/sdb 2、mkfs -t ext4 /dev/sdb 3、mount -a
再執行，還是報錯，但是報錯信息發生變化：
[root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology load --json=topology-sample.json Found node k8s-master01 on cluster 181637c237552c5367734c1956a6222d Adding device /data/sdb ... Unable to add device: Initializing device /data/sdb failed (already initialized or contains data?): Device /data/sdb not found. Found node k8s-master02 on cluster 181637c237552c5367734c1956a6222d Adding device /data/sdb ... Unable to add device: Initializing device /data/sdb failed (already initialized or contains data?): Device /data/sdb not found. Found node k8s-master03 on cluster 181637c237552c5367734c1956a6222d Adding device /data/sdb ... Unable to add device: Initializing device /data/sdb failed (already initialized or contains data?): Device /data/sdb not found.
查看是否有device
[root@k8s-master01 ~]#lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb 8:16 0 500G 0 disk /data/sdb sr0 11:0 1 1G 0 rom sda 8:0 0 200G 0 disk ├─sda2 8:2 0 199G 0 part │ ├─centos-swap 253:1 0 2G 0 lvm │ ├─centos-home 253:2 0 147G 0 lvm /home │ └─centos-root 253:0 0 50G 0 lvm / └─sda1 8:1 0 1G 0 part /boot [root@k8s-master01 ~]#
[root@k8s-master01 ~]#fdisk /dev/sdb -l Disk /dev/sdb: 536.9 GB, 536870912000 bytes, 1048576000 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes [root@k8s-master01 ~]#
存在sdb，再進入 gfs pod，執行 pvcreate /dev/sdb
進入docker容器 ： docker exec -ti <your-container-name> /bin/sh 進入Kubernetes的pod： kubectl exec -ti <your-pod-name> -n <your-namespace> -- /bin/sh
[root@k8s-master01 ~]#kubectl exec -it glusterfs-7cf9p -n devops -- /bin/bash [root@k8s-master02 /]# pvcreate /dev/sdb Can't open /dev/sdb exclusively. Mounted filesystem? Can't open /dev/sdb exclusively. Mounted filesystem? [root@k8s-master02 /]# pvcreate /dev/sdb -v Can't open /dev/sdb exclusively. Mounted filesystem? Can't open /dev/sdb exclusively. Mounted filesystem? [root@k8s-master02 /]# pvcreate /dev/sdb -vv devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ] global/lvmetad_update_wait_time not found in config: defaulting to 10 devices/filter not found in config: defaulting to filter = [ "a|.*/|" ] devices/cache not found in config: defaulting to /etc/lvm/cache/.cache metadata/record_lvs_history not found in config: defaulting to 0 File-based locking selected. /dev/initctl: stat failed: No such file or directory metadata/pvmetadataignore not found in config: defaulting to 0 metadata/pvmetadatasize not found in config: defaulting to 255 metadata/pvmetadatacopies not found in config: defaulting to 1 report/output_format not found in config: defaulting to basic log/report_command_log not found in config: defaulting to 0 Locking /run/lock/lvm/P_orphans WB /dev/sdb: size is 1048576000 sectors /dev/sdb: using cached size 1048576000 sectors /dev/initctl: stat failed: No such file or directory /dev/sda: size is 419430400 sectors /dev/sda1: size is 2097152 sectors /dev/sda1: using cached size 2097152 sectors /dev/sda2: size is 417331200 sectors /dev/sda2: using cached size 417331200 sectors /dev/sdb: using cached size 1048576000 sectors /dev/sdb: using cached size 1048576000 sectors Locking /run/lock/lvm/V_centos RB Reading VG centos Jm6F2D-GQCR-oFvp-Ob72-dCw4-ZKlK-dN0HXu /dev/sda2: using cached size 417331200 sectors Processing PV /dev/sda2 in VG centos. Unlocking /run/lock/lvm/V_centos Locking #orphans_lvm1 already done Reading VG #orphans_lvm1 Unlocking /run/lock/lvm/P_orphans Locking /run/lock/lvm/P_orphans WB Reading VG #orphans_lvm1 Locking #orphans_pool already done Reading VG #orphans_pool Unlocking /run/lock/lvm/P_orphans Locking /run/lock/lvm/P_orphans WB Reading VG #orphans_pool Locking #orphans_lvm2 already done Reading VG #orphans_lvm2 Unlocking /run/lock/lvm/P_orphans Locking /run/lock/lvm/P_orphans WB Reading VG #orphans_lvm2 Processing device /dev/sda1. Processing device /dev/sdb. Can't open /dev/sdb exclusively. Mounted filesystem? /dev/initctl: stat failed: No such file or directory Can't open /dev/sdb exclusively. Mounted filesystem? Unlocking /run/lock/lvm/P_orphans [root@k8s-master02 /]#
看到 Can't open /dev/sdb exclusively. Mounted filesystem? /dev/initctl: stat failed: No such file or directory Can't open /dev/sdb exclusively. Mounted filesystem? Unlocking /run/lock/lvm/P_orphans
解決辦法如下：
[root@k8s-master01 kubernetes]#kubectl get pods -n devops NAME READY STATUS RESTARTS AGE deploy-heketi-6565469fdf-kpfvl 1/1 Running 17 70d glusterfs-7cf9p 1/1 Running 24 70d glusterfs-7r4lf 1/1 Running 23 70d glusterfs-9vpg6 1/1 Running 18 70d jenkins-0 1/1 Running 11 60d postgres-57f59c66fd-lw48x 1/1 Running 19 72d sonarqube-649955d9b-fwnnx 1/1 Running 1 22d [root@k8s-master01 kubernetes]#kubectl exec -it glusterfs-7r4lf -n devops -- /bin/bash [root@k8s-master01 /]# pvcreate /dev/sdb WARNING: ext4 signature detected on /dev/sdb at offset 1080. Wipe it? [y/n]: y Wiping ext4 signature on /dev/sdb. Physical volume "/dev/sdb" successfully created. [root@k8s-master01 /]# exit exit [root@k8s-master01 kubernetes]#kubectl exec -it glusterfs-7cf9p -n devops -- /bin/bash [root@k8s-master02 /]# pvcreate /dev/sdb WARNING: ext4 signature detected on /dev/sdb at offset 1080. Wipe it? [y/n]: y Wiping ext4 signature on /dev/sdb. Physical volume "/dev/sdb" successfully created. [root@k8s-master02 /]# exit exit [root@k8s-master01 kubernetes]#kubectl exec -it glusterfs-9vpg6 -n devops -- /bin/bash [root@k8s-master03 /]# pvcreate /dev/sdb WARNING: ext4 signature detected on /dev/sdb at offset 1080. Wipe it? [y/n]: y Wiping ext4 signature on /dev/sdb. Physical volume "/dev/sdb" successfully created. [root@k8s-master03 /]# exit exit [root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology load --json=topology-sample.json Found node k8s-master01 on cluster 181637c237552c5367734c1956a6222d Found device /dev/sdb Found node k8s-master02 on cluster 181637c237552c5367734c1956a6222d Adding device /dev/sdb ... OK Found node k8s-master03 on cluster 181637c237552c5367734c1956a6222d Adding device /dev/sdb ... OK [root@k8s-master01 kubernetes]#
安裝完成，遇到問題，又再次成功！
以下命令是常用命令整理：
查看 topology info
heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' topology info [flags] 查看 node info heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' node info [node_id] [flags] 查看 device info heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' device info [device_id] [flags] 查看 cluster list heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' cluster list [flags] 查看 cluster info heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' cluster info [cluster_id] [flags]
查看 node info：
[root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' node info 0b5ec08be973e47535ed25a36b44141a
Node Id: 0b5ec08be973e47535ed25a36b44141a
State: online
Cluster Id: 1a24bdf9bc6a82a0530dcfbff24aad54
Zone: 1 Management Hostname: k8s-master03 Storage Hostname: 192.168.153.43 Devices: Id:936bddeece0f76fec700998c5520c6eb Name:/dev/sdb State:online Size (GiB):499 Used (GiB):2 Free (GiB):497 Bricks:1 [root@k8s-master01 kubernetes]#
查看 device info：
[root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' device info 936bddeece0f76fec700998c5520c6eb
Device Id: 936bddeece0f76fec700998c5520c6eb
Name: /dev/sdb State: online Size (GiB): 499 Used (GiB): 2 Free (GiB): 497 Bricks: Id:6b33d59f6da059a7d8e38696f8549001 Size (GiB):2 Path: /var/lib/heketi/mounts/vg_936bddeece0f76fec700998c5520c6eb/brick_6b33d59f6da059a7d8e38696f8549001/brick [root@k8s-master01 kubernetes]#
查看 cluster list：
[root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' cluster list
Clusters:
Id:1a24bdf9bc6a82a0530dcfbff24aad54 [file][block] [root@k8s-master01 kubernetes]#
查看 cluster info：
[root@k8s-master01 kubernetes]#heketi-cli -s $HEKETI_CLI_SERVER --user admin --secret 'My Secret' cluster info 1a24bdf9bc6a82a0530dcfbff24aad54
Cluster id: 1a24bdf9bc6a82a0530dcfbff24aad54 Nodes: 0b5ec08be973e47535ed25a36b44141a 3bfa2d1f005fe540df39843b8f8ea283 9c678039658836b8ed4e96c97bdc8c2b Volumes: Block: true File: true [root@k8s-master01 kubernetes]#