K8S中使用gfs當存儲

一開始是參考《再也不踩坑的Kubernetes實戰指南》中的以容器方式去安裝GFS集群的，但是在安裝Heketi的過程中卡了將近幾十分鍾也沒有好，就換成來EndPoints安裝。

環境信息

三台機器，操作系統CentOS 7.8或者阿里雲服務器。此處我都安裝測試過是可行的

k8s-master 192.168.1.180
k8s-node1 192.168.1.151
k8s-node2 192.168.1.152

1、搭建glusterFS集群

執行

[root@k8s-master ]# yum install centos-release-gluster
[root@k8s-master ]# yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma

配置 GlusterFS 集群

[root@k8s-master ]# systemctl start glusterd.service
[root@k8s-master ]# systemctl enable glusterd.service

主節點執行

[root@k8s-master ]# gluster peer probe k8s-master
[root@k8s-master ]#  gluster peer probe k8s-node1
[root@k8s-master ]# gluster peer probe k8s-node2

創建數據目錄

[root@k8s-master ]# mkdir -p /opt/gfs_data

創建復制卷

[root@k8s-master ]# gluster volume create k8s-volume replica 3 k8s-master:/opt/gfs_data k8s-node1:/opt/gfs_data k8s-node2:/opt/gfs_data force

啟動卷

[root@k8s-master ]# gluster volume start k8s-volume

查詢卷狀態

[root@k8s-master ~]# gluster volume status
Status of volume: k8s-volume
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick k8s-master:/opt/gfs_data              49152     0          Y       13987
Brick k8s-node1:/opt/gfs_data               49152     0          Y       13216
Brick k8s-node2:/opt/gfs_data               49152     0          Y       13932
Self-heal Daemon on localhost               N/A       N/A        Y       14014
Self-heal Daemon on k8s-node1               N/A       N/A        Y       13233
Self-heal Daemon on k8s-node2               N/A       N/A        Y       13949

Task Status of Volume k8s-volume
------------------------------------------------------------------------------
There are no active volume tasks
[root@k8s-master ~]#

[root@k8s-master ~]# gluster volume info

Volume Name: k8s-volume
Type: Replicate
Volume ID: fce4a311-6afb-428b-90dd-e0e9c335de3e
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: k8s-master:/opt/gfs_data
Brick2: k8s-node1:/opt/gfs_data
Brick3: k8s-node2:/opt/gfs_data
Options Reconfigured:
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@k8s-master ~]#

驗證glusterFS集群可用

選擇其中一台主機執行

yum install -y glusterfs glusterfs-fuse
mount -t glusterfs 192.168.1.180:/k8s-volume /mnt
cd /mnt
touch test

2、使用glusterfs(以下均在k8s master節點執行)

創建glusterfs的endpoints：kubectl apply -f glusterfs-cluster.yaml

apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
  namespace: default
subsets:
- addresses:
  - ip: 192.168.1.180
  - ip: 192.168.1.151
  - ip: 192.168.1.152
  ports:
  - port: 30003
    protocol: TCP

創建pv使用glusterfs：kubectl apply -f glusterfs-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: glusterfs-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: k8s-volume
    readOnly: false

創建pvc聲明：kubectl apply -f glusterfs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: glusterfs-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

創建應用使用pvc：kubectl apply -f nginx_deployment.yaml

[root@k8s-master ]# cat nginx_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
          volumeMounts:
            - name: storage001
              mountPath: "/usr/share/nginx/html"
      volumes:
      - name: storage001
        persistentVolumeClaim:
          claimName: glusterfs-pvc

執行下面命令會發現有test文件

kubectl exec nginx-deployment-6594f875fb-8dnl8 ls /usr/share/nginx/html/