StatefulSet 使用NFS配置動態持久化存儲 - 運維筆記

 

 StatefulSet是為了解決有狀態服務的容器問題而設計的，對應的Deployment和ReplicaSet是為了無狀態服務而設計的。

StatefulSet應用場景包括：

• 穩定的持久化存儲，即Pod重新調度后還是能訪問到相同的持久化數據，基於PVC來實現
• 穩定的網絡標志，即Pod重新調度后其PodName和HostName不變，基於Headless Service（即沒有Cluster IP的Service）來實現
• 有序部署，有序擴展，即Pod是有順序的。在部署或者擴展的時候要依據定義的順序依次依次進行（即從0到N-1，在下一個Pod運行之前所有之前的Pod必須都是Running和Ready狀態），基於init containers來實現
• 有序收縮，有序刪除（即從N-1到0）
• StatefulSet要求Pod的名稱是有順序的，每一個Pod都不能被隨意取代，即使Pod重建之后，名稱依然不變。

 

StatefulSet由以下幾部分組成：

• 用於定義網絡標志的Headless Service（headless-svc：無頭服務。因為沒有IP地址，所以它不具備負載均衡的功能了）
• 用於創建PersistentVolumes的volumeClaimTemplates。
• 定義具體應用的StatefulSet。

 

StatefulSet配置NFS動態持久化存儲的流程：

• 創建 NFS 服務器。
• 創建 Service Account。用來管控 NFS provisioner 在k8s集群中運行的權限。
• 創建 StorageClass。負責創建 PVC 並調用 NFS provisioner 進行預定的工作，並關聯 PV 和 PVC。
• 創建 NFS provisioner。有兩個功能,一個是在NFS共享目錄下創建掛載點(volume)；二是建立 PV 並將 PV 與 NFS 掛載點建立關聯。

 

StatefulSet有狀態應用pod訪問地址
pod_name.service_name.namespace_name.svc.cluster.local:pod_app_port

 

一、NFS部署
可以參考：NFS雙機熱備高可用環境
NFS服務器為：172.16.60.194
NFS掛載目錄為：/data/storage/mobile-decision-server_data

NFS的配置：

[root@k8s-storage01 ~]# cat /etc/exports
/data/storage 172.16.60.197(rw,sync,no_root_squash)
/data/storage 172.16.60.193(rw,sync,no_root_squash)
/data/storage 172.16.60.48(rw,sync,no_root_squash)
/data/storage 172.16.60.120(rw,sync,no_root_squash)

　　

二、StatefulSet使用NFS配置動態持久化存儲
下面是使用NFS作為StatefulSet持久化存儲的操作記錄，分別需要創建nfs-provisioner的rbac、storageclass、nfs-client-provisioner和statefulset的pod。

1）創建nfs-rbac.yaml

[root@k8s-master01 mobile-decision-server]# cat nfs-rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner
  namespace: kevin
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
   name: nfs-provisioner-runner
   namespace: kevin
rules:
   -  apiGroups: [""]
      resources: ["persistentvolumes"]
      verbs: ["get", "list", "watch", "create", "delete"]
   -  apiGroups: [""]
      resources: ["persistentvolumeclaims"]
      verbs: ["get", "list", "watch", "update"]
   -  apiGroups: ["storage.k8s.io"]
      resources: ["storageclasses"]
      verbs: ["get", "list", "watch"]
   -  apiGroups: [""]
      resources: ["events"]
      verbs: ["watch", "create", "update", "patch"]
   -  apiGroups: [""]
      resources: ["services", "endpoints"]
      verbs: ["get","create","list", "watch","update"]
   -  apiGroups: ["extensions"]
      resources: ["podsecuritypolicies"]
      resourceNames: ["nfs-provisioner"]
      verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: kevin
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

創建並查看

[root@k8s-master01 mobile-decision-server]# kubectl apply -f nfs-rbac.yaml
 
[root@k8s-master01 mobile-decision-server]# kubectl get sa -n kevin|grep nfs
nfs-provisioner                1         7h7m
[root@k8s-master01 mobile-decision-server]# kubectl get clusterrole -n kevin|grep nfs
nfs-provisioner-runner                                                 7h7m
[root@k8s-master01 mobile-decision-server]# kubectl get clusterrolebinding -n kevin|grep nfs
run-nfs-provisioner                                    7h7m

 

2）創建nfs-class.yaml

[root@k8s-master01 mobile-decision-server]# cat nfs-class.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: managed-nfs-storage
  namespace: kevin
provisioner: mobile-decision-server/nfs
reclaimPolicy: Retain

創建

[root@k8s-master01 mobile-decision-server]# kubectl apply -f nfs-class.yaml

　　

3）創建mobile-decision-server-nfs.yml
PROVISIONER_NAME的值一定要和StorageClass中的provisioner相等！
/persistentvolumes的mountPath 掛載點不能修改！

 

[root@k8s-master01 mobile-decision-server]# cat mobile-decision-server-nfs.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  namespace: kevin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-client-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccount: nfs-provisioner
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath:  /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: mobile-decision-server/nfs
            - name: NFS_SERVER
              value: 172.16.60.194
            - name: NFS_PATH
              value: /data/storage/mobile-decision-server_data
      volumes:
        - name: nfs-client-root
          nfs:
            server: 172.16.60.194
            path: /data/storage/mobile-decision-server_data

創建並查看

[root@k8s-master01 mobile-decision-server]# kubectl apply -f mobile-decision-server-nfs.yml
 
[root@k8s-master01 mobile-decision-server]# kubectl get pods -n kevin|grep nfs
nfs-client-provisioner-6d78cd7874-krpvj   1/1     Running   0          5h3m

　　

4）創建statefulset

[root@k8s-master01 mobile-decision-server]# cat mobile-decision-server.yml
apiVersion: v1
kind: Service
metadata:
  name: mobile-decision-server
  namespace: kevin
  labels:
    app: mobile-decision-server
spec:
  clusterIP: None
  selector:
    app: mobile-decision-server
  ports:
    - port: 8080
      name: server
      targetPort: 8080
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mobile-decision-server
  namespace: kevin
spec:
  serviceName: mobile-decision-server
  replicas: 2
  selector:
    matchLabels:
      app: mobile-decision-server
  template:
    metadata:
      labels:
        app: mobile-decision-server
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nodetype
                operator: In
                values:
                - mobile-decision-server
      serviceAccount: nfs-provisioner
      containers:
      - name: mobile-decision-server
        image: 172.16.60.196/finhub/mobile-decision-server__20
        imagePullPolicy: Always
        ports:
        - name: dserverport
          containerPort: 8080
        resources:
          requests:
            cpu: 200m
            memory: 600Mi
          limits:
            cpu: 200m
            memory: 1024Mi
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh","-c","touch /tmp/health"]
          preStop:
            exec:
              command: ["/bin/sh","-c","kill 1"]
        livenessProbe:
          exec:
            command: ["test","-e","/tmp/health"]
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: dserverport
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20
        volumeMounts:
        - name: data
          mountPath: /onestop/app
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: managed-nfs-storage
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 20Gi

　　

創建並查看

[root@k8s-master01 mobile-decision-server]# kubectl apply -f mobile-decision-server.yml
 
[root@k8s-master01 mobile-decision-server]# kubectl get pods -n kevin|grep mobile-decision-server
mobile-decision-server-0                  1/1     Running   0          5h4m
mobile-decision-server-1                  1/1     Running   0          5h4m

　　

查看pv和pvc

[root@k8s-master01 mobile-decision-server]# kubectl get pvc -n kevin
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
data-mobile-decision-server-0   Bound    pvc-9433d725-bc0a-4fae-a2dc-5c22b2c3928a   20Gi       RWX            managed-nfs-storage   5h10m
data-mobile-decision-server-1   Bound    pvc-3615cf99-31ea-491b-8808-8800057d0a21   20Gi       RWX            managed-nfs-storage   5h9m
 
[root@k8s-master01 mobile-decision-server]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                  STORAGECLASS          REASON   AGE
pvc-3615cf99-31ea-491b-8808-8800057d0a21   20Gi       RWX            Delete           Bound    kevin/data-mobile-decision-server-1   managed-nfs-storage            5h9m
pvc-9433d725-bc0a-4fae-a2dc-5c22b2c3928a   20Gi       RWX            Delete

　　

5）查看NFS共享目錄

只要pvc不刪除，NFS共享目錄下的文件夾名稱不會改變。

[root@k8s-storage01 ~]# cd /data/storage/mobile-decision-server_data/
[root@k8s-storage01 mobile-decision-server_data]# ll
total 8
drwxrwxrwx 4 root root 4096 Oct 10 11:53 kevin-data-mobile-decision-server-0-pvc-9433d725-bc0a-4fae-a2dc-5c22b2c3928a
drwxrwxrwx 4 root root 4096 Oct 10 11:53 kevin-data-mobile-decision-server-1-pvc-3615cf99-31ea-491b-8808-8800057d0a21

　

可以手動刪除pvc和pv

# kubectl delete pvc pvc-name -n namespace
# kubectl delete pv pv-name