045.Kubernetes集群存儲-CSI存儲機制

一 CSI存儲機制

1.1 CSI簡介

Kubernetes從1.9版本開始引入容器存儲接口Container Storage Interface（CSI）機制，用於在Kubernetes和外部存儲系統之間建立一套標准的存儲管理接口，通過該接口為容器提供存儲服務。

1.2 CSI的設計背景

Kubernetes通過PV、PVC、Storageclass已經提供了一種強大的基於插件的存儲管理機制，但是各種存儲插件提供的存儲服務都是基於一種被稱為“in-true”（樹內）的方式提供的，這要求存儲插件的代碼必須被放進Kubernetes的主干代碼庫中才能被Kubernetes調用，屬於緊耦合的開發模式。這種“in-tree”方式會帶來一些問題：

• 存儲插件的代碼需要與Kubernetes的代碼放在同一代碼庫中，並與Kubernetes的二進制文件共同發布；
• 存儲插件代碼的開發者必須遵循Kubernetes的代碼開發規范；
• 存儲插件代碼的開發者必須遵循Kubernetes的發布流程，包括添加對Kubernetes存儲系統的支持和錯誤修復；
• Kubernetes社區需要對存儲插件的代碼進行維護，包括審核、測試等工作；
• 存儲插件代碼中的問題可能會影響Kubernetes組件的運行，並且很難排查問題；
• 存儲插件代碼與Kubernetes的核心組件（kubelet和kubecontroller-manager）享有相同的系統特權權限，可能存在可靠性和安全性問題。

Kubernetes已有的FlexVolume插件機制試圖通過為外部存儲暴露一個基於可執行程序（exec）的API來解決這些問題。盡管它允許第三方存儲提供商在Kubernetes核心代碼之外開發存儲驅動，但仍然有兩個問題沒有得到很好的解決：

• 部署第三方驅動的可執行文件仍然需要宿主機的root權限，存在安全隱患；
• 存儲插件在執行mount、attach這些操作時，通常需要在宿主機上安裝一些第三方工具包和依賴庫，使得部署過程更加復雜，例如部署Ceph時需要安裝rbd庫，部署GlusterFS時需要安裝mount.glusterfs庫，等等。

基於以上這些問題和考慮，Kubernetes逐步推出與容器對接的存儲接口標准，存儲提供方只需要基於標准接口進行存儲插件的實現，就能使用Kubernetes的原生存儲機制為容器提供存儲服務。這套標准被稱為CSI（容器存儲接口）。

在CSI成為Kubernetes的存儲供應標准之后，存儲提供方的代碼就能和Kubernetes代碼徹底解耦，部署也與Kubernetes核心組件分離，顯然，存儲插件的開發由提供方自行維護，就能為Kubernetes用戶提供更多的存儲功能，也更加安全可靠。

基於CSI的存儲插件機制也被稱為“out-of-tree”（樹外）的服務提供方式，是未來Kubernetes第三方存儲插件的標准方案。

二 CSI架構

2.1 CSI存儲組件/部署架構

KubernetesCSI存儲插件的關鍵組件和推薦的容器化部署架構：

其中主要包括兩種組件：CSI Controller和CSI Node。

2.2 CSI Controller

CSI Controller的主要功能是提供存儲服務視角對存儲資源和存儲卷進行管理和操作。在Kubernetes中建議將其部署為單實例Pod，可以使用StatefulSet或Deployment控制器進行部署，設置副本數量為1，保證為一種存儲插件只運行一個控制器實例。

在這個Pod內部署兩個容器：

• 與Master（kube-controller-manager）通信的輔助sidecar容器。在sidecar容器內又可以包含external-attacher和external-provisioner兩個容器，它們的功能分別如下。

• external-attacher：監控VolumeAttachment資源對象的變更，觸發針對CSI端點的ControllerPublish和ControllerUnpublish操作。
• external-provisioner：監控PersistentVolumeClaim資源對象的變更，觸發針對CSI端點的CreateVolume和DeleteVolume操作。
• CSI Driver存儲驅動容器，由第三方存儲提供商提供，需要實現上述接口。

這兩個容器通過本地Socket（Unix DomainSocket，UDS），並使用gPRC協議進行通信。

sidecar容器通過Socket調用CSI Driver容器的CSI接口，CSI Driver容器負責具體的存儲卷操作。

2.3 CSI Node

CSI Node的主要功能是對主機（Node）上的Volume進行管理和操作。在Kubernetes中建議將其部署為DaemonSet，在每個Node上都運行一個Pod。

在這個Pod中部署以下兩個容器：

• 與kubelet通信的輔助sidecar容器node-driver-registrar，主要功能是將存儲驅動注冊到kubelet中；
• CSI Driver存儲驅動容器，由第三方存儲提供商提供，主要功能是接收kubelet的調用，需要實現一系列與Node相關的CSI接口，例如NodePublishVolume接口（用於將Volume掛載到容器內的目標路徑）、NodeUnpublishVolume接口（用於從容器中卸載Volume），等等。

node-driver-registrar容器與kubelet通過Node主機的一個hostPath目錄下的unixsocket進行通信。CSI Driver容器與kubelet通過Node主機的另一個hostPath目錄下的unixsocket進行通信，同時需要將kubelet的工作目錄（默認為/var/lib/kubelet）掛載給CSIDriver容器，用於為Pod進行Volume的管理操作（包括mount、umount等）。

三 CSI插件使用實踐

3.1 實驗說明

以csi-hostpath插件為例，演示部署CSI插件、用戶使用CSI插件提供的存儲資源。

3.2 開啟特性

設置Kubernetes服務啟動參數，為kube-apiserver、kubecontroller-manager和kubelet服務的啟動參數添加。

[root@k8smaster01 ~]# vi /etc/kubernetes/manifests/kube-apiserver.yaml

……
    - --allow-privileged=true
    - --feature-gates=CSIPersistentVolume=true
    - --runtime-config=storage.k8s.io/v1alpha1=true
……

[root@k8smaster01 ~]# vi /etc/kubernetes/manifests/kube-controller-manager.yaml

……
    - --feature-gates=CSIPersistentVolume=true
……

[root@k8smaster01 ~]# vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --feature-gates=CSIPersistentVolume=true"
……

[root@k8smaster01 ~]# systemctl daemon-reload

[root@k8smaster01 ~]# systemctl restart kubelet.service

3.3 創建CRD資源對象

創建CSINodeInfo和CSIDriverRegistry CRD資源對象：

[root@k8smaster01 ~]# vi csidriver.yaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: csidrivers.csi.storage.k8s.io
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  group: csi.storage.k8s.io
  names:
    kind: CSIDriver
    plural: csidrivers
  scope: Cluster
  validation:
    openAPIV3Schema:
      properties:
        spec:
          description: Specification of the CSI Driver.
          properties:
            attachRequired:
              description: Indicates this CSI volume driver requires an attach operation,and that Kubernetes should call attach and wait for any attach operationto complete before proceeding to mount.
              type: boolean
            podInfoOnMountVersion:
              description: Indicates this CSI volume driver requires additional pod
                information (like podName, podUID, etc.) during mount operations.
              type: string
  version: v1alpha1

[root@k8smaster01 ~]# vi csinodeinfo.yaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: csinodeinfos.csi.storage.k8s.io
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  group: csi.storage.k8s.io
  names:
    kind: CSINodeInfo
    plural: csinodeinfos
  scope: Cluster
  validation:
    openAPIV3Schema:
      properties:
        spec:
          description: Specification of CSINodeInfo
          properties:
            drivers:
              description: List of CSI drivers running on the node and their specs.
              type: array
              items:
                properties:
                  name:
                    description: The CSI driver that this object refers to.
                    type: string
                  nodeID:
                    description: The node from the driver point of view.
                    type: string
                  topologyKeys:
                    description: List of keys supported by the driver.
                    items:
                      type: string
                    type: array
        status:
          description: Status of CSINodeInfo
          properties:
            drivers:
              description: List of CSI drivers running on the node and their statuses.
              type: array
              items:
                properties:
                  name:
                    description: The CSI driver that this object refers to.
                    type: string
                  available:
                    description: Whether the CSI driver is installed.
                    type: boolean
                  volumePluginMechanism:
                    description: Indicates to external components the required mechanism
                      to use for any in-tree plugins replaced by this driver.
                    pattern: in-tree|csi
                    type: string
  version: v1alpha1

[root@k8smaster01 ~]# kubectl apply -f csidriver.yaml

[root@k8smaster01 ~]# kubectl apply -f csinodeinfo.yaml

3.4 創建相應RBAC

[root@k8smaster01 ~]# git clone https://github.com/kubernetes-csi/drivers

[root@k8smaster01 ~]# cd drivers/deploy/hostpath/

[root@k8smaster01 hostpath]# vi csi-hostpath-attacher-rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-attacher
  # replace with non-default namespace name
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-attacher-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csinodeinfos"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-attacher-role
subjects:
  - kind: ServiceAccount
    name: csi-attacher
    # replace with non-default namespace name
    namespace: default
roleRef:
  kind: ClusterRole
  name: external-attacher-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  # replace with non-default namespace name
  namespace: default
  name: external-attacher-cfg
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-attacher-role-cfg
  # replace with non-default namespace name
  namespace: default
subjects:
  - kind: ServiceAccount
    name: csi-attacher
    # replace with non-default namespace name
    namespace: default
roleRef:
  kind: Role
  name: external-attacher-cfg
  apiGroup: rbac.authorization.k8s.io

[root@k8smaster01 hostpath]# vi csi-hostpath-provisioner-rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-provisioner
  # replace with non-default namespace name
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["get", "list"]
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csinodeinfos"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-provisioner-role
subjects:
  - kind: ServiceAccount
    name: csi-provisioner
    # replace with non-default namespace name
    namespace: default
roleRef:
  kind: ClusterRole
  name: external-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  # replace with non-default namespace name
  namespace: default
  name: external-provisioner-cfg
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-provisioner-role-cfg
  # replace with non-default namespace name
  namespace: default
subjects:
  - kind: ServiceAccount
    name: csi-provisioner
    # replace with non-default namespace name
    namespace: default
roleRef:
  kind: Role
  name: external-provisioner-cfg
  apiGroup: rbac.authorization.k8s.io

[root@k8smaster01 hostpath]# vi csi-hostpathplugin-rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-driver-registrar
  # replace with non-default namespace name
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: driver-registrar-runner
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  # The following permissions are only needed when running
  # driver-registrar without the --kubelet-registration-path
  # parameter, i.e. when using driver-registrar instead of
  # kubelet to update the csi.volume.kubernetes.io/nodeid
  # annotation. That mode of operation is going to be deprecated
  # and should not be used anymore, but is needed on older
  # Kubernetes versions.
  # - apiGroups: [""]
  #   resources: ["nodes"]
  #   verbs: ["get", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-driver-registrar-role
subjects:
  - kind: ServiceAccount
    name: csi-driver-registrar
    # replace with non-default namespace name
    namespace: default
roleRef:
  kind: ClusterRole
  name: driver-registrar-runner
  apiGroup: rbac.authorization.k8s.io

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-attacher-rbac.yaml

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-provisioner-rbac.yaml

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpathplugin-rbac.yaml

3.5 正式部署

[root@k8smaster01 ~]# cd drivers/deploy/hostpath/

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-attacher.yaml

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-provisioner.yaml

[root@k8smaster01 hostpath]# kubectl create -f csi-hostpathplugin.yaml

提示：如上相應yaml建議修改鏡像源為國內：

gcr.io ----> gcr.azk8s.cn （國內）

quay.io ----> quay.azk8s.cn （國內）

四 測試使用

4.1 確認驗證

[root@k8smaster01 ~]# kubectl get pods

4.2 創建StorageClass

[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-hostpath-sc
provisioner: csi-hostpath
reclaimPolicy: Delete
volumeBindingMode: Immediate

[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-storageclass.yaml

4.3 創建PVC

[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-hostpath-sc

[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-pvc.yaml

[root@k8smaster01 ~]# kubectl get pvc

[root@k8smaster01 ~]# kubectl get pv

4.4 創建應用

[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-app.yaml

kind: Pod
apiVersion: v1
metadata:
  name: my-csi-app
spec:
  containers:
    - name: my-frontend
      image: busybox
      volumeMounts:
      - mountPath: "/data"
        name: my-csi-volume
      command: [ "sleep", "1000000" ]
  volumes:
    - name: my-csi-volume
      persistentVolumeClaim:
        claimName: csi-pvc

[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-app.yaml

[root@k8smaster01 ~]# kubectl get pods

提示：更多CSI插件示例參考：https://feisky.gitbooks.io/kubernetes/plugins/csi.html。

CSI官方文檔：https://kubernetes-csi.github.io/docs/