kubernetes存儲類與PV與PVC關系及實踐

StorageClass & PV & PVC關系圖

• Volumes是最基礎的存儲抽象，其支持多種類型，包括本地存儲、NFS、FC以及眾多的雲存儲，我們也可以編寫自己的存儲插件來支持特定的存儲系統。Volume可以被Pod直接使用，也可以被PV使用。普通的Volume和Pod之間是一種靜態的綁定關系，在定義Pod的同時，通過volume屬性來定義存儲的類型，通過volumeMount來定義容器內的掛載點。

• PersistentVolume。與普通的Volume不同，PV是Kubernetes中的一個資源對象，創建一個PV相當於創建了一個存儲資源對象，這個資源的使用要通過PVC來請求。

• PersistentVolumeClaim。PVC是用戶對存儲資源PV的請求，根據PVC中指定的條件Kubernetes動態的尋找系統中的PV資源並進行綁定。目前PVC與PV匹配可以通過StorageClassName、matchLabels或者matchExpressions三種方式。

• StorageClass。存儲類，目前kubernetes支持很多存儲，例如ceph,nfs,glusterfs等等。。。

接下來，本文使用前文《手把手教你使用rpm部署ceph集群》創建好的ceph集群來為kubernetes提供存儲。

創建存儲類

1、獲取admin key

grep key /etc/ceph/ceph.client.admin.keyring |awk '{printf "%s", $NF }'|base64

QVFCZ2ZZOWJ1dGdBQ0JBQXN5dGdLZ1BFOGlsblIzWjJqNVVKMUE9PQ==

2、寫入 ceph-secret-admin.yaml

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret-admin
type: "kubernetes.io/rbd"
data:
  key: QVFCZ2ZZOWJ1dGdBQ0JBQXN5dGdLZ1BFOGlsblIzWjJqNVVKMUE9PQ==

3、創建secret

kubectl  apply -f ceph-secret-admin.yaml
kubectl  get secret
NAME                                 TYPE                                  DATA      AGE
ceph-secret-admin                    kubernetes.io/rbd                     1         6m

4、 修改 rbd-storage-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rbd
provisioner: kubernetes.io/rbd
parameters:
    monitors: 192.168.100.100:6789,192.168.100.101:6789,192.168.100.102:6789
    adminId: admin
    adminSecretName: ceph-secret-admin
    adminSecretNamespace: default
    pool: k8s
    userId: admin
    userSecretName: ceph-secret-admin
    userSecretNamespace: default
    fsType: ext4
    imageFormat: "2"
    imageFeatures: "layering"

5、創建storage class

kubectl apply -f rbd-storage-class.yaml

[root@qd01-stop-cloud001 rbd]# kubectl  get sc
NAME      PROVISIONER         AGE
rbd         kubernetes.io/rbd   4m

創建PVC & PV

6、創建pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rbd
  resources:
    requests:
      storage: 1Gi

7、查看pvc,顯示狀態為Bound表示成功

kubectl apply -f pvc.yaml
kubectl  get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
claim    Bound     pvc-130c2445-b4a5-11e8-9d27-782bcb3bb379   1Gi        RWO            slow           13m

測試驗證

8、創建pod.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: server
spec:
  replicas: 1
  selector:
    role: server
  template:
    metadata:
      labels:
        role: server
    spec:
      containers:
      - name: server
        image: nginx
        volumeMounts:
          - mountPath: /var/lib/www/html
            name: mypvc
      volumes:
        - name: mypvc
          persistentVolumeClaim:
            claimName: claim

9、查看掛載存儲

/dev/rbd0       1G  9.1M  0.98G   1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/k8s-image-kubernetes-dynamic-pvc-198f56b3-b4a5-11e8-97eb-782bcb3bb379

錯誤處理

如果出錯failed to create rbd image: executable file not found in $PATH
參考https://blog.csdn.net/aixiaoyang168/article/details/79120095
可以使用下面的項目來創建存儲類

使用external-storage創建存儲類

$ git clone https://github.com/kubernetes-incubator/external-storage.git
$ tree external-storage/ceph/rbd/deploy/
├── README.md
├── non-rbac
│   └── deployment.yaml
└── rbac
    ├── clusterrole.yaml
    ├── clusterrolebinding.yaml
    ├── deployment.yaml
    └── serviceaccount.yaml

Install without RBAC roles:
cd $GOPATH/src/github.com/kubernetes-incubator/external-storage/ceph/rbd/deploy
kubectl apply -f ./non-rbac

Install with RBAC roles:
cd $GOPATH/src/github.com/kubernetes-incubator/external-storage/ceph/rbd/deploy
NAMESPACE=default     # change this if you want to deploy it in another namespace
sed -r -i "s/namespace: [^ ]+/namespace: $NAMESPACE/g" ./rbac/clusterrolebinding.yaml ./rbac/rolebinding.yaml
kubectl -n $NAMESPACE apply -f ./rbac

如果secret和provisioner不在同一個namespace中的話，獲取secret權限不夠。
解決方法：
以下文件添加secrets的權限
external-storage/ceph/rbd/deploy/rbac/clusterrole.yaml

  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "create", "delete"]