目錄

• Kubernetes之（十四）StatefulSet控制器
  • 簡介
  • StatefulSet使用
    • 滾動更新
    • 擴展伸縮
    • 更新策略和版本升級
      • 參考資料

Kubernetes之（十四）StatefulSet控制器

簡介

StatefulSet 作為 Controller 為 Pod 提供唯⼀的標識。 它可以保證部署和 scale 的順序。
StatefulSet是為了解決有狀態服務的問題（對應Deployments和ReplicaSets是為⽆狀態服務⽽設計） ， 其應⽤場景包括：

• 穩定的持久化存儲， 即Pod重新調度后還是能訪問到相同的持久化數據， 基於PVC來實現
• 穩定的⽹絡標志， 即Pod重新調度后其PodName和HostName不變， 基於Headless Service（即沒有Cluster IP的Service） 來實現
• 有序部署， 有序擴展， 即Pod是有順序的， 在部署或者擴展的時候要依據定義的順序依次依次進⾏（即從0到N-1，在下⼀個Pod運⾏之前所有之前的Pod必須都是Running和Ready狀態） ， 基於init containers來實現有序收縮， 有序刪除（即從N-1到0）

從上⾯的應⽤場景可以發現， StatefulSet由以下⼏個部分組成：

• ⽤於定義⽹絡標志（DNS domain） 的Headless Service
• ⽤於創建PersistentVolumes的volumeClaimTemplates
• 定義具體應⽤的StatefulSet

StatefulSet中每個Pod的DNS格式為 statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local ， 其中

• serviceName 為Headless Service的名字
• 0..N-1 為Pod所在的序號， 從0開始到N-1
• statefulSetName 為StatefulSet的名字
• namespace 為服務所在的namespace， Headless Servic和StatefulSet必須在相同的namespace
• .cluster.local 為Cluster Domain

部署和Scale保證:

• 對於有N個副本的StatefulSet，Pod將按照{0..N-1}的順序被創建和部署。
• 當刪除Pod的時候，將按照逆序來終結，從{N-1..0}
• 對Pod執⾏scale操作之前，它所有的前任必須處於Running和Ready狀態。
• 在終⽌Pod前，它所有的繼任者必須處於完全關閉狀態。

Headless Service：
在deployment中，每一個pod是沒有名稱，是隨機字符串，是無序的。而statefulset中是要求有序的，每一個pod的名稱必須是固定的。當節點掛了，重建之后的標識符是不變的，每一個節點的節點名稱是不能改變的。pod名稱是作為pod識別的唯一標識符，必須保證其標識符的穩定並且唯一。
為了實現標識符的穩定，這時候就需要一個headless service 解析直達到pod，還需要給pod配置一個唯一的名稱。

volumeClainTemplate：
大部分有狀態副本集都會用到持久存儲，比如分布式系統來說，由於數據是不一樣的，每個節點都需要自己專用的存儲節點。而在deployment中pod模板中創建的存儲卷是一個共享的存儲卷，多個pod使用同一個存儲卷，而statefulset定義中的每一個pod都不能使用同一個存儲卷，由此基於pod模板創建pod是不適應的，這就需要引入volumeClainTemplate，當在使用statefulset創建pod時，會自動生成一個PVC，從而請求綁定一個PV，從而有自己專用的存儲卷。Pod名稱、PVC和PV關系圖如下：

StatefulSet使用

使用之前配置的NFS服務器，PV

#nfs服務器
[root@nfs ~]# showmount -e 
Export list for nfs:
/data/volumes/v5 10.0.0.0/24
/data/volumes/v4 10.0.0.0/24
/data/volumes/v3 10.0.0.0/24
/data/volumes/v2 10.0.0.0/24
/data/volumes/v1 10.0.0.0/24


#PV
[root@master volume]# kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv001   1Gi        RWO,RWX        Retain           Available                                   3s
pv002   2Gi        RWO            Retain           Available                                   3s
pv003   2Gi        RWO,RWX        Retain           Available                                   3s
pv004   4Gi        RWO,RWX        Retain           Available                                   3s
pv005   4Gi        RWO,RWX        Retain           Available                                   3s                         23h

創建statefulSet

[root@master statefulset]# vim statefulset-v1.yaml
#定義Headless服務
apiVersion: v1
kind: Service
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: myapp-pod
---
#配置statefulset
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: myapp
spec:
  serviceName: myapp-svc
  replicas: 2
  selector:
    matchLabels:
      app: myapp-pod
  template:
    metadata:
      labels:
        app: myapp-pod
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v1
        ports:
        - name: web
          containerPort: 80
        volumeMounts:
        - name: myappdata
          mountPath: /usr/share/nginx/html/
  volumeClaimTemplates:
  - metadata:
      name: myappdata
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1.5Gi
[root@master statefulset]# kubectl apply -f statefulset-v1.yaml 
service/myapp created
statefulset.apps/myapp created

[root@master statefulset]# kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
myapp-0   1/1     Running   0          5s
myapp-1   1/1     Running   0          4s
myapp-2   1/1     Running   0          3s
#此時Pod名稱不是亂碼是從0~N-1
[root@master statefulset]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   7d
myapp        ClusterIP   None         <none>        80/TCP    31s

[root@master statefulset]# kubectl get sts
NAME    READY   AGE
myapp   3/3     48s

查看pvc pv資源

[root@master statefulset]# kubectl get pvc
NAME                STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myappdata-myapp-0   Bound    pv002    2Gi        RWO                           3m56s
myappdata-myapp-1   Bound    pv003    2Gi        RWO,RWX                       2m49s
myappdata-myapp-2   Bound    pv004    4Gi        RWO,RWX                       88s
[root@master statefulset]# kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
pv001   1Gi        RWO,RWX        Retain           Available                                                       22m
pv002   2Gi        RWO            Retain           Bound       default/myappdata-myapp-0                           22m
pv003   2Gi        RWO,RWX        Retain           Bound       default/myappdata-myapp-1                           22m
pv004   4Gi        RWO,RWX        Retain           Bound       default/myappdata-myapp-2                           22m
pv005   4Gi        RWO,RWX        Retain           Available                                                       22m

查看Pod使用的存儲卷

[root@master statefulset]# kubectl describe pods myapp-2
......
Volumes:
  myappdata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  myappdata-myapp-2
......

就算刪除Pod后重建的Podi依然綁定在myappdata-myapp-2這個PVC上

滾動更新

[root@master statefulset]# vim statefulset-v1.yaml

        image: ikubernetes/myapp:v3

[root@master statefulset]# kubectl apply -f statefulset-v1.yaml 
service/myapp unchanged
statefulset.apps/myapp configured

[root@master ~]# kubectl get pods -w
NAME      READY   STATUS    RESTARTS   AGE
myapp-0   1/1     Running   0          9m18s
myapp-1   1/1     Running   0          9m17s
myapp-2   1/1     Running   0          9m16s
myapp-2   1/1   Terminating   0     9m22s
myapp-2   0/1   Terminating   0     9m23s
myapp-2   0/1   Terminating   0     9m26s
myapp-2   0/1   Terminating   0     9m26s
myapp-2   0/1   Pending   0     0s
myapp-2   0/1   Pending   0     0s
myapp-2   0/1   ContainerCreating   0     0s
myapp-2   1/1   Running   0     2s
myapp-1   1/1   Terminating   0     9m29s
myapp-1   0/1   Terminating   0     9m30s
myapp-1   0/1   Terminating   0     9m31s
myapp-1   0/1   Terminating   0     9m31s
myapp-1   0/1   Pending   0     0s
myapp-1   0/1   Pending   0     0s
myapp-1   0/1   ContainerCreating   0     0s
myapp-1   1/1   Running   0     1s
myapp-0   1/1   Terminating   0     9m33s
myapp-0   0/1   Terminating   0     9m34s
myapp-0   0/1   Terminating   0     9m35s
myapp-0   0/1   Terminating   0     9m35s
myapp-0   0/1   Pending   0     0s
myapp-0   0/1   Pending   0     0s
myapp-0   0/1   ContainerCreating   0     0s
myapp-0   1/1   Running   0     1s

在創建的每一個Pod中，每一個pod自己的名稱都是可以被解析的，如下：

[root@master statefulset]# kubectl get pods -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
myapp-0   1/1     Running   0          39s   10.244.2.66   node02   <none>           <none>
myapp-1   1/1     Running   0          42s   10.244.1.56   node01   <none>           <none>
myapp-2   1/1     Running   0          46s   10.244.2.65   node02   <none>           <none>

[root@master statefulset]# kubectl exec -it myapp-1 -- /bin/sh
/ # nslookup myapp-1.myapp-svc.default.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve

Name:      myapp-1.myapp-svc.default.svc.cluster.local
Address 1: 10.244.1.56 myapp-1.myapp-svc.default.svc.cluster.local

從上面的解析，我們可以看到在容器當中可以通過對Pod的名稱進行解析到ip。其解析的域名格式如下：
pod_name.service_name.ns_name.svc.cluster.local
eg: myapp-0.myapp.default.svc.cluster.local

擴展伸縮

#擴展副本到4個
[root@master statefulset]# kubectl scale sts myapp --replicas=4
statefulset.apps/myapp scaled
#查看結果
[root@master ~]# kubectl get pods -w
NAME      READY   STATUS    RESTARTS   AGE
myapp-0   1/1     Running   0          6m17s
myapp-1   1/1     Running   0          6m20s
myapp-2   1/1     Running   0          6m24s
myapp-3   0/1   Pending   0     0s
myapp-3   0/1   Pending   0     0s
myapp-3   0/1   Pending   0     0s
myapp-3   0/1   ContainerCreating   0     0s
myapp-3   1/1   Running   0     2s

#查看PV綁定
[root@master statefulset]# kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
pv001   1Gi        RWO,RWX        Retain           Available                                                       38m
pv002   2Gi        RWO            Retain           Bound       default/myappdata-myapp-0                           38m
pv003   2Gi        RWO,RWX        Retain           Bound       default/myappdata-myapp-1                           38m
pv004   4Gi        RWO,RWX        Retain           Bound       default/myappdata-myapp-2                           38m
pv005   4Gi        RWO,RWX        Retain           Bound       default/myappdata-myapp-3   

 #打補丁方式縮容
 [root@master statefulset]# kubectl patch sts myapp -p '{"spec":{"replicas":2}}'
statefulset.apps/myapp patched

[root@master ~]# kubectl get pods -w
NAME      READY   STATUS    RESTARTS   AGE
myapp-0   1/1     Running   0          9m26s
myapp-1   1/1     Running   0          9m29s
myapp-2   1/1     Running   0          13s
myapp-3   1/1     Running   0          11s
myapp-3   1/1   Terminating   0     16s
myapp-3   0/1   Terminating   0     18s
myapp-3   0/1   Terminating   0     22s
myapp-3   0/1   Terminating   0     22s
myapp-2   1/1   Terminating   0     24s
myapp-2   0/1   Terminating   0     24s
myapp-2   0/1   Terminating   0     30s
myapp-2   0/1   Terminating   0     30s

更新策略和版本升級

修改更新策略，以partition方式進行更新，更新值為2，只有myapp編號大於等於2的才會進行更新。類似於金絲雀部署方式

[root@master statefulset]# kubectl patch sts myapp -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
statefulset.apps/myapp patched

[root@master statefulset]# kubectl describe sts myapp
Name:               myapp
Namespace:          default
CreationTimestamp:  Wed, 03 Apr 2019 16:53:24 +0800
Selector:           app=myapp-pod
Labels:             <none>
Annotations:        kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"myapp","namespace":"default"},"spec":{"replicas":3,"sele...

版本升級，將image的版本升級為v5，升級后對比myapp-2和myapp-1的image版本是不同的。這樣就實現了金絲雀發布的效果。

[root@master statefulset]# kubectl get sts -o wide
NAME    READY   AGE   CONTAINERS   IMAGES
myapp   4/4     23m   myapp        ikubernetes/myapp:v5

[root@master statefulset]# kubectl get pods myapp-2 -o yaml |grep image
  - image: ikubernetes/myapp:v5
    imagePullPolicy: IfNotPresent
    image: ikubernetes/myapp:v5
    imageID: docker-pullable://ikubernetes/myapp@sha256:85d1005d172aa8b97d7f1aa67519132cd450f59d01a607d4b4eaf5bcf402ce52
	
[root@master statefulset]# kubectl get pods myapp-0 -o yaml |grep image 
  - image: ikubernetes/myapp:v3
    imagePullPolicy: IfNotPresent
    image: ikubernetes/myapp:v3
    imageID: docker-pullable://ikubernetes/myapp@sha256:b8d74db2515d3c1391c78c5768272b9344428035ef6d72158fd9f6c4239b2c69

將剩余的Pod也更新版本，只需要將更新策略的partition值改為0即可，如下：

[root@master statefulset]# kubectl patch sts myapp -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
statefulset.apps/myapp patched

[root@master statefulset]# kubectl get pods myapp-0 -o yaml |grep image                                             - image: ikubernetes/myapp:v5
    imagePullPolicy: IfNotPresent
    image: ikubernetes/myapp:v5
    imageID: docker-pullable://ikubernetes/myapp@sha256:85d1005d172aa8b97d7f1aa67519132cd450f59d01a607d4b4eaf5bcf402ce52

生產中還是不建議把重要應用使用statefulset，如mysql redis 等 。

參考資料

https://www.cnblogs.com/linuxk
馬永亮. Kubernetes進階實戰 (雲計算與虛擬化技術叢書)
Kubernetes-handbook-jimmysong-20181218