k8s中部署redis集群(三主三從)

1 環境說明

主機名	系統版本	IP地址	cpu/內存/磁盤	用途	軟件版本
k8s_nfs	CentOS7.5	172.16.1.60	2核/2GB/60GB	nfs存儲	nfs-utils-1.3.0-0.68
k8s-master1	CentOS7.5	172.16.1.81	2核/2GB/60GB	kubernetes master1節點	k8s v1.20.0
k8s-master2	CentOS7.5	172.16.1.82	2核/2GB/60GB	kubernetes master2節點	k8s v1.20.0
k8s-node1	CentOS7.5	172.16.1.83	4核/8GB/60GB	kubernetes node1節點	k8s v1.20.0
k8s-node2	CentOS7.5	172.16.1.84	4核/8GB/60GB	kubernetes node2節點	k8s v1.20.0

補充: kubernetes集群的控制節點我打了污點不能被pod調度使用。

1 nfs服務部署
節點: k8s_nfs
用途: k8s pod 數據持久化存儲
說明: nfs服務的搭建過程不再贅述
驗證:
[root@k8s_nfs ~]# showmount -e 172.16.1.60
Export list for 172.16.1.60:
/ifs/kubernetes *
​
2 nfs-subdir-external-provisioner插件部署
節點: kubernetes集群
用途: 為中間件pod提供pvc自動供給
說明: nfs pvc自動供給插件的部署過程不再贅述。修改"deployment.yaml"文件中連接nfs服務的地址和nfs共享目錄參數；修改"class.yaml"
文件中"archiveOnDelete"(刪除時是否存檔)參數為 archiveOnDelete: "true"，刪除pod時保留pod數據，默認為false時為不保留數據。
注意: 在部署前需要在k8s各個節點上部署nfs的客戶端(yum install nfs-utils -y)，否則無法部署成功。
​
補充:
(1) gitlab項目地址: https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
(2) 下載 deploy 目錄如下文件
class.yaml、deployment.yaml、rbac.yaml
​
查看:
[root@k8s-master1 nfs-subdir-external-provisioner-master]# ls | xargs -i kubectl apply -f {}
[root@k8s-master1 nfs-subdir-external-provisioner-master]# kubectl get deployment,pod,svc,sc -n default

2 redis集群部署(三主三從)

2.1 yaml配置文件

 
           
 
            
1 配置文件說明
# 部署所在命名空間: redis-cluster

[root@k8s-master1 ~]# cd /root/redis-cluster/
[root@k8s-master1 redis-cluster]# ls -l
total 8
# redis 配置文件使用 configmap 方式進行掛載
# fix-ip.sh 腳本的作用用於當 redis 集群某 pod 重建后 Pod IP 發生變化，在 /data/nodes.conf 中將新的 Pod IP 替換原 Pod IP。不
# 然集群會出問題。
# redis.conf 文件
-rw-r--r-- 1 root root 2389 Feb 14 20:25 redis-cluster-configmap.yml
# StatefulSet、Service 配置文件
-rw-r--r-- 1 root root 1527 Feb 14 20:25 redis-cluster.yml

2 redis-cluster-configmap.yml
[root@k8s-master1 redis-cluster]# cat redis-cluster-configmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
  namespace: redis-cluster
data:
  fix-ip.sh: |
    #!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${POD_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}
    fi
    exec "$@"
  redis.conf: |
    bind 0.0.0.0
    protected-mode yes
    port 6379
    tcp-backlog 2048
    timeout 0
    tcp-keepalive 300
    daemonize no
    supervised no
    pidfile /var/run/redis.pid
    loglevel notice
    logfile /data/redis.log
    databases 16
    always-show-logo yes
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /data
    masterauth liuchang@2022
    replica-serve-stale-data yes
    replica-read-only no
    repl-diskless-sync no
    repl-diskless-sync-delay 5
    repl-disable-tcp-nodelay no
    replica-priority 100
    requirepass liuchang@2022
    maxclients 32768
    #maxmemory 6g
    maxmemory-policy allkeys-lru
    lazyfree-lazy-eviction no
    lazyfree-lazy-expire no
    lazyfree-lazy-server-del no
    replica-lazy-flush no
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    aof-load-truncated yes
    aof-use-rdb-preamble yes
    lua-time-limit 5000
    cluster-enabled yes
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 15000
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    latency-monitor-threshold 0
    notify-keyspace-events ""
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-size -2
    list-compress-depth 0
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    hll-sparse-max-bytes 3000
    stream-node-max-bytes 4096
    stream-node-max-entries 100
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit replica 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    dynamic-hz yes
    aof-rewrite-incremental-fsync yes
    rdb-save-incremental-fsync yes

3 redis-cluster.yml
[root@k8s-master1 redis-cluster]# cat redis-cluster.yml
---
apiVersion: v1
kind: Service
metadata:
  namespace: redis-cluster
  name: redis-cluster
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-cluster
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: redis-cluster
  name: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      containers:
      - name: redis
        image: redis:5.0.13
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/etc/redis/fix-ip.sh", "redis-server", "/etc/redis/redis.conf"]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /etc/redis/
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-cluster
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
      #annotations:
      #  volume.beta.kubernetes.io/storage-class: "redis-nfs-storage"
    spec:
      storageClassName: "managed-nfs-storage"
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
 
           

2.2 部署

1 創建命名空間
[root@k8s-master1 redis-cluster]# kubectl create namespace redis-cluster
​
2 應用yml文件
[root@k8s-master1 redis-cluster]# ls | xargs -i kubectl apply -f {}
configmap/redis-cluster created
service/redis-cluster created
statefulset.apps/redis-cluster created

3 查看pod
[root@k8s-master1 redis-cluster]# kubectl get pod -n redis-cluster

圖示:

4 查看pvc
[root@k8s-master1 redis-cluster]# kubectl get pvc -n redis-cluster

圖示:

5 查看pv
[root@k8s-master1 redis-cluster]# kubectl get pv

圖示:

6 查看svc
[root@k8s-master1 redis-cluster]# kubectl get svc,ep -n redis-cluster

圖示:

7 查看 nfs 共享
[root@k8s_nfs ~]# ls -l /ifs/kubernetes/
​
[root@k8s_nfs ~]# ls /ifs/kubernetes/redis-cluster-data-redis-cluster-0-pvc-ab60036d-dde2-4c5c-9b1d-5bf03816be24/
appendonly.aof nodes.conf redis.log
[root@k8s_nfs ~]# ls /ifs/kubernetes/redis-cluster-data-redis-cluster-1-pvc-b4b7b35b-de9a-4c57-ad77-73f3c50e891d/
appendonly.aof nodes.conf redis.log
注: 未列出的持久化存儲文件相同

圖示:

8 初始化 Redis Cluster 集群
(1) 注意: 必須使用 ip 進行初始化 redis 集群，使用域名會報如下錯誤
Node redis-cluster-1.redis-cluster.redis-cluster.svc.cluster.local:6379 replied with error:
ERR Invalid node address specified: redis-cluster-0.redis-cluster.redis-cluster.svc.cluster.local:6379
​
(2) 獲取 Redis 集群 6 個節點 Pod 的 ip 地址
1) 方式一
[root@k8s-master1 redis-cluster]# kubectl get pod -n redis-cluster -o wide
​
2) 方式二
# kubectl run -i --tty --image busybox:1.28.4 dns-test --restart=Never --rm /bin/sh
/ # nslookup redis-cluster.redis-cluster.svc.cluster.local # 應用連接 redis 集群時使用下面 pod 的域名

圖示:

(3) 創建集群
[root@k8s-master1 redis-cluster]# kubectl exec -it pod/redis-cluster-0 -n redis-cluster -- bash
root@redis-cluster-0:/data# 
redis-cli -a liuchang@2022 --cluster create \
172.27.36.87:6379 \
172.27.36.88:6379 \
172.27.36.89:6379 \
172.27.169.155:6379 \
172.27.169.156:6379 \
172.27.169.157:6379 \
--cluster-replicas 1

圖示:

9 驗證 Redis Cluster 集群
[root@k8s-master1 redis-cluster]# kubectl exec -it pod/redis-cluster-0 -n redis-cluster -- bash
# redis-cli -h redis-cluster-1.redis-cluster.redis-cluster.svc.cluster.local -c -a 'liuchang@2022'
redis-cluster-1.redis-cluster.redis-cluster.svc.cluster.local:6379> cluster info
redis-cluster-1.redis-cluster.redis-cluster.svc.cluster.local:6379> cluster nodes
​
注: 由下圖可以看到 redis 集群狀態正常

圖示:

2.3 測試

1 查看 redis 集群 pod
[root@k8s-master1 redis-cluster]# kubectl get pod -n redis-cluster -o wide
NAME             READY   STATUS   RESTARTS   AGE   IP               NODE     
redis-cluster-0   1/1     Running   0         3h1m   172.27.36.87     k8s-node1
redis-cluster-1   1/1     Running   0         3h1m   172.27.169.155   k8s-node2
redis-cluster-2   1/1     Running   0         3h     172.27.36.88     k8s-node1
redis-cluster-3   1/1     Running   0         3h     172.27.169.156   k8s-node2
redis-cluster-4   1/1     Running   0         3h     172.27.36.89     k8s-node1
redis-cluster-5   1/1     Running   0         3h     172.27.169.157   k8s-node2
​
2 刪除任意一個 pod(刪除名稱為 redis-cluster-3 的 pod)
[root@k8s-master1 redis-cluster]# kubectl delete pod/redis-cluster-3 -n redis-cluster
pod "redis-cluster-3" deleted
​
3 pod 被重新拉起(還占用原來的pvc 和 pv)
[root@k8s-master1 redis-cluster]# kubectl get pod -n redis-cluster -o wide
NAME             READY   STATUS   RESTARTS   AGE   IP               NODE     
redis-cluster-0   1/1     Running   0         3h5m   172.27.36.87     k8s-node1
redis-cluster-1   1/1     Running   0         3h4m   172.27.169.155   k8s-node2
redis-cluster-2   1/1     Running   0         3h4m   172.27.36.88     k8s-node1
redis-cluster-3   1/1     Running   0         3s     172.27.169.158   k8s-node2
redis-cluster-4   1/1     Running   0         3h4m   172.27.36.89     k8s-node1
redis-cluster-5   1/1     Running   0         3h4m   172.27.169.157   k8s-node2
​
注:
可以看到名稱為 redis-cluster-3 的 pod 啟動時長 AGE 為 3s，IP 由原來的 172.27.169.156 變為 172.27.169.158，
/data/nodes.conf 文件中 "myself" 對應的 ip 被 fix-ip.sh 腳本修改，redis 集群修復時會將該 ip 同步到其它 pod
節的 /data/nodes.conf 文件中，從而保證整個 redis 集群的可用性。
​
4 驗證集群狀態
[root@k8s-master1 redis-cluster]# kubectl exec -it pod/redis-cluster-4 -n redis-cluster -- bash
# redis-cli -h redis-cluster-5.redis-cluster.redis-cluster.svc.cluster.local -c -a 'liuchang@2022'
redis-cluster-5.redis-cluster.redis-cluster.svc.cluster.local:6379> cluster info
redis-cluster-5.redis-cluster.redis-cluster.svc.cluster.local:6379> cluster nodes
​
注: 由下圖可以看到集群狀態又恢復了正常

圖示:

2.4 補充

1 如果整個 redis 集群的 pod 全部都掛掉了，pod自動拉起后，集群不可用，需要重建集群。
​
2 重建集群的方法1: 刪除 redis 集群所有的資源，然后重新創建 redis 集群
(1) 刪除 redis 集群中所有的 pod
[root@k8s-master1 redis-cluster]# kubectl delete -f redis-cluster.yml
​
(2) 刪除 redis 集群中所有的 pvc(pv)
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-0 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-1 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-2 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-3 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-4 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-5 -n redis-cluster
​
(3) 刪除 redis 集群中 pod 對應的 nfs 持久化存儲目錄
[root@k8s_nfs ~]# rm -rf /ifs/kubernetes/archived-redis-cluster-data-redis-cluster-*/
​
(4) 重新創建 redis 集群
[root@k8s-master1 redis-cluster]# kubectl apply -f redis-cluster.yml
​
3 重建集群的方法2: 在原有 redis 集群的基礎上進行修復
(1) 刪除 redis 集群中所有的 pod
[root@k8s-master1 redis-cluster]# kubectl delete -f redis-cluster.yml
​
(2) 找到 redis 集群中 pod 對應的 nfs 持久化存儲目錄后刪除 nodes.conf
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-0*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-1*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-2*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-3*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-4*/nodes.conf*
[root@k8s_nfs ~]# rm -f /ifs/kubernetes/redis-cluster-data-redis-cluster-5*/nodes.conf*
​
(3) 重新創建 redis 集群
[root@k8s-master1 redis-cluster]# kubectl apply -f redis-cluster.yml