kubernetes實戰(五)：k8s持久化安裝Redis Sentinel

1、PV創建

　　在nfs或者其他類型后端存儲創建pv，首先創建共享目錄

[root@nfs ~]# cat /etc/exports
/k8s/redis-sentinel/0 *(rw,sync,no_subtree_check,no_root_squash)
/k8s/redis-sentinel/1 *(rw,sync,no_subtree_check,no_root_squash)
/k8s/redis-sentinel/2 *(rw,sync,no_subtree_check,no_root_squash)

　　下載yaml文件

https://github.com/dotbalo/k8s/

　　創建pv，注意Redis的空間大小按需修改

[root@k8s-master01 redis-sentinel]# kubectl create -f redis-sentinel-pv.yaml
[root@k8s-master01 redis-sentinel]# kubectl get pv | grep redis
pv-redis-sentinel-0   4Gi        RWX            Recycle          Bound     public-service/redis-sentinel-master-storage-redis-sentinel-master-ss-0   redis-sentinel-storage-class             16h
pv-redis-sentinel-1   4Gi        RWX            Recycle          Bound     public-service/redis-sentinel-slave-storage-redis-sentinel-slave-ss-0     redis-sentinel-storage-class             16h
pv-redis-sentinel-2   4Gi        RWX            Recycle          Bound     public-service/redis-sentinel-slave-storage-redis-sentinel-slave-ss-1     redis-sentinel-storage-class             16h

 

2、創建namespace

　　默認是在public-service中創建Redis哨兵模式

kubectl create namespace public-service
# 如果不使用public-service，需要更改所有yaml文件的public-service為你namespace。
# sed -i "s#public-service#YOUR_NAMESPACE#g" *.yaml

 

3、創建ConfigMap

　　Redis配置按需修改，默認使用的是rdb存儲模式

[root@k8s-master01 redis-sentinel]# kubectl create -f redis-sentinel-configmap.yaml
[root@k8s-master01 redis-sentinel]# kubectl get configmap -n public-service
NAME                    DATA      AGE
redis-sentinel-config   2         17h

　　注意，此時configmap中redis-slave.conf的slaveof的master地址為ss里面的Headless Service地址。

 

4、創建service

　　service主要提供pods之間的互訪，StatefulSet主要用Headless Service通訊，格式：statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local

　　- serviceName為Headless Service的名字

　　- 0..N-1為Pod所在的序號，從0開始到N-1

　　- statefulSetName為StatefulSet的名字

　　- namespace為服務所在的namespace，Headless Servic和StatefulSet必須在相同的namespace

　　- .cluster.local為Cluster Domain

　　如本集群的HS為：

　　　　Master：

　　　　　　redis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local:6379

　　　　Slave：

　　　　　　redis-sentinel-slave-ss-0.redis-sentinel-slave-ss.public-service.svc.cluster.local:6379

　　　　　　redis-sentinel-slave-ss-1.redis-sentinel-slave-ss.public-service.svc.cluster.local:6379

　　創建Service

[root@k8s-master01 redis-sentinel]# kubectl create -f redis-sentinel-service-master.yaml -f redis-sentinel-service-slave.yaml

[root@k8s-master01 redis-sentinel]# kubectl get service -n public-service
NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
redis-sentinel-master-ss   ClusterIP   None            <none>        6379/TCP                         16h
redis-sentinel-slave-ss    ClusterIP   None            <none>        6379/TCP                         <invalid>

 

5、創建StatefulSet

[root@k8s-master01 redis-sentinel]# kubectl create -f redis-sentinel-rbac.yaml -f redis-sentinel-ss-master.yaml -f redis-sentinel-ss-slave.yaml

[root@k8s-master01 redis-sentinel]# kubectl get statefulset -n public-service
NAME                       DESIRED   CURRENT   AGE
redis-sentinel-master-ss   1         1         16h
redis-sentinel-slave-ss    2         2         16h
rmq-cluster                3         3         3d
[root@k8s-master01 redis-sentinel]# kubectl get pods -n public-service
NAME                         READY     STATUS    RESTARTS   AGE
redis-sentinel-master-ss-0   1/1       Running   0          16h
redis-sentinel-slave-ss-0    1/1       Running   0          16h
redis-sentinel-slave-ss-1    1/1       Running   0          16h

　　此時相當於已經在k8s上創建了Redis的主從模式。

 

6、dashboard查看

　　狀態查看

　　pods通訊測試

　　master連接slave測試

[root@k8s-master01 redis-sentinel]# kubectl exec -ti redis-sentinel-master-ss-0 -n public-service -- redis-cli -h redis-sentinel-slave-ss-0.redis-sentinel-slave-ss.public-service.svc.cluster.local  ping
PONG
[root@k8s-master01 redis-sentinel]# kubectl exec -ti redis-sentinel-master-ss-0 -n public-service -- redis-cli -h redis-sentinel-slave-ss-1.redis-sentinel-slave-ss.public-service.svc.cluster.local  ping
PONG

　　slave連接master測試

[root@k8s-master01 redis-sentinel]# kubectl exec -ti redis-sentinel-slave-ss-0 -n public-service -- redis-cli -h redis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local  ping
PONG
[root@k8s-master01 redis-sentinel]# kubectl exec -ti redis-sentinel-slave-ss-1 -n public-service -- redis-cli -h redis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local  ping
PONG

　　同步狀態查看

[root@k8s-master01 redis-sentinel]# kubectl exec -ti redis-sentinel-slave-ss-1 -n public-service -- redis-cli -h redis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local  info replication
# Replication
role:master
connected_slaves:2
slave0:ip=172.168.5.94,port=6379,state=online,offset=80410,lag=1
slave1:ip=172.168.6.113,port=6379,state=online,offset=80410,lag=0
master_replid:ad4341815b25f12d4aeb390a19a8bd8452875879
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:80410
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:80410

　　同步測試

# master寫入數據
[root@k8s-master01 redis-sentinel]# kubectl exec -ti redis-sentinel-slave-ss-1 -n public-service -- redis-cli -h redis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local  set test test_data
OK
# master獲取數據
[root@k8s-master01 redis-sentinel]# kubectl exec -ti redis-sentinel-slave-ss-1 -n public-service -- redis-cli -h redis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local  get test
"test_data"
# slave獲取數據
[root@k8s-master01 redis-sentinel]# kubectl exec -ti redis-sentinel-slave-ss-1 -n public-service -- redis-cli   get test
"test_data"

　　從節點無法寫入數據

[root@k8s-master01 redis-sentinel]# kubectl exec -ti redis-sentinel-slave-ss-1 -n public-service -- redis-cli   set k v
(error) READONLY You can't write against a read only replica.

　　NFS查看數據存儲

[root@nfs redis-sentinel]# tree .
.
├── 0
│   └── dump.rdb
├── 1
│   └── dump.rdb
└── 2
    └── dump.rdb

3 directories, 3 files

　　說明：個人認為在k8s上搭建Redis sentinel完全沒有意義，經過測試，當master節點宕機后，sentinel選擇新的節點當主節點，當原master恢復后，此時無法再次成為集群節點。因為在物理機上部署時，sentinel探測以及更改配置文件都是以IP的形式，集群復制也是以IP的形式，但是在容器中，雖然采用的StatefulSet的Headless Service來建立的主從，但是主從建立后，master、slave、sentinel記錄還是解析后的IP，但是pod的IP每次重啟都會改變，所有sentinel無法識別宕機后又重新啟動的master節點，所以一直無法加入集群，雖然可以通過固定podIP或者使用NodePort的方式來固定，或者通過sentinel獲取當前master的IP來修改配置文件，但是個人覺得也是沒有必要的，sentinel實現的是高可用Redis主從，檢測Redis Master的狀態，進行主從切換等操作，但是在k8s中，無論是dc或者ss，都會保證pod以期望的值進行運行，再加上k8s自帶的活性檢測，當端口不可用或者服務不可用時會自動重啟pod或者pod的中的服務，所以當在k8s中建立了Redis主從同步后，相當於已經成為了高可用狀態，並且sentinel進行主從切換的時間不一定有k8s重建pod的時間快，所以個人認為在k8s上搭建sentinel沒有意義。所以下面搭建sentinel的步驟無需在看。 

 　　PS：Redis Cluster：https://github.com/dotbalo/k8s/tree/master/redis/k8s-redis-cluster

7、創建sentinel

[root@k8s-master01 redis-sentinel]# kubectl create -f redis-sentinel-ss-sentinel.yaml -f redis-sentinel-service-sentinel.yaml

[root@k8s-master01 redis-sentinel]# kubectl get service -n public-servicve
No resources found.
[root@k8s-master01 redis-sentinel]# kubectl get service -n public-service
NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
redis-sentinel-master-ss     ClusterIP   None            <none>        6379/TCP                         17h
redis-sentinel-sentinel-ss   ClusterIP   None            <none>        26379/TCP                        36m
redis-sentinel-slave-ss      ClusterIP   None            <none>        6379/TCP                         1h
rmq-cluster                  ClusterIP   None            <none>        5672/TCP                         3d
rmq-cluster-balancer         NodePort    10.107.221.85   <none>        15672:30051/TCP,5672:31892/TCP   3d
[root@k8s-master01 redis-sentinel]# kubectl get statefulset -n public-service
NAME                         DESIRED   CURRENT   AGE
redis-sentinel-master-ss     1         1         17h
redis-sentinel-sentinel-ss   3         3         8m
redis-sentinel-slave-ss      2         2         17h
rmq-cluster                  3         3         3d
[root@k8s-master01 redis-sentinel]# kubectl get pods -n public-service | grep sentinel
redis-sentinel-master-ss-0     1/1       Running   0          17h
redis-sentinel-sentinel-ss-0   1/1       Running   0          2m
redis-sentinel-sentinel-ss-1   1/1       Running   0          2m
redis-sentinel-sentinel-ss-2   1/1       Running   0          2m
redis-sentinel-slave-ss-0      1/1       Running   0          17h
redis-sentinel-slave-ss-1      1/1       Running   0          17h

 

8、查看日志

　　查看哨兵狀態

[root@k8s-master01 ~]# kubectl exec -ti redis-sentinel-sentinel-ss-0 -n public-service -- redis-cli -h 127.0.0.1 -p 26379 info Sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.168.6.111:6379,slaves=2,sentinels=3

 

9、容災測試

# 查看當前數據
[root@k8s-master01 ~]# kubectl exec -ti redis-sentinel-master-ss-0 -n public-service -- redis-cli -h 127.0.0.1 -p 6379 get test
"test_data"

　　關閉master節點

　　查看狀態

[root@k8s-master01 ~]# kubectl get pods -n public-service
NAME                           READY     STATUS    RESTARTS   AGE
redis-sentinel-sentinel-ss-0   1/1       Running   0          22m
redis-sentinel-sentinel-ss-1   1/1       Running   0          22m
redis-sentinel-sentinel-ss-2   1/1       Running   0          22m
redis-sentinel-slave-ss-0      1/1       Running   0          17h
redis-sentinel-slave-ss-1      1/1       Running   0          17h

　　查看sentinel狀態

[root@k8s-master01 redis]# kubectl exec -ti redis-sentinel-sentinel-ss-2 -n public-service -- redis-cli -h 127.0.0.1 -p 26379 info Sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.168.6.116:6379,slaves=2,sentinels=3
[root@k8s-master01 redis]# kubectl exec -ti redis-sentinel-slave-ss-0 -n public-service -- redis-cli -h 127.0.0.1 -p 6379 info replication
# Replication
role:slave
master_host:172.168.6.116
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:82961
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:4097ccd725a7ffc6f3767f7c726fc883baf3d7ef
master_replid2:603280e5266e0a6b0f299d2b33384c1fd8c3ee64
master_repl_offset:82961
second_repl_offset:68647
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:82961

贊助作者：