OpenShift上部署Redis主從集群

客戶有部署有狀態服務的需求，單機部署模式相對簡單，嘗試一下集群部署。

關於Redis的master,slave 以及sentinal的架構和作用不提，有興趣可以參考之前的博客

https://www.cnblogs.com/ericnie/p/7203950.html

github上參考的文章不少，但很多都需要基於連接互聯網進行構建，同時也創建不少的buildconfig,研究了一下，還需要在本地部署一個git.

立刻覺得把問題搞復雜了。所以本文主要以簡化為主，說明主要核心步驟，當然也掉到坑里浪費了些時間，不失為一種學習過程吧。

github上參考鏈接

https://github.com/mjudeikis/redis-openshift

 

架構

傳統的Redis個節點角色和端口

 

在OpenShift上部署架構圖

 

1.構建鏡像

先clone到本地

git clone https://github.com/mjudeikis/redis-openshift

目錄結構比較清晰明了，我喜歡

[root@master ~]# tree redis-openshift
redis-openshift
├── image
│   ├── Dockerfile
│   ├── Dockerfile.1
│   ├── epel-7.repo
│   ├── redis-master.conf
│   ├── redis-openshift.tar
│   ├── redis-slave.conf
│   └── run.sh
├── list.yaml
├── openshift
│   ├── build.yaml
│   ├── is-base.yaml
│   ├── is-output.yaml
│   ├── redis-master-dc.yaml
│   ├── redis-master-service.yaml
│   ├── redis-sentinel-dc.yaml
│   ├── redis-sentinel-services.yaml
│   └── redis.yaml
└── README.md

對了，這里我也根據我的環境修改了Dockerfile,主要是yum源的問題

先下載一個放到目錄下

http://mirrors.aliyun.com/repo/epel-7.repo，

Dockerfile

FROM rhel7

#RUN yum install epel-release -y
COPY epel-7.repo  /etc/yum.repos.d/epel.repo

RUN yum install redis hostname -y ; yum clean all

COPY redis-master.conf /redis-master/redis.conf
COPY redis-slave.conf /redis-slave/redis.conf
RUN mkdir -p /redis-sentinel ; chmod -R 777 /redis-sentinel /redis-slave
COPY run.sh /run.sh

CMD [ "/run.sh" ]

ENTRYPOINT [ "bash", "-c" ]

比較核心的run文件，我修改成這樣

[root@master image]# cat run.sh
#!/bin/bash

# Copyright 2014 The Kubernetes Authors All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

function launchmaster() {
  if [[ ! -e /redis-master-data ]]; then
    echo "Redis master data doesn't exist, data won't be persistent!"
    mkdir /redis-master-data
  fi
  redis-server /redis-master/redis.conf --protected-mode no
}

function launchsentinel() {
  echo "Enter launch sentinel...."
  
   while true; do
    master=$(redis-cli -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT} --csv SENTINEL get-master-addr-by-name mymaster | tr ',' ' ' | cut -d' ' -f1)
    if [[ -n ${master} ]]; then
      master="${master//\"}"
    else
      master=$(hostname -i)
    fi

    redis-cli -h ${master} INFO
    if [[ "$?" == "0" ]]; then
      break
    fi
    echo "Connecting to master failed.  Waiting..."
    sleep 10
  done

  sentinel_conf=/redis-sentinel/sentinel.conf

  echo "sentinel monitor mymaster ${master} 6379 2" > ${sentinel_conf}
  echo "sentinel down-after-milliseconds mymaster 60000" >> ${sentinel_conf}
  echo "sentinel failover-timeout mymaster 180000" >> ${sentinel_conf}
  echo "sentinel parallel-syncs mymaster 1" >> ${sentinel_conf}
  echo "bind 0.0.0.0" >> ${sentinel_conf}

  redis-sentinel ${sentinel_conf} --protected-mode no

}

function launchslave() {
  echo "Enter lauch slave....."

   while true; do
    master=$(redis-cli -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT} --csv SENTINEL get-master-addr-by-name mymaster | tr ',' ' ' | cut -d' ' -f1)
    if [[ -n ${master} ]]; then
      master="${master//\"}"
    else
      echo "Failed to find master."
      sleep 60
      exit 1
    fi 

    redis-cli -h ${master} INFO
    if [[ "$?" == "0" ]]; then
      break
    fi
    echo "Connecting to master failed.  Waiting..."
    sleep 10
  done
  sed -i "s/%master-ip%/${master}/" /redis-slave/redis.conf
  sed -i "s/%master-port%/6379/" /redis-slave/redis.conf
  redis-server /redis-slave/redis.conf --protected-mode no

}

if [[ "${MASTER}" == "true" ]]; then
  launchmaster
  exit 0
fi

if [[ "${SENTINEL}" == "true" ]]; then
  launchsentinel
  exit 0
fi

launchslave

 

oc new-project建立一個項目，然后

docker build -t  registry.example.com/openshift/redis-openshift:1.0 .

docker push  registry.example.com/openshift/redis-openshift:1.0

oc import-image redis-openshift:latest --from=registry.example.com/openshift/redis-openshift:1.0  --insecure --confirm

這里必須表揚一下openshift的了，因為在調試過程中我的run.sh改來改去，每次改動需要build鏡像，搞完后，直接運行上面三條，就把鏡像刷新了

而Openshift的dc的trigger功能發揮作用，每次發現鏡像更新后就自動更新，發布新應用，減少大量時間，感覺非常不錯！

 

2.構建一系列的dc和service

修改openshift下的配置文件，確保鏡像地址無誤。

oc create -f openshift/redis-master-dc.yaml
oc create -f openshift/redis-master-service.yaml

oc create -f openshift/redis-sentinel-dc.yaml
oc create -f openshift/redis-sentinel-services.yaml

oc create -f openshift/redis.yaml

會建立一個master,3個sentinel和3個slave,如圖

 進去逐個檢查日志，確保成功啟動

運行下面命令，查看集群信息。 

redis-cli -h redis-master  info Replication

 構建成功，可以愉快的使用了。

 

3.集群切換

講redis-master直接scale到0,模擬down掉

然后檢查sentinel的日志，發現mymaster已經從10.129.0.2切換到了10.129.0.8的redis pod.

 

在reids節點上運行 redis-cli -h 10.129.0.8  info Replication

可以看到集群狀態已經發生變化