Kubernetes-基於helm安裝部署高可用的Redis及其形態探索（二）

上一章，我們通過實踐和其他文章的幫助，在k8s的環境安裝了redis-ha，並且對其進行了一些實驗來驗證他的主從切換是否有效。本篇中將會分析，究竟是如何實現了redis-ha的主從切換，以及其與K8S平面進行的交互。

 

1.如何實現的redis的搭建

我曾經以為是在helm/chart中寫入了腳本來完成這件事，但是仔細看過代碼之后，並未發現明顯的內容，關於搭建redis-ha和主從切換的腳本。

地址：https://github.com/helm/charts/tree/master/stable/redis-ha

后來，通過查看redis鏡像的日志發現了一些內容，

地址：https://quay.io/repository/smile/redis/manifest/sha256:8948a952920d4495859c984546838d4c9b4c71e0036eef86570922d91cacb3df?tab=layers

可以看到，在這個鏡像構建日志中，有幾個疑似相關內容的文件，/usr/local/bin目錄下的promte.sh,redis-launcher.sh,label-updater.sh

進入到pod中，我們可以看到redis-launcher是作為啟動時就運行的腳本，所以我就推測這一切都是這個文件引起的。

 

 

2.腳本內容

redis-launcher.sh:

bash-4.4# cat /usr/local/bin/redis-launcher.sh 
#!/bin/bash
# Copyright 2017 Ismail KABOUBI
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This script determines whether the pod that executes it will be a Redis Sentinel, Master, or Slave
# The redis-ha Helm chart signals Sentinel status with environment variables. If they are not set, the newly
# launched pod will scan K8S to see if there is an active master. If not, it uses a deterministic means of
# sensing whether it should launch as master then writes master or slave to the label called redis-role
# appropriately. It's this label that determines which LB a pod can be seen through.
#
# The redis-role=master pod is the key for the cluster to get started. Sentinels will wait for it to appear
# in the LB before they finish launching. All other pods wait for the Sentinels to ID the master.
#
# Pods also set the labels podIP and runID. RunID is the first few characters of the unique run_id value
# generated by each Redis sever.
#
# During normal operation, there should be only one redis-role=master pod. If it fails, the Sentinels
# will nominate a new master and change all the redis-role values appropriately.

echo "Starting redis launcher"
echo "Setting labels"
label-updater.sh & plabeler=$!

echo "Selecting proper service to execute"
# Define config file locations
SENTINEL_CONF=/etc/redis/sentinel.conf
MASTER_CONF=/etc/redis/master.conf
SLAVE_CONF=/etc/redis/slave.conf

# Adapt to dynamically named env vars
ENV_VAR_PREFIX=`echo $REDIS_CHART_PREFIX|awk '{print toupper($0)}'|sed 's/-/_/g'`
PORTVAR="${ENV_VAR_PREFIX}MASTER_SVC_SERVICE_PORT"
HOSTVAR="${ENV_VAR_PREFIX}MASTER_SVC_SERVICE_HOST"
MASTER_LB_PORT="${!PORTVAR}"
MASTER_LB_HOST="${!HOSTVAR}"
QUORUM=${QUORUM:-2}

# Only sets AUTH if the ENV var REDIS_PASS is set.
REDISAUTH=""
[ -n "$REDIS_PASS" ] && REDISAUTH="-a $REDIS_PASS" || REDISAUTH=""

# Launch master when `MASTER` environment variable is set
function launchmaster() {
  # If we know we're a master, update the labels right away
  kubectl label --overwrite pod $HOSTNAME redis-role="master"
  echo "Using config file $MASTER_CONF"
  if [[ ! -e /redis-master-data ]]; then
    echo "Redis master data doesn't exist, data won't be persistent!"
    mkdir /redis-master-data
  fi

  if [ -n "$REDIS_PASS" ]; then
    sed -i "s/# requirepass/requirepass ${REDIS_PASS} \n#/" $MASTER_CONF
  fi

  redis-server $MASTER_CONF --protected-mode no $@
}

# Launch sentinel when `SENTINEL` environment variable is set
function launchsentinel() {
  # If we know we're a sentinel, update the labels right away
  kubectl label --overwrite pod $HOSTNAME redis-role="sentinel"
  echo "Using config file $SENTINEL_CONF"

  while true; do
    # The sentinels must wait for a load-balanced master to appear then ask it for its actual IP.
    MASTER_IP=$(kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {..podIP} {.status.containerStatuses[0].state}{"\n"}{end}' -l redis-role=master|grep running|grep $REDIS_CHART_PREFIX|awk '{print $2}'|xargs)
    echo "Current master is $MASTER_IP"

    if [[ -z ${MASTER_IP} ]]; then
      continue
    fi

    timeout -t 3 redis-cli ${REDISAUTH} -h ${MASTER_IP} -p ${MASTER_LB_PORT} INFO
    if [[ "$?" == "0" ]]; then
      break
    fi
    echo "Connecting to master failed.  Waiting..."
    sleep 10
  done

  echo "sentinel monitor mymaster ${MASTER_IP} ${MASTER_LB_PORT} ${QUORUM}" > ${SENTINEL_CONF}
  echo "sentinel down-after-milliseconds mymaster 15000" >> ${SENTINEL_CONF}
  echo "sentinel failover-timeout mymaster 30000" >> ${SENTINEL_CONF}
  echo "sentinel parallel-syncs mymaster 10" >> ${SENTINEL_CONF}
  echo "bind 0.0.0.0" >> ${SENTINEL_CONF}
  echo "sentinel client-reconfig-script mymaster /usr/local/bin/promote.sh" >> ${SENTINEL_CONF}

  if [ -n "$REDIS_PASS" ]; then
   echo "sentinel auth-pass mymaster ${REDIS_PASS}" >> ${SENTINEL_CONF}
  fi

  redis-sentinel ${SENTINEL_CONF} --protected-mode no $@
}

# Launch slave when `SLAVE` environment variable is set
function launchslave() {
  kubectl label --overwrite pod $HOSTNAME redis-role="slave"
  echo "Using config file $SLAVE_CONF"
  if [[ ! -e /redis-master-data ]]; then
    echo "Redis master data doesn't exist, data won't be persistent!"
    mkdir /redis-master-data
  fi

  i=0
  while true; do
    master=${MASTER_LB_HOST}
    timeout -t 3 redis-cli ${REDISAUTH} -h ${master} -p ${MASTER_LB_PORT} INFO
    if [[ "$?" == "0" ]]; then
      break
    fi
    i=$((i+1))
    if [[ "$i" -gt "30" ]]; then
      echo "Exiting after too many attempts"
      kill $plabeler
      exit 1
    fi
    echo "Connecting to master failed.  Waiting..."
    sleep 1
  done

  if [ -n "$REDIS_PASS" ]; then
    sed -i "s/# masterauth/masterauth ${REDIS_PASS} \n#/" $SLAVE_CONF
    sed -i "s/# requirepass/requirepass ${REDIS_PASS} \n#/" $SLAVE_CONF
  fi

  sed -i "s/%master-ip%/${MASTER_LB_HOST}/" $SLAVE_CONF
  sed -i "s/%master-port%/${MASTER_LB_PORT}/" $SLAVE_CONF
  redis-server $SLAVE_CONF --protected-mode no $@
}

#Check if MASTER environment variable is set
if [[ "${MASTER}" == "true" ]]; then
  echo "Launching Redis in Master mode"
  launchmaster
  exit 0
fi

# Check if SENTINEL environment variable is set
if [[ "${SENTINEL}" == "true" ]]; then
  echo "Launching Redis Sentinel"
  launchsentinel
  echo "Launcsentinel action completed"
  exit 0
fi

# Determine whether this should be a master or slave instance
echo "Looking for pods running as master"
MASTERS=`kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {..podIP} {.status.containerStatuses[0].state}{"\n"}{end}' -l redis-role=master|grep running|grep $REDIS_CHART_PREFIX`
if [[ "$MASTERS" == "" ]]; then
  echo "No masters found: \"$MASTERS\" Electing first master..."
  SLAVE1=`kubectl get pod -o jsonpath='{range .items[*]}{.metadata.creationTimestamp} {.metadata.name} {.status.containerStatuses[0].state} {"\n"} {end}' -l redis-node=true |grep running|sort|awk '{print $2}'|grep $REDIS_CHART_PREFIX|head -n1`
  if [[ "$SLAVE1" == "$HOSTNAME" ]] || [[ "$SLAVE1" == "" ]]; then
    echo "Taking master role"
    launchmaster
  else
    echo "Electing $SLAVE1 master"
    launchslave
  fi
  exit 0
else
  echo "Found $MASTERS"
  echo "Launching Redis in Slave mode"
  launchslave
  exit 0
fi

echo "Launching Redis in Slave mode"
launchslave
echo "Launchslave action completed"

 

label-updater.sh

bash-4.4# cat /usr/local/bin/label-updater.sh 
# Push some helpful vars into labels
PODIP=`hostname -i`
echo podIP $PODIP
kubectl label --overwrite pod $HOSTNAME podIP="$PODIP"

if [ "$SENTINEL" ]; then
    exit
fi

RUNID=""

# Only sets AUTH if the ENV var REDIS_PASS is set.
REDISAUTH=""
[ -n "$REDIS_PASS" ] && REDISAUTH="-a $REDIS_PASS" || REDISAUTH=""

while true; do
  RUNID=`redis-cli $REDISAUTH info server |grep run_id|awk -F: '{print $2}'|head -c6`
  if [ -n "$RUNID" ]; then
    kubectl label --overwrite pod $HOSTNAME runID="$RUNID"
    break
  else
    sleep 1
  fi
done

　　

promote.sh

bash-4.4# cat /usr/local/bin/promote.sh 
#!/usr/bin/env bash
MASTERIP=$6

# Convert the IP of the promoted pod to a hostname
MASTERPOD=`kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {..podIP} {.status.containerStatuses[0].state}{"\n"}{end}' -l redis-role=slave --sort-by=.metadata.name|grep running|grep $MASTERIP|awk '{print $1}'`
echo "PROMO ARGS: $@"
echo "PROMOTING $MASTERPOD ($MASTERIP) TO MASTER"
kubectl label --overwrite pod $MASTERPOD redis-role="master"

# Demote anyone else who jumped to master
kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {.status.containerStatuses[0].state}{"\n"}{end}' -l redis-role=master --sort-by=.metadata.name|grep running|awk '{print $1}'|grep $REDIS_CHART_PREFIX|grep -v $MASTERPOD|xargs -n1 -I% kubectl label --overwrite pod % redis-role="slave"
echo "OTHER MASTERS $MASTERS"

　　

 

3.大致原理

詳細的內容我還沒有開始看，但是可以講一下大致的原理，就是在每個pod在啟動的時候都會起這樣的一個redis-launcher的進程，這個就像一個agent一樣，主要完成redis的master，slave和sentinel的配置，同時，他會將各個pod的角色反向通過kubectl命令傳給K8S平面。

如果是發生了主從結構已經起來了，但是中途因為某種原因掛掉了，則會通過監控sentinel的狀態來觸發更改K8S平面pod的Role的過程。這個設定是在啟動sentinel完成的，代碼在這里：

  echo "sentinel client-reconfig-script mymaster /usr/local/bin/promote.sh" >> ${SENTINEL_CONF}

 

他會監控，如果對於mymaster這個集群中的sentinel發生了reconfig的事件的時候，就會去觸發/usr/local/bin/promote.sh這個腳本。

 

所以可以看到，是redis的pod在控制，而不是K8S平面在進行控制。之后有時間，我會詳細的讀一下這個腳本，然后加上一些注釋。

 

 

更多openstack/trove的文章：http://www.cnblogs.com/S-tec-songjian/

此文章屬博客園用戶S-tec原創作品，受國家《著作權法》保護，未經許可，任何單位及個人不得做營利性使用；若僅做個人學習、交流等非營利性使用，應當指明作者姓名、作品名稱，原文地址，並且不得侵犯作者依法享有的其他權利。