升級最新版Rancher 2.2.6

前言：之前采用離線方式部署好了 Rancher 2.2.4（https://www.cnblogs.com/weavepub/p/11053099.html），這次升級到最新版本 Rancher 2.2.6。

一、同步鏡像

1.1 安裝docker

新啟動一台虛擬主機，安裝docker（不要用生產環境，免得主機上的鏡像太多）

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce -y
systemctl restart docker
systemctl enable docker
systemctl status docker

1.2 下載鏡像列表文件

mkdir -p /opt/rancher/upgrade2.2.6 && cd /opt/rancher/upgrade2.2.6
https://github.com/rancher/rancher/releases/tag/v2.2.6/rancher-images.txt

1.3准備鏡像腳本

1）鏡像下載腳本

vim rancher-save-images.sh

內容如下：

#!/bin/bash
# 定義日志
workdir=`pwd`
log_file=${workdir}/sync_images_$(date +"%Y-%m-%d").log

logger()
{
    log=$1
    cur_time='['$(date +"%Y-%m-%d %H:%M:%S")']'
    echo ${cur_time} ${log} | tee -a ${log_file}
}

list="rancher-images.txt"
#images="rancher-images.tar.gz"

POSITIONAL=()
while [[ $# -gt 0 ]]; do
    key="$1"
    case $key in
        -i|--images)
        images="$2"
        shift # past argument
        shift # past value
        ;;
        -l|--image-list)
        list="$2"
        shift # past argument
        shift # past value
        ;;
        -h|--help)
        help="true"
        shift
    ;;
    esac
done

usage () {
    echo "USAGE: $0 [--image-list rancher-images.txt] [--images rancher-images.tar.gz]"
    echo "  [-l|--images-list path] text file with list of images. 1 per line."
    echo "  [-l|--images path] tar.gz generated by docker save."
    echo "  [-h|--help] Usage message"
}

if [[ $help ]]; then
    usage
    exit 0
fi

set -e -x

mkdir -p rancher-images-$(date +"%Y-%m-%d")
cd rancher-images-$(date +"%Y-%m-%d")

for i in $(cat ${list}); 
do
    docker pull ${i}

    if [ $? -ne 0 ]; then
        logger "${i} pull failed."
    else
        logger "${i} pull successfully."
    fi

    docker save ${i} | gzip > $(echo $i | sed "s#/#-#g; s#:#-#g").tgz

    if [ $? -ne 0 ]; then
        logger "${i} save failed."
    else
        logger "${i} save successfully."
    fi
done

 

2）鏡像上傳到私有倉庫腳本

vim rancher-push-images.sh

內容如下：

#!/bin/bash

## 鏡像上傳說明
# 需要先在鏡像倉庫中創建 rancher 項目
# 根據實際情況更改以下私有倉庫地址

# 定義日志
workdir=`pwd`
log_file=${workdir}/sync_images_$(date +"%Y-%m-%d").log

logger()
{
    log=$1
    cur_time='['$(date +"%Y-%m-%d %H:%M:%S")']'
    echo ${cur_time} ${log} | tee -a ${log_file}
}

images_hub() {

    while true; do
        read -p "輸入鏡像倉庫地址(不加http/https): " registry
        read -p "輸入鏡像倉庫用戶名: " registry_user
        read -p "輸入鏡像倉庫用戶密碼: " registry_password
        echo "您設置的倉庫地址為: ${registry},用戶名: ${registry_user},密碼: xxx"
        read -p "是否確認(Y/N): " confirm

        if [ $confirm != Y ] && [ $confirm != y ] && [ $confirm == '' ]; then
            echo "輸入不能為空，重新輸入"
        else
            break
        fi
    done
}

images_hub

echo "鏡像倉庫 $(docker login -u ${registry_user} -p ${registry_password} ${registry})"

images=$(cat ./rancher-images.txt)
namespace=rancher

docker_push() {
    for imgs in $(echo ${images}); do
        n=$(echo ${imgs} | awk -F"/" '{print NF-1}')
        #如果鏡像名中沒有/，那么此鏡像一定是library倉庫的鏡像；
        if [ ${n} -eq 0 ]; then
            img_tag=${imgs}
            #namespace=rancher
            #重命名鏡像
            docker tag ${imgs} ${registry}/${namespace}/${img_tag}
            #刪除原始鏡像
            #docker rmi ${imgs}
            #上傳鏡像
            docker push ${registry}/${namespace}/${img_tag}
        #如果鏡像名中有一個/，那么/左側為項目名，右側為鏡像名和tag
        elif [ ${n} -eq 1 ]; then
            img_tag=$(echo ${imgs} | awk -F"/" '{print $2}')
            #namespace=$(echo ${imgs} | awk -F"/" '{print $1}')
            #重命名鏡像
            docker tag ${imgs} ${registry}/${namespace}/${img_tag}
            #刪除舊鏡像
            #docker rmi ${imgs}
            #上傳鏡像
            docker push ${registry}/${namespace}/${img_tag}
        #如果鏡像名中有兩個/，
        elif [ ${n} -eq 2 ]; then
            img_tag=$(echo ${imgs} | awk -F"/" '{print $3}')
            #namespace=$(echo ${imgs} | awk -F"/" '{print $2}')
            #重命名鏡像
            docker tag ${imgs} ${registry}/${namespace}/${img_tag}
            #刪除舊鏡像
            #docker rmi ${imgs}
            #上傳鏡像
            docker push ${registry}/${namespace}/${img_tag}
        else
            #標准鏡像為四層結構，即：倉庫地址/項目名/鏡像名:tag,如不符合此標准，即為非有效鏡像。
            echo "No available images"
        fi
    done
}

docker_push

 

1.4 同步鏡像

1）下載鏡像

chmod +x rancher-save-images.sh
./rancher-save-images.sh --image-list /opt/rancher/upgrade2.2.6/rancher-images.txt

2）上傳鏡像

chmod +x rancher-push-images.sh
./rancher-push-images.sh
reg.nexus.wmqe.com
admin
*******

 

二、升級Rancher

2.1 下載 Rancher 2.2.6 離線包

登入到當初部署 Rancher 時的 prod-rancher-server-01 主機

1）更新本地helm repo緩存

helm repo update

2）查看Rancher 最新版本

helm search rancher
NAME                      CHART VERSION    APP VERSION    DESCRIPTION                                                 
rancher-stable/rancher    2.2.6            v2.2.6         Install Rancher Server to manage Kubernetes clusters acro...

3）獲取Rancher Charts離線包

mkdir -p /opt/rancher/upgrade2.2.6 && cd /opt/rancher/upgrade2.2.6
helm fetch rancher-stable/rancher --version v2.2.6

2.2 升級 Rancher

1）升級Rancher

升級參數應該以安裝時設置的參數為准。

tar zxvf rancher-2.2.6.tgz
kubeconfig=/root/.kube/config

helm --kubeconfig=$kubeconfig upgrade rancher ./rancher \
    --set hostname=rancher.wmqe.com \
    --set ingress.tls.source=secret \
    --set auditLog.level=3 \
    --set auditLog.destination=hostPath \
    --set auditLog.hostPath=/opt/rancher/auditlogs/ \
    --set auditLog.maxAge=20 \
    --set auditLog.maxBackups=20 \
    --set auditLog.maxSize=100 \
    --set rancherImage=reg.nexus.wmqe.com/rancher/rancher

2）查看pod狀態

kubectl get pod -A

NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE
cattle-system   cattle-cluster-agent-c87688dc5-xzlqg      1/1     Running     0          27m
cattle-system   cattle-node-agent-9tl5c                   1/1     Running     0          26m
cattle-system   cattle-node-agent-vrrmr                   1/1     Running     0          27m
cattle-system   cattle-node-agent-xrdkq                   1/1     Running     0          26m
cattle-system   rancher-5ddf8d4897-lmw5v                  2/2     Running     0          28m
cattle-system   rancher-5ddf8d4897-wnfqn                  2/2     Running     0          29m
cattle-system   rancher-5ddf8d4897-xcx5s                  2/2     Running     0 28m
ingress-nginx   default-http-backend-6d4b9c4cc7-5qh4c     1/1     Running     0          41d
ingress-nginx   nginx-ingress-controller-dll87            1/1     Running     0          41d
ingress-nginx   nginx-ingress-controller-fnwvb            1/1     Running     0          41d
ingress-nginx   nginx-ingress-controller-pzcxb            1/1     Running     0          41d
kube-system     canal-4qnsr                               2/2     Running     0          41d
kube-system     canal-b7p6v                               2/2     Running     0          41d
kube-system     canal-wv8p5                               2/2     Running     0          41d
kube-system     kube-dns-7fbcd74695-j7fx7                 3/3     Running     0          41d
kube-system     kube-dns-autoscaler-6b777db656-58xxz      1/1     Running     0          41d
kube-system     metrics-server-7c57d997f9-j7lt4           1/1     Running     0          41d
kube-system     rke-ingress-controller-deploy-job-nq7sq   0/1     Completed   0          41d
kube-system     rke-kube-dns-addon-deploy-job-5flrd       0/1     Completed   0          41d
kube-system     rke-metrics-addon-deploy-job-8tzcr        0/1     Completed   0          41d
kube-system     rke-network-plugin-deploy-job-vq4gh       0/1     Completed   0          41d
kube-system     tiller-deploy-768c796496-c4cd2            1/1     Running     0          41d

標紅的都重新創建后，就可以成功訪問Rancher主界面了。

 

三、清理節點

當升級時候，某一台節點若升級失敗，則需清理該節點，然后重新加入，下面介紹清理節點的步驟：

1、在 rancher 中剔除主機

2、登入該主機刪除所有容器

docker rm -f $(docker ps -qa)

3、刪除所有容器卷

docker volume rm $(docker volume ls -q)

4、備份目錄

mv /etc/kubernetes /etc/kubernetes-bak-$(date +"%Y%m%d%H%M")
mv /var/lib/etcd /var/lib/etcd-bak-$(date +"%Y%m%d%H%M")
mv /var/lib/rancher /var/lib/rancher-bak-$(date +"%Y%m%d%H%M")
mv /opt/rke /opt/rke-bak-$(date +"%Y%m%d%H%M")

5、刪除殘留路徑

rm -rf /etc/ceph \
/etc/cni \
/opt/cni \
/run/secrets/kubernetes.io \
/run/calico \
/run/flannel \
/var/lib/calico \
/var/lib/cni \
/var/lib/kubelet \
/var/log/containers \
/var/log/pods \
/var/run/calico

6、重啟docker

systemctl restart docker

參考：https://www.cnrancher.com/docs/rancher/v2.x/cn/install-prepare/remove-node/