基於Kubernetes構建Jenkins微服務發布平台

發布流程設計

軟件環境：Jenkins + Kubernetes + Gitlab + Harbor+helm

工作流程：手動/自動構建-> Jenkins 調度K8S API-＞動態生成Jenkins Slave pod -＞Slave pod 拉取Git 代碼／編譯／打包鏡像-＞推送到鏡像倉庫Harbor -＞Slave 工作完成，Pod 自動銷毀-＞helm部署到測試或生產Kubernetes平台。

准備基礎前提環境

1、K8s（Ingress Controller，CoreDNS，PV自動供給）
2、Helm v3
3、Gitlab
4、Harbor，並啟用Chart存儲功能
5、MySQL（微服務數據庫）
6、Eureka（注冊中心）

在Kubernetes中部署Jenkins

參考：https://github.com/jenkinsci/kubernetes-plugin/tree/fc40c869edfd9e3904a9a56b0f80c5a25e988fa1/src/main/kubernetes

安裝插件: Git Parameter/Git/Pipeline/Config File Provider/kubernetes/Extended Choice Parameter

由於默認插件源在國外服務器，大多數網絡無法順利下載，需修改國內插件源地址：

cd jenkins_home/updates
sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && \
sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json

Jenkins Pipeline 及參數化構建

參考：https://jenkins.io/doc/book/pipeline/syntax/

• Jenkins Pipeline是一套插件，支持在Jenkins中實現集成和持續交付管道；
• Pipeline通過特定語法對簡單到復雜的傳輸管道進行建模；
• 聲明式：遵循與Groovy相同語法。pipeline { }
• 腳本式：支持Groovy大部分功能，也是非常表達和靈活的工具。node { }
• Jenkins Pipeline的定義被寫入一個文本文件，稱為Jenkinsfile。

Jenkins在Kubernetes中動態創建代理

Jenkins Master/Slave架構

在K8S中Jenkins Master/Slave架構

Kubernetes插件：Jenkins在Kubernetes集群中運行動態代理
插件介紹：https://github.com/jenkinsci/kubernetes-plugin

自定義構建Jenkins Slave鏡像

參考：https://github.com/jenkinsci/docker-jnlp-slave

FROM centos:7
LABEL maintainer ops

RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \
    yum clean all && \
    rm -rf /var/cache/yum/* && \
    mkdir -p /usr/share/jenkins

COPY slave.jar /usr/share/jenkins/slave.jar
COPY jenkins-slave /usr/bin/jenkins-slave
COPY settings.xml /etc/maven/settings.xml
RUN chmod +x /usr/bin/jenkins-slave
COPY helm kubectl /usr/bin/

ENTRYPOINT ["jenkins-slave"]

構建jenkins-slave推送至harbor倉庫

docker build . -t 192.168.0.241/library/jenkins-slave:jdk-1.8
docker push 192.168.0.241/library/jenkins-slave:jdk-1.8

 基於Kubernetes構建Jenkins CI系統

添加憑據

jenkinsfile

拉取代碼 => 代碼編譯 => 單元測試 => 構建鏡像 => Helm部署到K8S => 測試

#!/usr/bin/env groovy

// 公共
def registry = "192.168.0.241"
// 項目
def project = "microservice"
def git_url = "http://192.168.0.138:12580/root/ms.git"
def gateway_domain_name = "gateway.xxxx.com"
def portal_domain_name = "portal.xxxx.com"
// 認證
def image_pull_secret = "registry-pull-secret"
def harbor_registry_auth = "c176b2b2-7eaf-4968-a1e0-8988d6e04057"
def git_auth = "03e08854-c546-43c1-82c9-d053a75953d2"
// ConfigFileProvider ID
def k8s_auth = "6e3f7dd8-6e26-42de-91c3-5b52ff3117ec"

pipeline {
  agent {
    kubernetes {
        label "jenkins-slave"
        yaml """
kind: Pod
metadata:
  name: jenkins-slave
spec:
  containers:
  - name: jnlp
    image: "${registry}/library/jenkins-slave:jdk-1.8"
    imagePullPolicy: Always
    volumeMounts:
      - name: docker-cmd
        mountPath: /usr/bin/docker
      - name: docker-sock
        mountPath: /var/run/docker.sock
      - name: maven-cache
        mountPath: /root/.m2
  volumes:
    - name: docker-cmd
      hostPath:
        path: /usr/bin/docker
    - name: docker-sock
      hostPath:
        path: /var/run/docker.sock
    - name: maven-cache
      hostPath:
        path: /tmp/m2
"""
        }
      
      }
    parameters {
        gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: '選擇發布的分支', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH'        
        extendedChoice defaultValue: 'none', description: '選擇發布的微服務', \
          multiSelectDelimiter: ',', name: 'Service', type: 'PT_CHECKBOX', \
          value: 'gateway-service:9999,portal-service:8080,product-service:8010,order-service:8020,stock-service:8030'
        choice (choices: ['ms', 'demo'], description: '部署模板', name: 'Template')
        choice (choices: ['1', '3', '5', '7'], description: '副本數', name: 'ReplicaCount')
        choice (choices: ['ms'], description: '命名空間', name: 'Namespace')
    }
    stages {
        stage('拉取代碼'){
            steps {
                checkout([$class: 'GitSCM', 
                branches: [[name: "${params.Branch}"]], 
                doGenerateSubmoduleConfigurations: false, 
                extensions: [], submoduleCfg: [], 
                userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_url}"]]
                ])
            }
        }
        stage('代碼編譯') {
            // 編譯指定服務
            steps {
                sh """
                  mvn clean package -Dmaven.test.skip=true
                """
            }
        }
        stage('構建鏡像') {
          steps {
              withCredentials([usernamePassword(credentialsId: "${harbor_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
                sh """
                 docker login -u ${username} -p '${password}' ${registry}
                 for service in \$(echo ${Service} |sed 's/,/ /g'); do
                    service_name=\${service%:*}
                    image_name=${registry}/${project}/\${service_name}:${BUILD_NUMBER}
                    cd \${service_name}
                    if ls |grep biz &>/dev/null; then
                        cd \${service_name}-biz
                    fi
                    docker build -t \${image_name} .
                    docker push \${image_name}
                    cd ${WORKSPACE}
                  done
                """
                configFileProvider([configFile(fileId: "${k8s_auth}", targetLocation: "admin.kubeconfig")]){
                    sh """
                    # 添加鏡像拉取認證
                    kubectl create secret docker-registry ${image_pull_secret} --docker-username=${username} --docker-password=${password} --docker-server=${registry} -n ${Namespace} --kubeconfig admin.kubeconfig |true
                    # 添加私有chart倉庫
                    helm repo add  --username ${username} --password ${password} myrepo http://${registry}/chartrepo/${project}
                    """
                }
              }
          }
        }
        stage('Helm部署到K8S') {
          steps {
              sh """
              common_args="-n ${Namespace} --kubeconfig admin.kubeconfig"
              
              for service in  \$(echo ${Service} |sed 's/,/ /g'); do
                service_name=\${service%:*}
                service_port=\${service#*:}
                image=${registry}/${project}/\${service_name}
                tag=${BUILD_NUMBER}
                helm_args="\${service_name} --set image.repository=\${image} --set image.tag=\${tag} --set replicaCount=${replicaCount} --set imagePullSecrets[0].name=${image_pull_secret} --set service.targetPort=\${service_port} myrepo/${Template}"

                # 判斷是否為新部署
                if helm history \${service_name} \${common_args} &>/dev/null;then
                  action=upgrade
                else
                  action=install
                fi

                # 針對服務啟用ingress
                if [ \${service_name} == "gateway-service" ]; then
                  helm \${action} \${helm_args} \
                  --set ingress.enabled=true \
                  --set ingress.host=${gateway_domain_name} \
                   \${common_args}
                elif [ \${service_name} == "portal-service" ]; then
                  helm \${action} \${helm_args} \
                  --set ingress.enabled=true \
                  --set ingress.host=${portal_domain_name} \
                   \${common_args}
                else
                  helm \${action} \${helm_args} \${common_args}
                fi
              done
              # 查看Pod狀態
              sleep 10
              kubectl get pods \${common_args}
              """
          }
        }
    }
}

 

Pipeline 集成Helm 發布微服務項目

小結：

❖使用Jenkins的插件
  •Git & gitParameter
  •Kubernetes
  •Pipeline
  •Kubernetes Continuous Deploy
  •Config File Provider
  •Extended Choice Parameter
❖CI/CD環境特點
  •Slave彈性伸縮
  •基於鏡像隔離構建環境
  •流水線發布，易維護
❖Jenkins參數化構建可幫助你完成更復雜環境CI/CD