碼上快樂

相關內容    簡體    繁體

基於Kubernetes構建企業Jenkins CI/CD平台

本文轉載自   查看原文   2020-04-29 11:09   2381    kubernetes

基於Kubernetes構建企業Jenkins CI/CD平台

1.藍綠發布

項目邏輯上分為AB組，在項目升級時，首先把A組從負 載均衡中摘除，進行新版本的部署。

B組仍然繼續提供 服務。A組升級完成上線，B組從負載均衡中摘除。

 

特點：

  • 策略簡單
  • 升級/回滾速度快
  • 用戶無感知，平滑過渡

 

缺點：

  • 需要兩倍以上服務器資源
  • 短時間內浪費一定資源成本

image.png

 

2.灰度發布

灰度發布：只升級部分服務，即讓一部分用戶繼續用 老版本，一部分用戶開始用新版本，如果用戶對新版 本沒有什么意見，那么逐步擴大范圍，把所有用戶都 遷移到新版本上面來。

特點：

  • 保證整體系統穩定性
  • 用戶無感知，平滑過渡

 

缺點：

  • 自動化要求高

image.png

 

k8s中的落地方式
image.png

3.滾動發布

滾動發布：

每次只升級一個或多個服務，升級完成 后加入生產環境，不斷執行這個過程，直到集群中 的全部舊版升級新版本。 特點：

  • 用戶無感知，平滑過渡

缺點：

  • 部署周期長
  • 發布策略較復雜
  • 不易回滾

image.png

 

4.發布流程

image.png

image.png

 

5在Kubernetes中部署Jenkins

image.png

部署文檔：https://github.com/jenkinsci/kubernetes-plugin/tree/fc40c869edfd9e3904a9a56b0f80c5a25e988fa1/src/main/kubernetes

 

mkdir k8s-ci/jenkins -p && cd k8s-cli/jenkins

rabc.yml

--- # 創建名為jenkins的ServiceAccount apiVersion: v1 kind: ServiceAccount metadata:  name: jenkins  --- # 創建名為jenkins的Role，授予允許管理API組的資源Pod kind: Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata:  name: jenkins rules: - apiGroups: [""]  resources: ["pods"]  verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""]  resources: ["pods/exec"]  verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""]  resources: ["pods/log"]  verbs: ["get","list","watch"] - apiGroups: [""]  resources: ["secrets"]  verbs: ["get"]  --- # 將名為jenkins的Role綁定到名為jenkins的ServiceAccount apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata:  name: jenkins roleRef:  apiGroup: rbac.authorization.k8s.io  kind: Role  name: jenkins subjects: - kind: ServiceAccount  name: jenkins

statefulset.yml

apiVersion: apps/v1 kind: StatefulSet metadata:  name: jenkins  labels:  name: jenkins spec:  serviceName: jenkins  replicas: 1  updateStrategy:  type: RollingUpdate  selector:  matchLabels:  name: jenkins  template:  metadata:  name: jenkins  labels:  name: jenkins  spec:  terminationGracePeriodSeconds: 10  serviceAccountName: jenkins  containers:  - name: jenkins  image: jenkins/jenkins:lts-alpine  imagePullPolicy: Always  ports:  - containerPort: 8080  - containerPort: 50000  resources:  limits:  cpu: 1  memory: 1Gi  requests:  cpu: 0.5  memory: 500Mi  env:  - name: LIMITS_MEMORY  valueFrom:  resourceFieldRef:  resource: limits.memory  divisor: 1Mi  - name: JAVA_OPTS  value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MAR GIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 volumeMounts:  - name: jenkins-home  mountPath: /var/jenkins_home  livenessProbe:  httpGet:  path: /login  port: 8080  initialDelaySeconds: 60  timeoutSeconds: 5  failureThreshold: 12  readinessProbe:  httpGet:  path: /login  port: 8080  initialDelaySeconds: 60  timeoutSeconds: 5  failureThreshold: 12  securityContext:  fsGroup: 1000  volumeClaimTemplates:  - metadata:  name: jenkins-home  spec:  storageClassName: "managed-nfs-storage"  accessModes: [ "ReadWriteOnce" ]  resources:  requests:  storage: 1Gi

service.yml

apiVersion: v1 kind: Service metadata:  name: jenkins spec:  selector:  name: jenkins  type: NodePort  ports:  -  name: http  port: 80  targetPort: 8080  protocol: TCP  nodePort: 30006  -  name: agent  port: 50000  protocol: TCP

ingress.yml

apiVersion: extensions/v1beta1 kind: Ingress metadata:  name: jenkins  annotations:  nginx.ingress.kubernetes.io/ssl-redirect: "true"  kubernetes.io/tls-acme: "true"  # 如果上傳插件超出默認會報"413 Request Entity Too Large", 增加 client_max_body_size  nginx.ingress.kubernetes.io/proxy-body-size: 50m  nginx.ingress.kubernetes.io/proxy-request-buffering: "off"  # nginx-ingress controller版本小於 0.9.0.beta-18 的配置  ingress.kubernetes.io/ssl-redirect: "true"  ingress.kubernetes.io/proxy-body-size: 50m  ingress.kubernetes.io/proxy-request-buffering: "off" spec:  rules:  - host: jenkins.example.com  http:  paths:  - path: /  backend:  serviceName: jenkins  servicePort: 80

批量完成創建

image.png

這樣創建之后 就會通過 sts 控制器在node上 拉起一個jenkins 容器，可以通過node ip:svc port 訪問jenkins

插件默認先不安裝--》創建個admin 用戶---》更換默認插件源 ---》重啟jenkins 容器----》安裝 git  pipline kubernetes插件

 

 更改配置 使用api 重啟jenkins   頁面訪問 http://192.168.31.65:30006/restart

jenkins master/slave架構

這種架構主要解決 單jenkins執行效率低，資源不足等問題，jenkins master 調度任務到 slave上 並發執行任務。

提升任務執行的效率

image.png

image.png

 

傳統添加jenkins 節點是在jenkins頁面 jenins nodes 添加

image.png

那么在 k8s 中如何動態創建jenkins slave 節點呢？

jienkins中的Kubernetes插件：Jenkins在Kubernetes集群中運行動態代理 插件介紹：https://github.com/jenkinsci/kubernetes-plugin

 

 

一個 slave節點具體的條件：


1.git

2.鏡像構建，推送倉庫（docker in docker）

3.slave.jar （jenkins slave）  從jenkins 中獲取   http://192.168.31.65:30006/jnlpJars/slave.jar

4.maven  jdk （java 項目）

 

mkdir k8s-ci/jenkins-slave && cd k8s-ci/jenkins-slave

 

jenkins  slave  Dockerfile

FROM centos:7 LABEL maintainer lizhenliang  RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \  yum clean all && \  rm -rf /var/cache/yum/* && \  mkdir -p /usr/share/jenkins  COPY slave.jar /usr/share/jenkins/slave.jar COPY jenkins-slave /usr/bin/jenkins-slave COPY settings.xml /etc/maven/settings.xml RUN chmod +x /usr/bin/jenkins-slave  ENTRYPOINT ["jenkins-slave"]

📎slave.jar📎settings.xml

jenkins-slave       用來設置 這個鏡像的  entrypoint  啟動jenkins slave

#!/usr/bin/env sh  if [ $# -eq 1 ]; then   # if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image  exec "$@"  else   # if -tunnel is not provided try env vars  case "$@" in  *"-tunnel "*) ;;  *)  if [ ! -z "$JENKINS_TUNNEL" ]; then  TUNNEL="-tunnel $JENKINS_TUNNEL"  fi ;;  esac   # if -workDir is not provided try env vars  if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then  case "$@" in  *"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;  *)  WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;  esac  fi   if [ -n "$JENKINS_URL" ]; then  URL="-url $JENKINS_URL"  fi   if [ -n "$JENKINS_NAME" ]; then  JENKINS_AGENT_NAME="$JENKINS_NAME"  fi   if [ -z "$JNLP_PROTOCOL_OPTS" ]; then  echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"  JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"  fi   # If both required options are defined, do not pass the parameters  OPT_JENKINS_SECRET=""  if [ -n "$JENKINS_SECRET" ]; then  case "$@" in  *"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;  *)  OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;  esac  fi   OPT_JENKINS_AGENT_NAME=""  if [ -n "$JENKINS_AGENT_NAME" ]; then  case "$@" in  *"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;  *)  OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;  esac  fi   #TODO: Handle the case when the command-line and Environment variable contain different values.  #It is fine it blows up for now since it should lead to an error anyway.   exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@" fi

 

構建 jenkins slave 鏡像 並推送至 私有倉庫

 

 docker build -t 192.168.31.70/library/jenkins-slave-jdk:1.8 .  docker push 192.168.31.70/library/jenkins-slave-jdk:1.8

 

測試 job

image.png

配置插件

manage jenkins --》configur system ---》拉到最底下 add cloud 選擇kubernetes

image.png

image.png

測試的pipeline 腳本

從私有倉庫拉取 jenkins-slave鏡像，並起一個pod，完成 打印123

實際過程是，jenkins 會提交一個 yaml文件 給k8s 編排生成一個pod

// 公共 def registry = "192.168.31.70"  podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [  containerTemplate(  name: 'jnlp',  image: "${registry}/library/jenkins-slave-jdk:1.8"  ),  ],  volumes: [  hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),  hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')  ], ) {  node("jenkins-slave"){  // 第一步  stage('拉取代碼'){  echo "123"  }  } }

點擊構建 最終會創建出一個 jenkins-slave pod提供 構建功能

image.png

image.png

 

 

6.k8s完整發布流程

添加憑據

添加harbor 憑據

image.png

添加git憑據

image.png

 

安裝插件

 

Kubernetes Continuous Deploy插件：

用於將資源配置部署到Kubernetes。

插件介紹：https://plugins.jenkins.io/kubernetes-cd

支持以下資源類型:

  • Deployment
  • Replica Set
  • Daemon Set
  • StatefulSet
  • Pod • Job
  • Service
  • Ingress
  • Secret

生成config文件

 

mkdir admin-cert && cd admin-cert

admin-csr.json

 

{  "CN": "admin",  "hosts": [],  "key": {  "algo": "rsa",  "size": 2048  },  "names": [  {  "C": "CN",  "ST": "BeiJing",  "L": "BeiJing",  "O": "system:masters",  "OU": "System"  }  ] }

ca-config.json

{  "signing": {  "default": {  "expiry": "87600h"  },  "profiles": {  "kubernetes": {  "expiry": "87600h",  "usages": [  "signing",  "key encipherment",  "server auth",  "client auth"  ]  }  }  } }

 

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

 

cp admin*pem /opt/kubernetes/ssl/ export KUBE_APISERVER="https://192.168.31.63:6443" kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISE RVER}  kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISE RVER}  kubectl config set-credentials admin --client-certificate=/opt/kubernetes/ssl/admin.pem --embed-certs=true --client-key=/opt/kube rnetes/ssl/admin-key.pem  kubectl config set-context kubernetes --cluster=kubernetes --user=admin kubectl config use-context kubernetes

 

image.png

image.png

image.png

既然定義了 部署時使用的yaml  ，那么可以將yaml文件存放在git 倉庫中。


image.png

將以下deploy.yaml復制到 頁面git中,這次拉群 代碼時也會將這個yam 拉取下來。

deploy.yaml

--- apiVersion: apps/v1 kind: Deployment metadata:  name: web spec:  replicas: 3  selector:  matchLabels:  app: java-demo  template:  metadata:  labels:  app: java-demo  spec:  imagePullSecrets:  - name: $SECRET_NAME  containers:  - name: tomcat  image: $IMAGE_NAME  ports:  - containerPort: 8080  name: web  livenessProbe:  httpGet:  path: /  port: 8080  initialDelaySeconds: 60  timeoutSeconds: 5  failureThreshold: 12  readinessProbe:  httpGet:  path: /  port: 8080  initialDelaySeconds: 60  timeoutSeconds: 5  failureThreshold: 12   --- apiVersion: v1 kind: Service metadata:  name: web spec:  type: NodePort  selector:  app: java-demo  ports:  - protocol: TCP  port: 80  targetPort: 8080  --- apiVersion: extensions/v1beta1 kind: Ingress metadata:  name: web spec:  rules:  - host: java.example.com  http:  paths:  - path: /  backend:  serviceName: web  servicePort: 80

 

創建一個登錄registry的secret

 

kubectl create secret docker-registry docker-regsitry-auth --docker-username=admin --docker-password=Harbor12345 --docker-server=192.16 8.31.70

 

 

完整發布應用的 pipeline 腳本

Jenkinsfile

// 公共 def registry = "192.168.31.70" // 項目 def project = "welcome" def app_name = "demo" def image_name = "${registry}/${project}/${app_name}:${Branch}-${BUILD_NUMBER}" def git_address = "http://192.168.31.61:9999/root/java-demo.git" // 認證 def secret_name = "docker-regsitry-auth" def docker_registry_auth = "6170ebd8-3b94-409b-ad69-2a4753701041" def git_auth = "3ca1f434-a566-442b-b598-188ac3d07ae2" def k8s_auth = "58cc1edc-4a58-4541-9e74-cbc29c776b9e"  podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [  containerTemplate(  name: 'jnlp',  image: "${registry}/library/jenkins-slave-jdk:1.8"  ),  ],  volumes: [  hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),  hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')  ], ) {  node("jenkins-slave"){  // 第一步  stage('拉取代碼'){  checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"] ]]) }  // 第二步  stage('代碼編譯'){  sh "mvn clean package -Dmaven.test.skip=true"  }  // 第三步  stage('構建鏡像'){  withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')])  {  sh """  echo '  FROM lizhenliang/tomcat  RUN rm -rf /usr/local/tomcat/webapps/*  ADD target/*.war /usr/local/tomcat/webapps/ROOT.war  ' > Dockerfile  docker build -t ${image_name} .  docker login -u ${username} -p '${password}' ${registry}  docker push ${image_name}  """  }  }  // 第四步  stage('部署到K8S平台'){  sh """  sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yaml  sed -i 's#\$SECRET_NAME#${secret_name}#' deploy.yaml  """  kubernetesDeploy configs: 'deploy.yaml', kubeconfigId: "${k8s_auth}"  }  } }

執行構建等待構建完成

image.png

7.驗證部署

驗證 svc  和 pod 的部署

image.png

image.png

 

驗證ingress

image.png

在本地hosts文件 添加對應的域名記錄

image.png

image.png

 

8.總結：

❖ 使用Jenkins的插件

  • Git
  • Kubernetes
  • Pipeline
  • Kubernetes Continuous Deploy

 

❖ CI/CD環境特點

  • Slave彈性伸縮
  • 基於鏡像隔離構建環境
  • 流水線發布，易維護

 

❖ Jenkins參數化構建

  • 可幫助你完成更復雜環境CI/CD


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 DevOps實踐之一:基於Docker構建企業Jenkins CI平台 jenkins-gitlab-harbor-ceph基於Kubernetes的CI/CD運用（一） jenkins-gitlab-harbor-ceph基於Kubernetes的CI/CD運用（二） Jenkins實現CI/CD自動構建和發布步驟 【CI/CD】Jenkins找不到本地依賴包構建失敗 .NET 微服務——CI/CD（1）：Jenkins+Gitee自動構建 jenkins部署與CI/CD Jenkins 配置CI/CD任務 k8s + gitlab + jenkins +docker 搭建企業級CI/CD 流水線操作 gitlab+jenkins+tomcat CI/CD 部署
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM