使用Jenkins Operator在k8s中部署Jenkins

目錄

• 1、Jenkins Operator介紹
• 2、Jenkins Operator的架構和設計
• 3、使用Operator部署Jenkins
  • 3.1 前提條件
  • 3.2 獲取並創建CRD
  • 3.3 部署Jenkins Operator
  • 3.4 部署Jenkins

1、Jenkins Operator介紹

Kubernetes Operator是一種特定於應用的控制器，可擴展Kubernetes API的功能，來代表Kubernetes用戶創建、配置和管理復雜應用的實例

這里對Operator的相關介紹就不多贅述了，主要還是回到Jenkins Operator這個話題

基於k8s上Jenkins的常規安裝是使用yaml資源清單，更為方便一點的是helm chart，但是常常我們在安裝后還需要做很多的動作。例如插件問題，這盡管可以通過Configuration as Code的方式來解決，根據個人實際經驗來看，還是存在一定幾率會因主鏡像版本、環境等存在諸多不可預知的問題。當然不同的部署方式都各有利弊，大家根據實際情況選擇即可。

直到官方支持Jenkins可以在k8s中通過Operator方式部署，在4月中旬，Jenkins blog說道：Jenkins Operator 正式成為了 Jenkins 的子項目，填補了Jenkins與Kubernetes間的縫隙。也就是說，最初由（個人）三方團隊編寫的Jenkins Operator被Jenkins官方認可了

參考官方說明，Jenkins Operator可以幫我們解決以下問題：

• 安裝指定版本的插件

  即使最新版本插件不兼容或具備安全漏洞，還是為了插件穩定性而使用（因為常常會出現我們通過一鍵升級插件導致很多問題而去手動安裝舊版本插件的情況）

• 更好的自定義配置

  包含在安裝指定版本插件時指定插件配置等聲明式配置

• 開箱即用的安全配置

• 可靈活調整的debug錯誤調試

• 備份和還原作業歷史記錄

  ......

2、Jenkins Operator的架構和設計

參考Jenkins Operator Architecture and design

Jenkins Operator的設計包含以下概念

• 監視清單的任何更改，並根據已部署的自定義資源清單維護所需的狀態
• 實現主reconciliation循環，由兩個較小的reconciliation循環：base和user

Base reconciliation 循環負責監聽Jenkins基礎配置：

• 確認清單-監聽清單中發生的任何更改
• 確保Jenkins Pod狀態，創建和驗證Jenkins Server Pod的狀態
• 確認Jenkins的配置，包括安全加固、初始化配置等
• 確認Jenkins API token，生成token並初始化Jenkins Client

User reconciliation循環負責協調用戶提供的配置：

• 確保恢復任務，創建恢復任務，並確保恢復已成功執行
• 確保Seed Jobs，創建Seed Jobs並確保所有這些工作都已成功執行
• 確保用戶配置，執行用戶提供的配置，如groovy腳本，配置為代碼或插件
• 確保備份任務，創建備份任務並確保備份成功

Operator狀態
Operator狀態保存在自定義資源狀態部分中，該部分用於存儲Operator管理的任何配置事件或Job狀態

即使操作者或Jenkins重新啟動，它也能幫助保持或恢復所需的狀態

3、使用Operator部署Jenkins

3.1 前提條件

參考Jenkins Operator官方文檔，需要有一個1.11+版本的Kubernetes集群，這里我的環境如下

# kubectl version -o yaml
clientVersion:
  buildDate: "2020-12-08T17:59:43Z"
  compiler: gc
  gitCommit: af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38
  gitTreeState: clean
  gitVersion: v1.20.0
  goVersion: go1.15.5
  major: "1"
  minor: "20"
  platform: darwin/amd64
serverVersion:
  buildDate: "2021-01-13T13:20:00Z"
  compiler: gc
  gitCommit: faecb196815e248d3ecfb03c680a4507229c2a56
  gitTreeState: clean
  gitVersion: v1.20.2
  goVersion: go1.15.5
  major: "1"
  minor: "20"
  platform: linux/amd64

3.2 獲取並創建CRD

獲取yaml並創建crd，當然也可以通過直接apply遠程地址，這里先將其保存到本地

# wget -c https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/crds/jenkins_v1alpha2_jenkins_crd.yaml
# kubectl apply -f jenkins_v1alpha2_jenkins_crd.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/jenkins.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/jenkinsimages.jenkins.io created

3.3 部署Jenkins Operator

有以下兩種方式部署Jenkins Operator

• 使用yaml一鍵安裝，默認將安裝在default命名空間下

  # kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/all-in-one-v1alpha2.yaml
  # kubectl get pods -w
  

• 使用helm並自定義安裝，依賴helm在v3以上版本

創建ns

# kubectl create ns jenkins

添加helm倉庫並獲取chart

# helm repo add jenkins https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart
"jenkins" has been added to your repositories
# helm pull jenkins/jenkins-operator
# tar xf jenkins-operator-0.4.3.tgz && cd jenkins-operator 

修改value yaml部分內容，可以定義關於jenkins實例、operator deployment、backup備份相關、Configuration配置相關字段

• 指定ns
• 指定插件
• 默認情況只持久化了備份卷，這里將數據卷也做持久化，sc使用csi-rbd-sc
• 默認開啟configurationAsCode，並通過configmap和secret注入

jenkins:
...
  namespace: jenkins
...
  basePlugins:
    - name: kubernetes
      version: "1.28.6"
    - name: workflow-job
      version: "2.40"
    - name: workflow-aggregator
      version: "2.6"
    - name: git
      version: "4.5.0"
    - name: job-dsl
      version: "1.77"
    - name: configuration-as-code
      version: "1.46"
    - name: kubernetes-credentials-provider
      version: "0.15"
  plugins:
    - name: simple-theme-plugin
      version: "0.6"
#  plugins: []
...
  # volumes used by Jenkins
  # By default, we are only using backup
  volumes:
    - name: backup # PVC volume where backups will be stored
      persistentVolumeClaim:
        claimName: jenkins-backup

  # volumeMounts are mounts for Jenkins pod
  volumeMounts: []
...
  backup:
    # enabled is enable/disable switch for backup feature
    # By default the feature is enabled
    enabled: true

    # image used by backup feature
    # By default using prebuilt backup PVC image by VirtusLab
    image: virtuslab/jenkins-operator-backup-pvc:v0.1.0

    # containerName is backup container name
    containerName: backup

    # interval defines how often make backup in seconds
    interval: 30

    # makeBackupBeforePodDeletion when enabled will make backup before pod deletion
    makeBackupBeforePodDeletion: true

    # backupCommand is backup container command
    backupCommand:
      - /home/user/bin/backup.sh

    # restoreCommand is backup restore command
    restoreCommand:
      - /home/user/bin/restore.sh

    getLatestAction:
      - /home/user/bin/get-latest.sh

    # pvc is Persistent Volume Claim Kubernetes resource
    pvc:
      # enabled is enable/disable switch for PVC
      enabled: true

      # size is size of PVC
      size: 5Gi

      # className is storageClassName for PVC
      # See https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1 for more details
      className: "csi-rbd-sc"

    # env contains container environment variables
    # PVC backup provider handles these variables:
    # BACKUP_DIR - path for storing backup files (default: "/backup")
    # JENKINS_HOME - path to jenkins home (default: "/jenkins-home")
    # BACKUP_COUNT - define how much recent backups will be kept
    env:
      - name: BACKUP_DIR
        value: /backup
      - name: JENKINS_HOME
        value: /jenkins-home
      - name: BACKUP_COUNT
        value: "3" # keep only the 3 most recent backups

    # volumeMounts holds the mount points for volumes
    volumeMounts:
      - name: jenkins-home
        mountPath: /jenkins-home # Jenkins home volume
      - mountPath: /backup # backup volume
        name: backup
...
  configuration:
    configurationAsCode: {}
    #    - configMapName: jenkins-casc
    #      content: {}
    groovyScripts: {}
    #    - configMapName: jenkins-gs
    #      content: {}

    # secretRefName of existing secret (previously created)
    secretRefName: ""

    # secretData creates new secret if secretRefName is empty and fills with data provided in secretData
    secretData: {}

執行安裝

# helm install jenkins jenkins-operator -n jenkins --values ./jenkins-operator/values.yaml
NAME: jenkins
LAST DEPLOYED: Sun May 16 19:42:32 2021
NAMESPACE: jenkins
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Watch Jenkins instance being created:
$ kubectl --namespace jenkins get pods -w

2. Get Jenkins credentials:
$ kubectl --namespace jenkins get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.user}' | base64 -d
$ kubectl --namespace jenkins get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.password}' | base64 -d

3. Connect to Jenkins (actual Kubernetes cluster):
$ kubectl --namespace jenkins port-forward jenkins-jenkins 8080:8080

Now open the browser and enter http://localhost:8080

檢查創建的operator

# kubectl get pods -n jenkins
NAME                                       READY   STATUS    RESTARTS   AGE
jenkins-jenkins                            1/2     Running   0          44s
jenkins-jenkins-operator-996887c4b-wftz2   1/1     Running   0          1m29s
# kubectl -n jenkins get jenkins        
NAME      AGE
jenkins   70s

3.4 部署Jenkins

一旦上面的Jenkins Operator部署后啟動並正常運行，就自動會部署一個Jenkins實例Pod了

實際上可以看到，通過Jenkins Operator部署的Jenkins的控制器不是場景k8s自帶的三大控制器，而是由operator自己管控

觀察operator的日志如下

# kubectl -n jenkins logs -f jenkins-jenkins-operator-996887c4b-wftz2
2021-05-16T11:59:05.017Z        INFO    controller-jenkins      jenkins/jenkins_controller.go:432       Setting default Jenkins API settings    {"cr": "jenkins"}
2021-05-16T11:59:05.073Z        INFO    controller-jenkins      jenkins/handler.go:88   *v1alpha2.Jenkins/jenkins has been updated      {"cr": "jenkins"}
2021-05-16T11:59:06.568Z        INFO    controller-jenkins      base/pod.go:159 Creating a new Jenkins Master Pod jenkins/jenkins-jenkins       {"cr": "jenkins"}

觀察jenkins pod中jenkins master的日志如下，正在下載插件（此步驟稍慢）

# kubectl -n jenkins logs -f jenkins-jenkins -c jenkins-master
...
 > bootstrap4-api depends on font-awesome-api:5.15.2-2,jquery3-api:3.5.1-3,popper-api:1.16.1-2
Downloading plugin: font-awesome-api from https://updates.jenkins.io/dynamic-2.263//latest/font-awesome-api.hpi
Downloading plugin: jquery3-api from https://updates.jenkins.io/dynamic-2.263//latest/jquery3-api.hpi
Downloading plugin: popper-api from https://updates.jenkins.io/dynamic-2.263//latest/popper-api.hpi

如果在有限時間（健康檢查時間）內沒有下載成功，這通常是由於網絡原因引起的，Operator會中斷該Pod並重新創建

# kubectl -n jenkins logs -f jenkins-jenkins-operator-996887c4b-wftz2
2021-05-16T12:09:42.854Z        INFO    controller-jenkins      base/reconcile.go:370   Container 'jenkins-master' is terminated, status '{Name:jenkins-master State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:137,Signal:0,Reason:Error,Message:,StartedAt:2021-05-16 12:05:54 +0000 UTC,FinishedAt:2021-05-16 12:09:42 +0000 UTC,ContainerID:docker://342349d9e4045cd312345937797a2d9f048f3623fc1ce6bf1c2b77ff4f04d8da,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:jenkins/jenkins:2.263.2-lts-alpine ImageID:docker-pullable://jenkins/jenkins@sha256:496142509b7d3e3f22f5cdc81b1d1322db61ec929d34dfd66b9ec3257bca13e5 ContainerID:docker://342349d9e4045cd312345937797a2d9f048f3623fc1ce6bf1c2b77ff4f04d8da Started:0xc0005aa2c6}' {"cr": "jenkins"}

可行的一個解決辦法是將value.yaml中的健康檢查時間微調或者臨時去掉健康檢查，並helm更新讓其正常啟動並持久化后再次恢復，或者新創建一個Jenkins控制器將其覆蓋

# helm -n jenkins upgrade jenkins jenkins-operator --values ./jenkins-operator/values.yaml 

最終直到看見這樣的日志，就表示Jenkins啟動成功了

2021-05-16 13:26:14.221+0000 [id=28]    INFO    o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@295b7e33]: org.springframework.beans.factory.support.DefaultListableBeanFactory@52880f75
2021-05-16 13:26:14.223+0000 [id=28]    INFO    o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@52880f75: defining beans [filter,legacy]; root of factory hierarchy
2021-05-16 13:26:14.489+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
2021-05-16 13:26:14.767+0000 [id=20]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running

到這里，通過Jenkins Operator部署Jenkins就完成了（盡管看上去也沒多少比helm或傳統方式部署的優勢），其實Jenkins Operator還有更為好用的的其他功能，后續再介紹。

See you ~