簡單研究下kubesphere 中devops 的使用。
1. 安裝
實際上就是修改安裝過程中的 cluster-configuration.yaml 文件的devops.enable 設為true 即可, kebesphere 會自己下載jenkins，我們通過kubesphere 操作流水線最終也會到jenkins 中操作對應的流水線。
參考: https://kubesphere.io/zh/docs/pluggable-components/devops/
安裝過程中出現的問題: nfs服務器磁盤空間、k8snode01 磁盤空間、k8snode01 內存問題等等。建議在虛擬機初始化就設置的大一點。
1. 安裝成功后查看相關pods、service 如下：
[root@k8smaster01 kubesphere]# kubectl get pods -n kubesphere-devops-system NAME READY STATUS RESTARTS AGE devops-27439710-f4q5f 0/1 Completed 0 8m12s devops-apiserver-7c6774fff5-nmj84 1/1 Running 0 21m devops-controller-98975d478-4xhxn 1/1 Running 0 21m devops-jenkins-64464f495f-4z8br 1/1 Running 1 21m s2ioperator-0 1/1 Running 0 21m [root@k8smaster01 kubesphere]# kubectl get svc -n kubesphere-devops-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE devops-apiserver ClusterIP 10.1.248.197 <none> 9090/TCP 21m devops-jenkins NodePort 10.1.45.197 <none> 80:30180/TCP 21m devops-jenkins-agent ClusterIP 10.1.113.105 <none> 50000/TCP 21m s2ioperator-metrics-service ClusterIP 10.1.161.145 <none> 8080/TCP 21m s2ioperator-trigger-service ClusterIP 10.1.119.55 <none> 8081/TCP 21m webhook-server-service ClusterIP 10.1.46.76 <none> 443/TCP 21m
2. 然后登錄kubesphere 控制台看到如下:
2. 登錄jenkins
上面可以看到service， 30180 端口可以訪問jenkins，在找密碼的時候發現其沒有密碼文件。 最后通過官網發現： 是用kubesphere 的用戶體系進行登錄。
參考: https://v2-1.docs.kubesphere.io/docs/zh-CN/devops/jenkins-setting/
接下來可以簡單的看一下jenkins 控制台界面安裝的一些插件等信息。
2. 使用
接下來就是使用kubesphere 的devops 發布一個簡單的springboot 項目。 這里使用github作為代碼倉庫，阿里鏡像倉庫作為鏡像倉庫。
官網對其運行過程解釋如下：
首先，Jenkins Master 創建一個 Pod 來運行流水線。Kubernetes 創建 Pod 作為 Jenkins Master 的 Agent，該 Pod 會在流水線完成之后銷毀。主要流程包括克隆代碼、構建和推送鏡像以及部署工作負載。
官方提供的兩個java 項目：
https://github.com/kubesphere/devops-java-sample
https://github.com/kubesphere/devops-maven-sample 本次測試使用這個項目進行測試
搭建過程參考：
https://kubesphere.io/docs/devops-user-guide/how-to-use/create-a-pipeline-using-jenkinsfile/
1. fork項目到自己的github 倉庫
如果網絡波動太大的話可以使用國內的gitee，作為代碼倉庫進行測試。我下面使用gitee。
2. 到kubesphere 進行操作
這里用戶都用admin 進行操作，如果想測試kubesphere 的用戶體系可以自己創建多個用戶進行操作。
1. 到訪問控制-》企業空間創建一個企業空間 dev， 用於獨立的測試。這個可以理解為大的一個資源隔離的空間。
一個空間可以有多個普通項目和devops 項目。
普通項目可以用來管理獨立的kubernetes 資源，包括pod、svc 等，實際kubesphere 后台的操作就是新建了一個對應的namespace用於隔離相應資源，namespace 的名稱就是項目名稱；
devops 項目可以包含多個jenkins流水線， 用於devops 操作，kubesphere 后台也創建了一個對應的namespace，namespace名稱為devops 項目名稱加5位隨機字母。
2. dev 企業空間創建一個項目，名稱為 demo。 查看namespace如下
[root@k8smaster01 ~]# kubectl get ns|grep demo demo59wjg Active 26s
3. 在demo 項目下面建立自己的憑證：
1》gitee 的憑證
2》鏡像倉庫的憑證
3》kubeconfig 用於kubernetes 使用
4. 本地復制項目Jenkinsfile-online，命名為Jenkinsfile。 原來文件用於備份，新文件用於jenkins 流水線腳本。修改Jenkinsfile 里面內容的憑證信息，最終如下：
1 pipeline { 2 agent { 3 node { 4 label 'maven' 5 } 6 } 7 8 parameters { 9 string(name:'TAG_NAME',defaultValue: '',description:'') 10 } 11 12 environment { 13 DOCKER_CREDENTIAL_ID = 'ali-registery' 14 GITHUB_CREDENTIAL_ID = 'gitee-secret' 15 KUBECONFIG_CREDENTIAL_ID = 'demo-kubeconfig' 16 REGISTRY = 'registry.cn-hangzhou.aliyuncs.com' 17 DOCKERHUB_NAMESPACE = 'qlq_repository' 18 GITHUB_ACCOUNT = 'qiao-zhi' 19 APP_NAME = 'devops-maven-sample' 20 } 21 22 stages { 23 stage ('checkout scm') { 24 steps { 25 checkout(scm) 26 } 27 } 28 29 stage ('unit test') { 30 steps { 31 container ('maven') { 32 sh 'mvn clean test' 33 } 34 } 35 } 36 37 stage ('build & push') { 38 steps { 39 container ('maven') { 40 sh 'mvn clean package -DskipTests' 41 sh 'docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .' 42 withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) { 43 sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin' 44 sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER' 45 } 46 } 47 } 48 } 49 50 stage('push latest'){ 51 when{ 52 branch 'master' 53 } 54 steps{ 55 container ('maven') { 56 sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest ' 57 sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest ' 58 } 59 } 60 } 61 62 stage('deploy to dev') { 63 when{ 64 branch 'master' 65 } 66 steps { 67 input(id: 'deploy-to-dev', message: 'deploy to dev?') 68 container ('maven') { 69 withCredentials([ 70 kubeconfigFile( 71 credentialsId: env.KUBECONFIG_CREDENTIAL_ID, 72 variable: 'KUBECONFIG') 73 ]) { 74 sh 'envsubst < deploy/dev-all-in-one/devops-sample.yaml | kubectl apply -f -' 75 } 76 } 77 } 78 } 79 stage('push with tag'){ 80 when{ 81 expression{ 82 return params.TAG_NAME =~ /v.*/ 83 } 84 } 85 steps { 86 container ('maven') { 87 input(id: 'release-image-with-tag', message: 'release image with tag?') 88 withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) { 89 sh 'git config --global user.email "kubesphere@yunify.com" ' 90 sh 'git config --global user.name "kubesphere" ' 91 sh 'git tag -a $TAG_NAME -m "$TAG_NAME" ' 92 sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@gitee.com/$GITHUB_ACCOUNT/$APP_NAME.git --tags --ipv4' 93 } 94 sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ' 95 sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ' 96 } 97 } 98 } 99 stage('deploy to production') { 100 when{ 101 expression{ 102 return params.TAG_NAME =~ /v.*/ 103 } 104 } 105 steps { 106 input(id: 'deploy-to-production', message: 'deploy to production?') 107 container ('maven') { 108 withCredentials([ 109 kubeconfigFile( 110 credentialsId: env.KUBECONFIG_CREDENTIAL_ID, 111 variable: 'KUBECONFIG') 112 ]) { 113 sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -' 114 } 115 } 116 } 117 } 118 } 119 }
和原來官網項目文件相比修改的包括environment 中的憑證的id 和 自己的相關信息； 92 行推送tag 時候的相關的github 信息替換為gitee 服務器信息和用$APP_NAME 獲取倉庫名稱。
簡單理解上面相關憑證用於拉代碼、推鏡像、部署到kubernetes。 REGISTRY 是docker 鏡像倉庫地址；DOCKERHUB_NAMESPACE 是鏡像倉庫命名空間；GITHUB_ACCOUNT 用於下面拼接github 代碼倉庫路徑； APP_NAME 是打包名稱和鏡像名稱。
5. 到阿里鏡像倉庫創建新的鏡像倉庫，名稱為 devops-maven-sample(這里創建為公共倉庫，不然拉取鏡像會失敗， 想解決參考： https://developer.aliyun.com/ask/11155?spm=a2c6h.13706215.ask-content.1.3c732a3aUTxxTA)
6. 到kubesphere demo 項目下新建流水線， 名稱為 devops-maven-sample, 然后選擇代碼倉庫，后面默認即可(Jenkinsfile 名稱也匹配)：
7. 點到流水線中，然后點擊掃描項目，獲取git 上面的Jenkinsfile 文件和代碼分支，最終的掃描日志如下：
Started by user admin [Sun Mar 06 03:47:01 UTC 2022] Starting branch indexing... > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git ls-remote --symref -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching & pruning origin... Listing remote references... > git config --get remote.origin.url # timeout=10 > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git ls-remote -h -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching upstream changes from origin > git config --get remote.origin.url # timeout=10 using GIT_ASKPASS to set credentials > git fetch --tags --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10 Checking branches... Checking branch master ‘Jenkinsfile’ found Met criteria No changes detected: master (still at 4b2c785bec6d7c4d2899ec78682b431455529c16) Processed 1 branches [Sun Mar 06 03:47:10 UTC 2022] Finished branch indexing. Indexing took 9.2 sec Finished: SUCCESS
8. 接下來在kubernetes 環境中創建兩個namespace， 不創建namespace 在創建kubernetes 資源的時候會報錯
創建dev 和 prod 環境對應的namespace
kubectl create ns kubesphere-sample-dev
kubectl create ns kubesphere-sample-prod
9. 然后選擇master分支后，點擊運行，輸入參數v1(該參數用於jenkins 流水線腳本進行判斷是否需要推送代碼tag和鏡像tag)運行流水線，等待其結果：(當我們不輸入tag的時候相當於只是部署到開發環境，不打tag等操作)
中間build 過程中，我們全部選擇處理
10. 完成后輸出如下：
查看日志如下：
Started by user admin > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching origin... Fetching upstream changes from origin > git --version # timeout=10 > git --version # 'git version 2.11.0' > git config --get remote.origin.url # timeout=10 using GIT_ASKPASS to set credentials > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10 Seen branch in repository origin/master Seen 1 remote branch Obtained Jenkinsfile from 67401e9a817a371f9da32d5606d591e7b3e0598f Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Still waiting to schedule task ‘maven-2whbc’ is offline Agent maven-2whbc is provisioned from template maven --- apiVersion: "v1" kind: "Pod" metadata: annotations: {} labels: jenkins: "slave" jenkins/label-digest: "f02c587acd12db3d7b4a28edb5c2eae5f526ce28" jenkins/label: "maven" name: "maven-2whbc" spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: "node-role.kubernetes.io/worker" operator: "In" values: - "ci" weight: 1 containers: - args: - "********" - "maven-2whbc" command: - "jenkins-slave" env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "devops-jenkins-agent.kubesphere-devops-system:50000" - name: "JENKINS_AGENT_NAME" value: "maven-2whbc" - name: "JENKINS_NAME" value: "maven-2whbc" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://devops-jenkins.kubesphere-devops-system:80/" image: "jenkins/jnlp-slave:3.27-1" imagePullPolicy: "IfNotPresent" name: "jnlp" resources: limits: memory: "1536Mi" cpu: "500m" requests: memory: "400Mi" cpu: "50m" tty: false volumeMounts: - mountPath: "/root/.sonar/cache" name: "volume-2" readOnly: false - mountPath: "/root/.m2" name: "volume-1" readOnly: false - mountPath: "/var/run/docker.sock" name: "volume-0" readOnly: false - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "cat" image: "kubesphere/builder-maven:v3.2.0" imagePullPolicy: "IfNotPresent" name: "maven" resources: limits: ephemeral-storage: "10Gi" memory: "8192Mi" cpu: "4000m" requests: ephemeral-storage: "1Gi" memory: "100Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/opt/apache-maven-3.5.3/conf/settings.xml" name: "config-volume" subPath: "settings.xml" - mountPath: "/root/.sonar/cache" name: "volume-2" readOnly: false - mountPath: "/root/.m2" name: "volume-1" readOnly: false - mountPath: "/var/run/docker.sock" name: "volume-0" readOnly: false - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false workingDir: "/home/jenkins/agent" nodeSelector: {} restartPolicy: "Never" securityContext: fsGroup: 1000 tolerations: - effect: "NoSchedule" key: "node.kubernetes.io/ci" operator: "Exists" - effect: "PreferNoSchedule" key: "node.kubernetes.io/ci" operator: "Exists" volumes: - hostPath: path: "/var/run/docker.sock" name: "volume-0" - hostPath: path: "/var/data/jenkins_sonar_cache" name: "volume-2" - hostPath: path: "/var/data/jenkins_maven_cache" name: "volume-1" - emptyDir: medium: "" name: "workspace-volume" - configMap: items: - key: "MavenSetting" path: "settings.xml" name: "ks-devops-agent" name: "config-volume" Running on maven-2whbc in /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE using credential gitee-secret Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git init /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master # timeout=10 Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master) > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 Commit message: "v1" > git rev-list --no-walk 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (checkout scm) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE using credential gitee-secret Fetching changes from the remote Git repository Fetching without tags > git rev-parse --is-inside-work-tree # timeout=10 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master) Commit message: "v1" [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (unit test) [Pipeline] container [Pipeline] { [Pipeline] sh > git config core.sparsecheckout # timeout=10 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 + mvn clean test [INFO] Scanning for projects... [INFO] [INFO] -----------------< io.kubesphere.devops:devops-sample >----------------- [INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample --- [INFO] [INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running io.kubesphere.devops.HelloWorldControllerTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.527 s - in io.kubesphere.devops.HelloWorldControllerTest [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15.872 s [INFO] Finished at: 2022-03-06T04:25:53Z [INFO] ------------------------------------------------------------------------ [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (build & push) [Pipeline] container [Pipeline] { [Pipeline] sh + mvn clean package -DskipTests [INFO] Scanning for projects... [INFO] [INFO] -----------------< io.kubesphere.devops:devops-sample >----------------- [INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample --- [INFO] Deleting /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target [INFO] [INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ devops-sample --- [INFO] Building jar: /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/devops-sample-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- spring-boot-maven-plugin:2.1.11.RELEASE:repackage (repackage) @ devops-sample --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14.580 s [INFO] Finished at: 2022-03-06T04:26:16Z [INFO] ------------------------------------------------------------------------ [Pipeline] sh + docker build -f Dockerfile-online -t registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 . Sending build context to Docker daemon 17.52MB Step 1/4 : FROM java:openjdk-8-jre-alpine ---> fdc893b19a14 Step 2/4 : WORKDIR /home ---> Using cache ---> 1e0dc3b20420 Step 3/4 : COPY target/*.jar /home ---> fd0896260f0e Step 4/4 : ENTRYPOINT java -jar *.jar ---> Running in 27d46c26a862 Removing intermediate container 27d46c26a862 ---> 8867ca53b5ca Successfully built 8867ca53b5ca Successfully tagged registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 [Pipeline] withCredentials Masking supported pattern matches of $DOCKER_USERNAME or $DOCKER_PASSWORD [Pipeline] { [Pipeline] sh + echo **** + docker login registry.cn-hangzhou.aliyuncs.com -u **** --password-stdin WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 20dd87a4c2ab: Layer already exists 78075328e0da: Layer already exists 9f8566ee5135: Layer already exists 0e1bd0a16495: Pushed SNAPSHOT-master-3: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (push latest) [Pipeline] container [Pipeline] { [Pipeline] sh + docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 78075328e0da: Layer already exists 20dd87a4c2ab: Layer already exists 9f8566ee5135: Layer already exists 0e1bd0a16495: Layer already exists latest: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (deploy to dev) [Pipeline] input deploy to dev? Proceed or Abort Approved by admin [Pipeline] container [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $KUBECONFIG [Pipeline] { [Pipeline] sh + envsubst + kubectl apply -f - deployment.apps/ks-sample-dev configured service/ks-sample-dev unchanged [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (push with tag) [Pipeline] container [Pipeline] { [Pipeline] input release image with tag? Proceed or Abort Approved by admin [Pipeline] withCredentials Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD [Pipeline] { [Pipeline] sh + git config --global user.email kubesphere@yunify.com [Pipeline] sh + git config --global user.name kubesphere [Pipeline] sh + git tag -a v1 -m v1 [Pipeline] sh + git push http://****:****@gitee.com/****/devops-maven-sample.git --tags --ipv4 remote: Powered by [01;33mGITEE.COM [0m[[01;35mGNK-6.3[0m][0m To http://gitee.com/****/devops-maven-sample.git * [new tag] v1 -> v1 [Pipeline] } [Pipeline] // withCredentials [Pipeline] sh + docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1 [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1 The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 9f8566ee5135: Layer already exists 0e1bd0a16495: Layer already exists 20dd87a4c2ab: Layer already exists 78075328e0da: Layer already exists v1: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (deploy to production) [Pipeline] input deploy to production? Proceed or Abort Approved by admin [Pipeline] container [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $KUBECONFIG [Pipeline] { [Pipeline] sh + envsubst + kubectl apply -f - deployment.apps/ks-sample created service/ks-sample created [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
1》jenkins 查看日志如下
日志如下：
Started by user admin > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching origin... Fetching upstream changes from origin > git --version # timeout=10 > git --version # 'git version 2.11.0' > git config --get remote.origin.url # timeout=10 using GIT_ASKPASS to set credentials > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10 Seen branch in repository origin/master Seen 1 remote branch Obtained Jenkinsfile from 67401e9a817a371f9da32d5606d591e7b3e0598f Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Still waiting to schedule task ‘maven-2whbc’ is offline Agent maven-2whbc is provisioned from template maven --- apiVersion: "v1" kind: "Pod" metadata: annotations: {} labels: jenkins: "slave" jenkins/label-digest: "f02c587acd12db3d7b4a28edb5c2eae5f526ce28" jenkins/label: "maven" name: "maven-2whbc" spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: "node-role.kubernetes.io/worker" operator: "In" values: - "ci" weight: 1 containers: - args: - "********" - "maven-2whbc" command: - "jenkins-slave" env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "devops-jenkins-agent.kubesphere-devops-system:50000" - name: "JENKINS_AGENT_NAME" value: "maven-2whbc" - name: "JENKINS_NAME" value: "maven-2whbc" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://devops-jenkins.kubesphere-devops-system:80/" image: "jenkins/jnlp-slave:3.27-1" imagePullPolicy: "IfNotPresent" name: "jnlp" resources: limits: memory: "1536Mi" cpu: "500m" requests: memory: "400Mi" cpu: "50m" tty: false volumeMounts: - mountPath: "/root/.sonar/cache" name: "volume-2" readOnly: false - mountPath: "/root/.m2" name: "volume-1" readOnly: false - mountPath: "/var/run/docker.sock" name: "volume-0" readOnly: false - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - command: - "cat" image: "kubesphere/builder-maven:v3.2.0" imagePullPolicy: "IfNotPresent" name: "maven" resources: limits: ephemeral-storage: "10Gi" memory: "8192Mi" cpu: "4000m" requests: ephemeral-storage: "1Gi" memory: "100Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/opt/apache-maven-3.5.3/conf/settings.xml" name: "config-volume" subPath: "settings.xml" - mountPath: "/root/.sonar/cache" name: "volume-2" readOnly: false - mountPath: "/root/.m2" name: "volume-1" readOnly: false - mountPath: "/var/run/docker.sock" name: "volume-0" readOnly: false - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false workingDir: "/home/jenkins/agent" nodeSelector: {} restartPolicy: "Never" securityContext: fsGroup: 1000 tolerations: - effect: "NoSchedule" key: "node.kubernetes.io/ci" operator: "Exists" - effect: "PreferNoSchedule" key: "node.kubernetes.io/ci" operator: "Exists" volumes: - hostPath: path: "/var/run/docker.sock" name: "volume-0" - hostPath: path: "/var/data/jenkins_sonar_cache" name: "volume-2" - hostPath: path: "/var/data/jenkins_maven_cache" name: "volume-1" - emptyDir: medium: "" name: "workspace-volume" - configMap: items: - key: "MavenSetting" path: "settings.xml" name: "ks-devops-agent" name: "config-volume" Running on maven-2whbc in /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE using credential gitee-secret Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git init /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master # timeout=10 Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master) > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 Commit message: "v1" > git rev-list --no-walk 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (checkout scm) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE using credential gitee-secret Fetching changes from the remote Git repository Fetching without tags > git rev-parse --is-inside-work-tree # timeout=10 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10 Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git > git --version # timeout=10 > git --version # 'git version 2.11.0' using GIT_ASKPASS to set credentials > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master) Commit message: "v1" [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (unit test) [Pipeline] container [Pipeline] { [Pipeline] sh > git config core.sparsecheckout # timeout=10 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10 + mvn clean test [INFO] Scanning for projects... [INFO] [INFO] -----------------< io.kubesphere.devops:devops-sample >----------------- [INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample --- [INFO] [INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running io.kubesphere.devops.HelloWorldControllerTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.527 s - in io.kubesphere.devops.HelloWorldControllerTest [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15.872 s [INFO] Finished at: 2022-03-06T04:25:53Z [INFO] ------------------------------------------------------------------------ [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (build & push) [Pipeline] container [Pipeline] { [Pipeline] sh + mvn clean package -DskipTests [INFO] Scanning for projects... [INFO] [INFO] -----------------< io.kubesphere.devops:devops-sample >----------------- [INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample --- [INFO] Deleting /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target [INFO] [INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample --- [INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true [INFO] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes [INFO] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ devops-sample --- [INFO] Building jar: /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/devops-sample-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- spring-boot-maven-plugin:2.1.11.RELEASE:repackage (repackage) @ devops-sample --- [INFO] Replacing main artifact with repackaged archive [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14.580 s [INFO] Finished at: 2022-03-06T04:26:16Z [INFO] ------------------------------------------------------------------------ [Pipeline] sh + docker build -f Dockerfile-online -t registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 . Sending build context to Docker daemon 17.52MB Step 1/4 : FROM java:openjdk-8-jre-alpine ---> fdc893b19a14 Step 2/4 : WORKDIR /home ---> Using cache ---> 1e0dc3b20420 Step 3/4 : COPY target/*.jar /home ---> fd0896260f0e Step 4/4 : ENTRYPOINT java -jar *.jar ---> Running in 27d46c26a862 Removing intermediate container 27d46c26a862 ---> 8867ca53b5ca Successfully built 8867ca53b5ca Successfully tagged registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 [Pipeline] withCredentials Masking supported pattern matches of $DOCKER_USERNAME or $DOCKER_PASSWORD [Pipeline] { [Pipeline] sh + echo **** + docker login registry.cn-hangzhou.aliyuncs.com -u **** --password-stdin WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 20dd87a4c2ab: Layer already exists 78075328e0da: Layer already exists 9f8566ee5135: Layer already exists 0e1bd0a16495: Pushed SNAPSHOT-master-3: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (push latest) [Pipeline] container [Pipeline] { [Pipeline] sh + docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 78075328e0da: Layer already exists 20dd87a4c2ab: Layer already exists 9f8566ee5135: Layer already exists 0e1bd0a16495: Layer already exists latest: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (deploy to dev) [Pipeline] input deploy to dev? Proceed or Abort Approved by admin [Pipeline] container [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $KUBECONFIG [Pipeline] { [Pipeline] sh + envsubst + kubectl apply -f - deployment.apps/ks-sample-dev configured service/ks-sample-dev unchanged [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (push with tag) [Pipeline] container [Pipeline] { [Pipeline] input release image with tag? Proceed or Abort Approved by admin [Pipeline] withCredentials Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD [Pipeline] { [Pipeline] sh + git config --global user.email kubesphere@yunify.com [Pipeline] sh + git config --global user.name kubesphere [Pipeline] sh + git tag -a v1 -m v1 [Pipeline] sh + git push http://****:****@gitee.com/****/devops-maven-sample.git --tags --ipv4 remote: Powered by [01;33mGITEE.COM [0m[[01;35mGNK-6.3[0m][0m To http://gitee.com/****/devops-maven-sample.git * [new tag] v1 -> v1 [Pipeline] } [Pipeline] // withCredentials [Pipeline] sh + docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1 [Pipeline] sh + docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1 The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample] 0e1bd0a16495: Preparing 20dd87a4c2ab: Preparing 78075328e0da: Preparing 9f8566ee5135: Preparing 9f8566ee5135: Layer already exists 0e1bd0a16495: Layer already exists 20dd87a4c2ab: Layer already exists 78075328e0da: Layer already exists v1: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159 [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (deploy to production) [Pipeline] input deploy to production? Proceed or Abort Approved by admin [Pipeline] container [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $KUBECONFIG [Pipeline] { [Pipeline] sh + envsubst + kubectl apply -f - deployment.apps/ks-sample created service/ks-sample created [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
2》 kubernetes 環境查看相關的資源
[root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-devops-dev No resources found in kubesphere-devops-dev namespace. [root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-sample-dev NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ks-sample-dev 1/1 1 1 12h NAME READY STATUS RESTARTS AGE pod/ks-sample-dev-5dc9786dcc-6f9fx 1/1 Running 0 5m48s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ks-sample-dev NodePort 10.1.66.88 <none> 8080:30861/TCP 14h [root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-sample-prod NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ks-sample 2/2 2 2 4m57s NAME READY STATUS RESTARTS AGE pod/ks-sample-68b68dfbc6-dpkwk 1/1 Running 0 4m57s pod/ks-sample-68b68dfbc6-qs2sw 1/1 Running 0 4m57s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ks-sample NodePort 10.1.21.72 <none> 8080:30961/TCP 4m57s
3》gitee 查看打的tag 信息
4》阿里鏡像倉庫查看版本信息如下：
5》測試訪問
xx@xx MINGW64 /d/study/devops-maven-sample (master) $ curl http://192.168.13.107:30861 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2 100 2 0 0 166 0 --:--:-- --:--:-- --:--:-- 181v1 xx@xx MINGW64 /d/study/devops-maven-sample (master) $ curl http://192.168.13.107:30961 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2 100 2 0 0 68 0 --:--:-- --:--:-- --:--:-- 71v1
11. 總結
(0) 前置
其實通過kubesphere 流水線操作，最終還是都會到Jenkins中進行操作，其操作和自定義流水線一樣。大致就是打代碼、測試代碼、打包、制作鏡像並打tag、傳送到鏡像倉庫、創建kubernetes 相關資源。在jenkins 執行過程中，會創建一個代理pod，比如：
[root@k8smaster01 ~]# kubectl get pods -A| grep maven kubesphere-devops-worker maven-cfqfb 2/2 Running 0 47s kubesphere-devops-worker maven-gzjdm 1/2 Error 0 18h
其中git 拉取代碼、maven打包都是在下面pods 操作的。進入一個pod，查看其環境如下：
(1)Jenkins file文件查看分析
pipeline { agent { node { label 'maven' } } parameters { string(name:'TAG_NAME',defaultValue: '',description:'') } environment { DOCKER_CREDENTIAL_ID = 'ali-registery' GITHUB_CREDENTIAL_ID = 'gitee-secret' KUBECONFIG_CREDENTIAL_ID = 'demo-kubeconfig' REGISTRY = 'registry.cn-hangzhou.aliyuncs.com' DOCKERHUB_NAMESPACE = 'qlq_repository' GITHUB_ACCOUNT = 'qiao-zhi' APP_NAME = 'devops-maven-sample' } stages { stage ('checkout scm') { steps { checkout(scm) } } stage ('unit test') { steps { container ('maven') { sh 'mvn clean test' } } } stage ('build & push') { steps { container ('maven') { sh 'mvn clean package -DskipTests' sh 'docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .' withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) { sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin' sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER' } } } } stage('push latest'){ when{ branch 'master' } steps{ container ('maven') { sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest ' sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest ' } } } stage('deploy to dev') { when{ branch 'master' } steps { input(id: 'deploy-to-dev', message: 'deploy to dev?') container ('maven') { withCredentials([ kubeconfigFile( credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG') ]) { sh 'envsubst < deploy/dev-all-in-one/devops-sample.yaml | kubectl apply -f -' } } } } stage('push with tag'){ when{ expression{ return params.TAG_NAME =~ /v.*/ } } steps { container ('maven') { input(id: 'release-image-with-tag', message: 'release image with tag?') withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) { sh 'git config --global user.email "kubesphere@yunify.com" ' sh 'git config --global user.name "kubesphere" ' sh 'git tag -a $TAG_NAME -m "$TAG_NAME" ' sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@gitee.com/$GITHUB_ACCOUNT/$APP_NAME.git --tags --ipv4' } sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ' sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME ' } } } stage('deploy to production') { when{ expression{ return params.TAG_NAME =~ /v.*/ } } steps { input(id: 'deploy-to-production', message: 'deploy to production?') container ('maven') { withCredentials([ kubeconfigFile( credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG') ]) { sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -' } } } } } }
前面定義了一些全局環境變量，按階段進行分析
1》checkout scm 拉取代碼
2》unit test 執行單元測試
3》build & push: 執行mvn打包， 然后基於Dockerfile-online 構造本地鏡像，鏡像的版本用代碼分支和buildnumber 做區分，然后將鏡像推送代碼鏡像倉庫
4》push latest： 當代碼分支是master 的時候，將3》的鏡像打包為latest版本的鏡像，然后推到鏡像倉庫
5》deploy to dev：當代碼分支是master 的時候，相當於是執行 deploy/dev-all-in-one/devops-sample.yaml文件，相當於到kubernetes 創建資源
6》push with tag：當我們傳遞的參數是vxxx的時候，將代碼打上相應的標簽並推送到代碼倉庫；將鏡像也打上相應的標簽並推送到相應的鏡像倉庫
7》deploy to production：當我們傳遞的參數是vxxx的時候， 需要我們驗證下是否需要將相關資源創建到kubernetes環境的prod 倉庫，使用的文件是deploy/prod-all-in-one/devops-sample.yaml。
envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -
envsubst 是用shell格式字符串中的值替換環境變量。要替換的變量應位於${var}或$var格式。例如
[root@k8smaster01 ~]# export MYKEY=123456 [root@k8smaster01 ~]# echo $MYKEY 123456 [root@k8smaster01 ~]# cat test.txt $MYKEY [root@k8smaster01 ~]# envsubst < test.txt 123456
(2) 幾個重要kubernetes 文件如下：
deploy/dev-all-in-one/devops-sample.yaml：
--- apiVersion: apps/v1 kind: Deployment metadata: labels: app: kubesphere component: ks-sample-dev tier: backend name: ks-sample-dev namespace: kubesphere-sample-dev spec: progressDeadlineSeconds: 600 replicas: 1 selector: matchLabels: app: kubesphere component: ks-sample-dev tier: backend template: metadata: labels: app: kubesphere component: ks-sample-dev tier: backend spec: containers: - env: - name: CACHE_IGNORE value: js|html - name: CACHE_PUBLIC_EXPIRATION value: 3d image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER readinessProbe: httpGet: path: / port: 8080 timeoutSeconds: 10 failureThreshold: 30 periodSeconds: 5 imagePullPolicy: Always name: ks-sample ports: - containerPort: 8080 protocol: TCP resources: limits: cpu: 300m memory: 600Mi requests: cpu: 100m memory: 100Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always terminationGracePeriodSeconds: 30 --- apiVersion: v1 kind: Service metadata: labels: app: kubesphere component: ks-sample-dev name: ks-sample-dev namespace: kubesphere-sample-dev spec: ports: - name: http port: 8080 protocol: TCP targetPort: 8080 nodePort: 30861 selector: app: kubesphere component: ks-sample-dev tier: backend sessionAffinity: None type: NodePort
deploy/prod-all-in-one/devops-sample.yaml
--- apiVersion: apps/v1 kind: Deployment metadata: labels: app: kubesphere component: ks-sample tier: backend name: ks-sample namespace: kubesphere-sample-prod spec: progressDeadlineSeconds: 600 replicas: 2 selector: matchLabels: app: kubesphere component: ks-sample tier: backend strategy: rollingUpdate: maxSurge: 100% maxUnavailable: 100% type: RollingUpdate template: metadata: labels: app: kubesphere component: ks-sample tier: backend spec: containers: - env: - name: CACHE_IGNORE value: js|html - name: CACHE_PUBLIC_EXPIRATION value: 3d image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME readinessProbe: httpGet: path: / port: 8080 timeoutSeconds: 10 failureThreshold: 30 periodSeconds: 5 imagePullPolicy: Always name: ks ports: - containerPort: 8080 protocol: TCP resources: limits: cpu: 300m memory: 600Mi requests: cpu: 100m memory: 100Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always terminationGracePeriodSeconds: 30 --- apiVersion: v1 kind: Service metadata: labels: app: kubesphere component: ks-sample name: ks-sample namespace: kubesphere-sample-prod spec: ports: - name: http port: 8080 protocol: TCP targetPort: 8080 nodePort: 30961 selector: app: kubesphere component: ks-sample tier: backend sessionAffinity: None type: NodePort
代碼倉庫和修改后配置文件：https://gitee.com/Qiao-Zhi/devops-maven-sample
補充： 在安裝過程中遇到一些磁盤不足，報錯信息如下:
Warning Evicted 49m kubelet The node had condition: [DiskPressure].
解決參考：
https://blog.51cto.com/riverxyz/2758421
vm中對centos 擴容: https://ld246.com/article/1566021346577