[k8s]jenkins配合kubernetes插件實現k8s集群構建的持續集成

另一個結合harbor自動構建鏡像的思路: 即code+baseimage一體的方案

- 程序員將代碼提交到代碼倉庫gitlab
- 鈎子觸發jenkins master啟動一次構建
- jenkins master從k8s申請一個jenkins slave編譯容器
- 在容器內編譯完成以后，獲得最終產物
- 將最終產物通過dockerfile生成生產部署鏡像（這里省略了測試，其實部署鏡像需要測試通過）
- 將生產鏡像推送到harbor鏡像倉庫
- jenkins slave生命周期結束，k8s銷毀slave容器
- 一次構建完成

k8s持續集成的一個思路:

這里要說的是部署部分

注: 這只是一個持續集成思想.本篇按照這個思想來搞,在我的環境里我為了速度快當然還有別的因素我用這種方案,用的很6.
其他思路:

• 可以將code+image打在一起做升級
• 可以rbac+環境+ns+supervisor 每個開發一個環境這樣搞
  后面我抽空一一實現下.

這篇文章思路:

手動構建war包(集成測試)-->本地tomcat測試通過(功能測試)-->k8s容器化tomcat(pv+deploy+svc+ingress)-->將war包拖入k8s的tomcat測試.

jenkins jnlp鏡像構建(mvn+git+kubectl)-> jnlp鏡像測試,確保可被server動態調度-->配置war包的pipeline測試.

注: 本篇jenkins server部署在vm上,非docker部署,jenkins-jnlp-slave是容器化自動創建的.
其他內容參考: 容器ci索引: http://www.cnblogs.com/iiiiher/p/8026689.html

構建jnlp鏡像的dockerfile

• 准備dockerfile所需文件

git clone https://github.com/jenkinsci/docker-jnlp-slave.git
cd docker-jnlp-slave

$ ls
Dockerfile  jenkins-slave  kubectl  README.md

構建mvn3.5.2+git+kubectl的鏡像

基於jenkinsci/slave:alpine的基礎鏡像
參考: https://github.com/jenkinsci/docker-slave/blob/master/Dockerfile
https://github.com/jenkinsci/docker-jnlp-slave/blob/master/Dockerfile
https://hub.docker.com/r/jenkinsci/slave/tags/

alpine-git安裝參考:
https://hub.docker.com/r/alpine/git/~/dockerfile/

$ cat Dockerfile 
FROM jenkinsci/slave:alpine

USER root
RUN apk add --no-cache curl tar bash

## Install Maven
ARG MAVEN_VERSION=3.5.2
ARG USER_HOME_DIR="/root"
ARG SHA=707b1f6e390a65bde4af4cdaf2a24d45fc19a6ded00fff02e91626e3e42ceaff
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries

RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
  && curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
  && echo "${SHA}  /tmp/apache-maven.tar.gz" | sha256sum -c - \
  && tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
  && rm -f /tmp/apache-maven.tar.gz \
  && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn \
  && apk --update add git openssh \
  && rm -rf /var/lib/apt/lists/* \
  && rm /var/cache/apk/* \
  && mkdir /src /target \
  && chown jenkins.jenkins /src /target

ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"

# install kubectl
COPY kubectl /usr/local/bin/kubectl

## install jenkins-slave
COPY jenkins-slave /usr/local/bin/jenkins-slave
USER jenkins
WORKDIR /home/jenkins
ENTRYPOINT ["jenkins-slave"]

鏡像已可以從dockerhub下載:

docker pull lanny/mvn-git-kubectl-jnlp:3.5.2

測試jnlp鏡像

主要看他能否用jenkins-server動態調用跑起來
當然首先安裝jenkins kubernetes插件,新建一朵雲:
參考: http://www.cnblogs.com/iiiiher/p/7979336.html

配置項目: 選擇pipeline script

podTemplate(name: 'maotai-dev', cloud: 'kubernetes',
  namespace: 'kube-public', label: 'maotai-dev',
  serviceAccount: 'default', containers: [
  containerTemplate(
      name: 'jnlp',
      image: 'lanny/mvn-git-kubectl-jnlp:3.5.2',
      args: '${computer.jnlpmac} ${computer.name}',
      ttyEnabled: true,
      privileged: false,
      alwaysPullImage: false)
  ],
) {

  node('maotai-dev') {
    stage('git-clone') {
      container('jnlp') {
          sh """
          date +%F;
          sleep 30;
          """
      }
    }
  }
}

點擊構建-->顯示構建成功

構建成功后jnlp鏡像隨着構建結束自動刪除.

tomcat java-helloworld項目

kubernetes插件的pipeline使用:
參考:
https://github.com/jenkinsci/kubernetes-plugin
https://help.aliyun.com/document_detail/56336.html?spm=5176.doc56336.6.851.wAqCzu

javahelloworld代碼: https://github.com/lannyMa/trucks ,構建可形成helloworld的war包.可以部署在tomcat用於測試.

jenkins項目配置: 新建項目 test-pipeline

podTemplate(name: 'maotai-dev', cloud: 'kubernetes',
  namespace: 'kube-public', label: 'maotai-dev',
  serviceAccount: 'default', containers: [
  containerTemplate(
      name: 'jnlp',
      image: 'lanny/mvn-git-kubectl-jnlp:3.5.2',
      args: '${computer.jnlpmac} ${computer.name}',
      ttyEnabled: true,
      privileged: false,
      alwaysPullImage: false)
  ],
  volumes: [
    persistentVolumeClaim(mountPath: '/tmp/', claimName: 'spring-pvc')
  ]) {
  node('maotai-dev') {
    stage('git-clone') {
      container('jnlp') {
          sh """
          git clone https://github.com/lannyMa/trucks.git
          """
      }
    }

    stage('mvn-package') {
      container('jnlp') {
          sh """
          cd trucks && mvn clean package && cp -rpf target/*.war /tmp/
          """
      }
    }

    stage('restart') {
      container('jnlp') {
          sh """
          pod_name=`kubectl -s 192.168.x.x:8080 -n kube-public get pods -l name=maotai-dev -o name | cut -d"/" -f2`
          kubectl -s kube-apiserver-http.kube-public -n kube-public delete pod \$pod_name
          """
      }
    }
  }
}

配置tomcat項目

tomcat-pvc.yaml #前提是配置好stroragecalss: 參考: http://www.cnblogs.com/iiiiher/p/7988803.html

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: spring-pvc
  namespace: kube-public
spec:
  storageClassName: "managed-nfs-storage"
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 100Mi

tomcat-deploy.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: spring
  namespace: kube-public
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: spring
        name: maotai-dev  #這里標簽設置需注意,因為jenkins配置kubectl的stage時需要根據標簽過濾重啟它: kubectl -s 192.168.x.x:8080 -n kube-public get pods -l name=spring -o name | cut -d"/" -f2
    spec:
      containers:
      - name: spring
        image: tomcat:latest
        imagePullPolicy: IfNotPresent
        ports:
        - name: web
          containerPort: 8080
        volumeMounts:
        - mountPath: /usr/local/tomcat/webapps
          name: spring-folder
      volumes:
      - name: spring-folder
        persistentVolumeClaim:
          claimName: spring-pvc

執行成功:

k8s集群容器化tomcat項目

• 容器化tomcat項目: 配置k8s集群的tomcat 包含了 pvc+deploy+svc+ingress
• 做法:
  • 先手動編譯項目,本次tomcat測試通過
  • 集成到k8s集群的tomcat,測試,確保項目可以正常運行

tomcat-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: spring
  namespace: kube-public
  labels:
    name: spring
spec:
  ports:
    - name: web
      port: 8080
      targetPort: web
  selector:
    name: spring

tomcat-ingress.yaml #前提是已配置好了ingress,nginx-ingress配置參考:http://www.cnblogs.com/iiiiher/p/8006801.html

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: spring
  namespace: kube-public
spec:
  rules:
  - host: spring.maotai.net
    http:
      paths:
      - path: /
        backend:
          serviceName: spring
          servicePort: web

創建好后確保能夠訪問:

接下來需要手動編譯,確保項目通過集成測試(可成功編譯),功能測試(部署tomcat后可訪問)

項目代碼: https://github.com/lannyMa/trucks.git
mvn配置改源等參考: https://github.com/lannyMa/java-helloword.git

確保沒問題后將war包放到上一步創建的pv里.我的是nfs,直接到nfs-server上把war包托上去,然后重啟tomcat,測試效果.