Kubernetes（十三） CI／CD（1）

本文轉載自查看原文 2020-02-22 13:41 906 kubernetes/ CI/CD

本文通過在kubernetes上啟動Jenkins服務，並將宿主機上的docker、docker.sock掛載到Jenkins容器中，實現在Jenkins容器中直接打鏡像的形式實現CI功能。

Kubernetes 集群的安裝請參考kubernetes安裝

部署Jenkins

下載Jenkins對應的鏡像
```
docker pull jenkins/jenkins:2.221
```

將jenkins鏡像上傳到自己的私有鏡像倉庫中

docker tag jenkins/jenkins:2.221 192.168.0.107/k8s/jenkins:2.221

docker push 	192.168.0.107/k8s/jenkins:2.221

編寫啟動Jenkins的yml文件

cat > jenkins.yml << EOF
kind: PersistentVolume
apiVersion: v1
metadata:
  name: jenkins
  labels:
    type: local
    app: jenkins
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /opt/k8s/yml/jenkins/data
---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: default
automountServiceAccountToken: true
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: Jenkins-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: default
---

apiVersion: v1
kind: Service
metadata:
  name: jenkins
  labels:
    app: jenkins
spec:
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 8888
      name: jenkins
    - port: 50000
      targetPort: 50000
      nodePort: 50000
      name: agent
  selector:
    app: jenkins
    tier: jenkins
  type: NodePort
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  labels:
    app: jenkins
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: jenkins
      tier: jenkins
  template:
    metadata:
      labels:
        app: jenkins
        tier: jenkins
    spec:
      serviceAccountName: jenkins
      containers:
      - image: 192.168.0.107/k8s/jenkins:2.221
        imagePullPolicy: IfNotPresent
        name: jenkins
        securityContext:
          privileged: true
          runAsUser: 0
        volumeMounts:
        - name: kubeconfig
          mountPath: /var/jenkins_home/.kube
        - name: docker
          mountPath: /var/run/docker.sock
        - name: docker-bin
          mountPath: /usr/bin/docker
        - name: jenkins-persistent-storage
          mountPath: /var/jenkins_home
        ports:
        - containerPort: 8080
          name: jenkins
        - containerPort: 50000
        	name: agent
      volumes:
      - name: kubeconfig
        emptyDir: {}
      - name: docker
        hostPath:
          path: /var/run/docker.sock
      - name: docker-bin
        hostPath:
          path: /opt/k8s/bin/docker
      - name: jenkins-persistent-storage
        persistentVolumeClaim:
          claimName: jenkins-claim
EOF

其中ServiceAccount和ClusterRoleBinding是為了后續步驟在kubernets集群中啟動pod完成構建任務而創建
為了能在Jenkins容器內部直接使用dokcer命令打鏡像，此處直接將宿主機上的docker命令以及docker.sock掛載到Jenkins中，沒有在Jenkins容器中進行docker-ce的安裝
Jenkins容器設置成特權用戶執行，並把執行用戶Id設置成0（root用戶），原因參看遇到問題一節

啟動Jenkins

mkdir -p /opt/k8s/yml/jenkins/data
chmod -R 777 /opt/k8s/yml/jenkins/data
kubectl create -f jenkins.yml

啟動后首次登陸密碼可在日志中查看，或通過如下命令獲取

kubectl exec -it `kubectl get pods  --selector=app=jenkins --output=jsonpath={.items..metadata.name}` cat /var/jenkins_home/secrets/initialAdminPassword

安裝插件

安裝：git-parameter、git-client、git、pipeline相關插件,可在jenkins插件管理界面上選擇安裝，如果下載失敗，可以查看對應軟件的版本從https://mirrors.tuna.tsinghua.edu.cn/jenkins/plugins下載后放到Jenkins工作目錄下的plugins目錄下。

驗證Jenkins

創建git工程，在gitlab上創建一個簡單的hello-ci工程，功能是：基於nginx鏡像打一個自己的鏡像，替換其中的歡迎頁 index.html

工程代碼

index.html

<html>
    <p><h2 style="font-family:sans-serif">Hello from ci! You've successfully built and run the Hello-ci app.</h2> </p>
    <p style="font-family:sans-serif">The Hello-ci app is a modified version of the <a href="https://hub.docker.com/_/nginx/">nginx web server image</a>. If you open up the <b>kubernetes-ci-cd/hello-ci/DockerFile</b>, you will note several things:</p>
    <p style="font-family:sans-serif">welcome to ci </p>
</html>

Dockerfile

FROM 192.168.0.107/k8s/nginx:1.9.1

COPY index.html /usr/share/nginx/html/index.html

EXPOSE 80

Jenkinsfile

node {


	    properties([parameters([[$class: 'GitParameterDefinition', branch: '', branchFilter: '.*', defaultValue: '', description: '', name: 'release_version', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH_TAG']])])
	
	    checkout scm
	    
	    stage ("edit parameters") {
	    
	        echo "release_version:${release_version}"
	        
	        real_version = release_version.replaceAll("origin/","")
	        
	        echo "real_version:${real_version}"
	    
	    }
	    
	    imageName = "192.168.0.107/k8s/hello-ci:${real_version}"
	    
	    
	    stage ("docker login") {
	        sh "docker login -u admin -p Harbor12345 192.168.0.107"
	    }
	    
	    stage ("Build") {
	    
	        sh "docker build -t ${imageName} application"
	    }
	    stage ("Push") {
	
	        sh "docker push ${imageName}"
	    }
	    
	   
	
	}

采用腳本形式編輯pipeline，也可以采用聲明方式
properties屬性指定此Jenkins工程是參數化構建，構建參數是 branch或者是tag
根據選定的分支或tag決定打出的鏡像對應的版本號

創建Jenkins工程
在Jenkins界面新建一個item，名稱hello-pipeline，類型選擇:流水線(pipeline)

配置工程為參數化構建，參數列表中選擇gitparameter,類型為branchortag
- 此步驟也可不執行，首次執行工程后Jenkins會根據Jenkinsfile中的內容自動把工程變成參數化構建，但是這樣第一次就不能選擇對應的版本，所以此處加了一個配置
設置工程路徑，對應的腳本路徑

執行構建，選擇一個分支，以master為例，執行構建

構建日志

Started by user admin
Lightweight checkout support not available, falling back to full checkout.
Checking out git http://192.168.0.107:9090/ci-cd/hello-ci.git into /var/jenkins_home/workspace/hello-pipeline@script to read application/Jenkinsfile
Cloning the remote Git repository

 ...
 
[Pipeline] Start of Pipeline

...

[Pipeline] stage
[Pipeline] { (edit parameters)
[Pipeline] echo
release_version:origin/master
[Pipeline] echo
real_version:master
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (docker login)
[Pipeline] sh
+ docker login -u admin -p Harbor12345 192.168.0.107
...

Login Succeeded
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] sh
+ docker build -t 192.168.0.107/k8s/hello-ci:master application
...

Successfully built b2b4f45901a6
Successfully tagged 192.168.0.107/k8s/hello-ci:master
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Push)
[Pipeline] sh
+ docker push 192.168.0.107/k8s/hello-ci:master
The push refers to repository [192.168.0.107/k8s/hello-ci]

...

4fc9a49e07e9: Pushed
master: digest: sha256:a90710b35388915d2b01dfc6173da996f8191be2a850b9c8453534e85c91a7f9 size: 3012
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

可以看到根據我們選擇的master分枝，打出來了一個192.168.0.107/k8s/hello-ci:master的鏡像

驗證構建好的鏡像文件

編寫啟動鏡像的文件

cat > hello-ci.yml << EOF
apiVersion: v1
kind: Service
metadata:
  name: hello-ci
  labels:
    app: hello-ci
spec:
  type: NodePort
  selector:
    app: hello-ci
  ports:
  - name: http
    port: 8089
    targetPort: 80
    nodePort: 8089
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-ci-deployment
spec:
  selector:
    matchLabels:
      app: hello-ci
  replicas: 1
  template:
    metadata:
      labels:
        app: hello-ci
    spec:
      containers:
      - name: hello-ci
        image: 192.168.0.107/k8s/hello-ci:master
        ports:
        - containerPort: 80
EOF

啟動hello-ci
```
kubectl create -f hello-ci.yml
```
訪問界面

向Jenkins中追加slave node

配置slave節點（在slave節點上執行）

為Jenkins執行用戶（按照本文創建的Jenkins用戶是root（runAsUser: 0 配置）生成可信賴的認證key，（如果已經生成過，可以直接拿來用）
```
cd ~/.ssh
ssh-keygen -t rsa -C "admin@example.com"

cat id_rsa.pub >> authorized_keys

chmod 700 authorized_keys

service sshd restart
```
在Jenkins界面

Manage Jenkins -> Manage Nodes -> New Node追加一個node

點擊OK后進入node配置界面

Credentials追加
保存后Jenkins就會自動的launch 對應的slave，並檢查節點上的環境，如是否有java，如果沒有就嘗試去下載安裝（因為現在oracle下載jdk需要登陸，此步驟不會自動成功，所以需要提前在slave節點上安裝好jdk工具）

遇到問題

長時間處於Please wait while Jenkins is getting ready to work ...

修改hudson.model.UpdateCenter.xml文件


# 刪除
#https://updates.jenkins-ci.org/update-center.json

#追加
<url>https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json</url>

jenkinsfile中執行 sh "docker ..."相關命令時出錯
```
dial unix /var/run/docker.sock: connect: permission denied
```
因為宿主機上docker是以root身份啟動的，每次啟動docker服務會生成docker.sock，此時docker.sock默認對應的用戶和用戶組都是root的
```
root@master:/opt/k8s/yml/jenkins# ls -l /var/run/docker.sock
```

srw-rw---- 1 root root 0 2月 19 20:11 /var/run/docker.sock

```

而默認的Jenkins鏡像是以jenkins用戶執行

```
root@master:/opt/k8s/yml/jenkins/data/plugins# kubectl exec -it jenkins-798d66fc78-x9zbr bash

jenkins@jenkins-798d66fc78-x9zbr:/$ 

```

所以不具有訪問/var/run/docker.sock的權限，解決方法是在jenkins對應的container的securityContext屬性中追加<font color=red>runAsUser: 0 </font>配置

jenkinsfile執行中docker pull和docker push鏡像時沒有權限訪問私有倉庫，
```
docker push 192.168.0.107/k8s/hello-ci:v1.0.0
...
denied: requested access to the resource is denied
```
可以先在宿主機上執行好docker login,然后把認證后的/root/.docker/config.json掛載到jenkins容器中，或者在Jenkinsfile中追加docker login的步驟
service 和容器沒有啟動 50000端口，而agent和master之間通信用的是這個端口，造成一直出錯,
```
SEVERE: http://192.168.0.107:8888/ provided port:50000 is not reachable
```

java.io.IOException: http://192.168.0.107:8888/ provided port:50000 is not reachable
at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:303)
at hudson.remoting.Engine.innerRun(Engine.java:527)
at hudson.remoting.Engine.run(Engine.java:488)

```

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 基於Kubernetes構建企業Jenkins CI/CD平台 kubernetes-jenkins CI/CD平台（十八）什么是 CI/CD？什么是CI/CD？什么是CI/CD？ CI/CD jenkins-gitlab-harbor-ceph基於Kubernetes的CI/CD運用（一）基於 KubeSphere CI/CD 將 Spring Boot 項目發布至 Kubernetes 基於 KubeSphere CI/CD 將 Spring Boot 項目發布至 Kubernetes jenkins-gitlab-harbor-ceph基於Kubernetes的CI/CD運用（二）