linux運維、架構之路-K8s中部署Jenkins集群高可用


一、在K8s中 部署Jenkins優點

1、傳統Jenkins集群架構一些問題

  • Master發生故障時,整個流程都不可用
  • Slave集群的環境配置不一樣,來完成不同語言的編譯打包,但是這些差異化的配置導致管理起來不方便,維護麻煩
  • 資源分配不均衡,有的slave要運行的job出現排隊等待,而有的salve處於空閑狀態
  • 資源浪費,每台slave可能是物理機或者虛擬機,當slave處於空閑狀態時,也不能完全釋放掉資源

2、K8s中Jenkins集群架構優點

  • 當Jenkins Master接受到Build請求后,會根據配置的Label動態創建一個運行在Pod中的Jenkins Slave並注冊到Master上,當運行完Job后,這個Slave會被注銷並且這個Pod也會自動刪除,恢復到最初的狀態(這個策略可以設置)
  • 服務高可用,當Jenkins Master出現故障時,Kubernetes會自動創建一個新的Jenkins Master容器,並且將Volume分配給新創建的容器,保證數據不丟失,從而達到集群服務高可用的作用
  • 動態伸縮,合理使用資源,每次運行Job時,會自動創建一個Jenkins Slave,Job完成后,Slave自動注銷並刪除容器,資源自動釋放,並且Kubernetes會根據每個資源的使用情況,動態分配slave到空閑的節點上創建,降低出現因某節點資源利用率高,降低出現因某節點利用率高出現排隊的情況
  • 擴展性好,當Kubernetes集群的資源嚴重不足導致Job排隊等待時,可以很容器的添加一個Kubernetes Node到集群,從而實現擴展

二、K8s 部署Jenkins

1、集群環境

 2、k8s-node1部署NFS服務

①NFS服務端配置

mkdir -p /data/k8s
chown -R nfsnobody.nfsnobody /data
echo "/data/k8s 192.168.56.0/24(rw,no_root_squash,sync)" >/etc/exports
systemctl enable rpcbind
systemctl enable nfs
systemctl start rpcbind
systemctl start nf

②客戶端掛載

systemctl start rpcbind
systemctl enable rpcbind
mkdir /data/k8s -p
mount -t nfs 192.168.56.11:/data/k8s /data/k8

3、創建Jenkins集群所需的YAML文件

①創建命名空間和存放Jenkins的YAML目錄

kubectl create namespace demon
mkdir -p /opt/jenkins

②為Jenkins數據持久化存儲創建一個PV

cat >/opt/jenkins/jenkins_pv.yaml <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: opspv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 192.168.56.11
    path: /data/k8s ---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: opspvc
  namespace: demon
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
EOF

③創建Jenkins集群權限serviceAccount文件

cat >/opt/jenkins/jenkins_rbac.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: demon

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: demon
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: demon
EOF

④創建Jenkins Deployment

cat jenkins_deployment.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: demon
spec:
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080                #外部訪問端口
          name: web
          protocol: TCP
        - containerPort: 50000              #jenkins save發現端口
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60          #容器初始化完成后,等待60秒進行探針檢查
          timeoutSeconds: 5
          failureThreshold: 12          #當Pod成功啟動且檢查失敗時,Kubernetes將在放棄之前嘗試failureThreshold次。放棄生存檢查意味着重新啟動Pod。而放棄就緒檢查,Pod將被標記為未就緒。默認為3.最小值為1
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:                       #需要將jenkins_home目錄掛載出來         - name: jenkinshome
          subPath: jenkins
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: opspvc

⑤創建Jenkins SVC

cat >/opt/jenkins/jenkins_svc.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: demon
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 30002
  - name: agent
    port: 50000
    targetPort: agent
EOF

⑥依次創建

[root@k8s-node1 jenkins]# ll
總用量 16
-rw-r--r-- 1 root root 2281 12月 27 14:59 jenkins_deployment.yaml
-rw-r--r-- 1 root root  404 12月 27 14:57 jenkins_pv.yaml
-rw-r--r-- 1 root root 1108 12月 27 14:58 jenkins_rbac.yaml
-rw-r--r-- 1 root root  285 12月 27 14:59 jenkins_svc.yaml
[root@k8s-node1 jenkins]# kubectl apply -f jenkins_pv.yaml 
persistentvolume/opspv created
persistentvolumeclaim/opspvc created

[root@k8s-node1 jenkins]# kubectl apply -f jenkins_rbac.yaml 
serviceaccount/jenkins created
clusterrole.rbac.authorization.k8s.io/jenkins created
clusterrolebinding.rbac.authorization.k8s.io/jenkins created

[root@k8s-node1 jenkins]# kubectl apply -f jenkins_deployment.yaml 
deployment.extensions/jenkins created

[root@k8s-node1 jenkins]# kubectl apply -f jenkins_svc.yaml 
service/jenkins create

⑦查看結果

[root@k8s-node1 jenkins]# kubectl get pv,pvc,pod,svc -n demon
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM          STORAGECLASS   REASON    AGE
persistentvolume/opspv   10Gi       RWX            Delete           Bound     demon/opspvc                            1h

NAME                           STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/opspvc   Bound     opspv     10Gi       RWX                           1h

NAME                           READY     STATUS    RESTARTS   AGE
pod/jenkins-6d7bc49b74-d9jxc   1/1       Running   0          1h

NAME              TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                          AGE
service/jenkins   NodePort   10.1.148.201   <none>        8080:30002/TCP,50000:26723/TCP   1h

8080:端口為我們jenkins訪問端口  50000:端口為jenkins save發現端口

⑧通過瀏覽器訪問集群任意IP的svc端口

管理員密碼路徑:持久化在/data/k8s下,所以jenkins的所有配置都在這下面

cat /data/k8s/jenkins/secrets/initialAdminPassword

 

 直接推薦安裝即可

⑨安裝完成后我們進入jenkins主頁面

 ⑩enkins–>插件–>安裝插件Kubernetes

 三、Jenkins中配置k8s

1、系統管理->系統配置

 

 2、配置拉到最下面找到Kubernetes插件

Name 配置的名稱 Kubernetes URL 這里的URL是K8s內部的URL,實際上就是svcname https://kubernetes.default.svc.cluster.local Kubernetes Namespace k8s的命名空間 (實際上就是Jenkins所在的命名空間)

 3、Jenkins URL配置

 

 

 

Jenkins URL 這里的URL是jenkins的svc名稱加上命名空間,實際上就是在k8s集群內部訪問jenkins的一個方法,這里也不需要修改 http://jenkins.demon.svc.cluster.local:8080

 4、配置添 Jenkins Slave Pod模板

Name = Pod 名稱 Namespave = Pod命名空間 Labels = Pod標簽

5、容器的模板配置

 

 

 6、創建volume的配置

 

 Jenkins Master收到Build請求時,會根據配置的Label動態創建一個運行在Pod中的Jenkins Slave並注冊到Master上,當Job運行完,這個Slave會被注銷並且這個Pod也會自動刪除,恢復到最初狀態

四、測試驗證

1、新建Job選擇流水線

 

 2、流水線Pipeline

def label = "jenkins-slave"
podTemplate(label: label, cloud: 'kubernetes')
{
node(label) {
        stage('pull code') {
            echo "拉取代碼"
        }
        stage('build') {
            echo "代碼編譯"
        }
        stage('SonarQube') {
            echo "質量掃描"
        }
    }
}

3、執行效果

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM