一、前提
本次實踐前，需已完成以下過程：
1、搭建好一個Kubernetes集群(本實踐為單節點集群)，網上參考較多，不贅述。
2、選取kubernetes集群外的一台服務器安裝 NFS服務端，並在集群內每個節點安裝 NFS客戶端;
NFS服務端所在的服務器IP為 10.141.211.178，記為 nfs server；而集群master服務器，記為 master;
(1) nfs server創建存儲目錄 /data/k8s/ 並執行命令：chmod 755 /data/k8s/, 並關閉防火牆
(2) nfs server安裝NFS，執行: yum -y install nfs-utils rpcbind，
再配置NFS，執行：vi /etc/exports，在該文件內添加內容：/data/k8s *(rw,sync,no_root_squash)，
然后啟動NFS服務，執行：
# systemctl start rpcbind && systemctl enable rpcbind
# systemctl start nfs && systemctl enable nfs
(3) master同樣安裝並啟動NFS，執行：
# yum -y install nfs-utils rpcbind
# systemctl start rpcbind && systemctl enable rpcbind
# systemctl start nfs && systemctl enable nfs
再執行：showmount -e 10.141.211.178，可看到共享目錄 /data/k8s
二、集群安裝Jenkins
Jenkins master的安裝，需要將數據持久化。可以利用NFS作為存儲資源，創建PVC對象來掛載。PV/PVC配置文件pvc.yaml如下：
apiVersion: v1
kind: PersistentVolume
metadata:
name: opspv
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Delete
nfs:
server: 10.141.211.178 #注意：此處為NFS服務器的地址
path: /data/k8s
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: opspvc
namespace: kube-ops
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
同時，對於即將創建的Jenkins master資源對象，需要授予其一些權限，比如增刪改查等。相應的配置文件rbac.yaml如下：
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins
namespace: kube-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-ops
然后，基於jenkins/jenkins:lts 鏡像創建jenkins master鏡像，配置文件jenkins.yaml 如下：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jenkins
namespace: kube-ops
spec:
template:
metadata:
labels:
app: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccount: jenkins
containers:
- name: jenkins
image: jenkins/jenkins:lts
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
volumeMounts:
- name: jenkinshome
subPath: jenkins
mountPath: /var/jenkins_home
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
securityContext:
fsGroup: 1000
volumes:
- name: jenkinshome
persistentVolumeClaim:
claimName: opspvc
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: kube-ops
labels:
app: jenkins
spec:
selector:
app: jenkins
type: NodePort
ports:
- name: web
port: 8080
targetPort: web
nodePort: 30002
- name: agent
port: 50000
targetPort: agent
最后，在一個目錄內分別創建以上3個文件，執行命令如下：
# kubectl create namespace kube-ops
# kubectl create -f pvc.yaml
# kubectl create -f rbac.yaml
# kubectl create -f jenkins.yaml(此步執行會出現文件權限問題，解決辦法為：
先在nfs server服務器執行：chown -R 1000 /data/k8s/jenkins
然后在master執行：kubectl delete -f jenkins.yaml
kubectl create -f jenkins.yaml )
此時，我們通過命令kubectl -n kube-ops get pod可以查看到jenkins已成功創建。
三、Jenkins配置動態slave
1、初始化Jenkins配置
瀏覽器打開masterIP:30002，如下：
其中的管理員密碼，我們既可以進入容器內對應的目錄查看，也可以在nfs server服務器上執行命令：
cat /data/k8s/jenkins/secrets/initialAdminPassword 來查看；然后選擇安裝推薦的插件，如下：
然后添加管理員賬戶即可進入Jenkins界面。
2、配置jenkins slave
(1) 安裝Kubernetes插件
進入 Manage Jenkins—>Manage Plugins—>可選插件(Available)—>Kubernetes plugin勾選，直接安裝即可。
(2) 配置Kubernetes插件
點擊Manage Jenkins—>Configure System—>雲—>新增一個雲—>Kubernetes，如下：
然后配置如下：
先注意 名稱默認為kubernetes，然后 Kubernetes地址 填寫https://kubernetes.default.svc.cluster.local，命名空間為kube-ops；接着點擊右邊的 連接測試 按鈕，如果顯示Connection test successful，表示Jenkins可以和Kubernetes集群正常通信了。最后，在Jenkins地址，填入：http://服務名.kube-ops.svc.cluster.local:8080，如下所示：
(3) 配置 Kubernetes Pod Template
關於 Kubernetes Pod Template部分的配置，其實就是對jenkins slave的配置。具體配置如下：
圖中標記的地方較為重要，不要填錯。其中標簽列表部分 后面仍有用到；Docker 鏡像部分，是本人基於 cnych/jenkins:jnlp6 鏡像基礎上繼續定制的鏡像，包含maven、docker、docker-compose、kubectl等工具。
另外，添加卷如下：
添加這兩個 Host Path Volume，是為了更好地在jenkins slave容器中使用docker 和 kubectl 工具，所以掛載了宿主機的部分目錄。
然后，設置Service Account如下：
最后，點擊 保存 即可。
3.測試jenkins slave
首先新建一個 名為test 的 Freestyle project 項目，其配置如下：
這里的標簽表達式，正是Kubernetes Pod Template的標簽列表的內容。
然后，增加構建步驟—>執行shell，如下:
具體shell如下：
echo "測試 Kubernetes 動態生成 jenkins slave"
echo "===========mvn==========="
mvn --version
echo $PATH
echo "==============docker in docker==========="
which docker
docker version
echo "==============docker-compose==========="
docker-compose version
echo "=============kubectl============="
kubectl get pods
保存之后，點擊 立即構建。控制台輸出如下：
Started by user admin
Running as SYSTEM
Agent jnlp-slave-pk06f is provisioned from template Kubernetes Pod Template
---
apiVersion: "v1"
kind: "Pod"
metadata:
annotations: {}
labels:
jenkins: "slave"
jenkins/jnlp-slave: "true"
name: "jnlp-slave-pk06f"
spec:
containers:
- env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_AGENT_NAME"
value: "jnlp-slave-pk06f"
- name: "JENKINS_NAME"
value: "jnlp-slave-pk06f"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://jenkins.kube-ops.svc.cluster.local:8080/"
image: "zhongyuanzhao000/jenkins-slave:jnlp"
imagePullPolicy: "IfNotPresent"
name: "jnlp"
resources:
limits: {}
requests: {}
securityContext:
privileged: false
tty: true
volumeMounts:
- mountPath: "/var/run/docker.sock"
name: "volume-0"
readOnly: false
- mountPath: "/root/.kube"
name: "volume-1"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
workingDir: "/home/jenkins/agent"
nodeSelector: {}
restartPolicy: "Never"
serviceAccount: "jenkins"
volumes:
- hostPath:
path: "/var/run/docker.sock"
name: "volume-0"
- hostPath:
path: "/root/.kube"
name: "volume-1"
- emptyDir:
medium: ""
name: "workspace-volume"
Building remotely on jnlp-slave-pk06f (jnlp-slave) in workspace /home/jenkins/agent/workspace/test
[test] $ /bin/sh -xe /tmp/jenkins3820575614440094591.sh
+ echo 測試 Kubernetes 動態生成 jenkins slave
測試 Kubernetes 動態生成 jenkins slave
+ echo ===========mvn===========
===========mvn===========
+ mvn --version
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
Maven home: /usr/local
Java version: 1.8.0_212, vendor: Oracle Corporation, runtime: /usr/local/openjdk-8/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix"
+ echo /usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ echo ==============docker in docker===========
==============docker in docker===========
+ which docker
/usr/local/bin/docker
+ docker version
Client: Docker Engine - Community
Version: 18.09.8
API version: 1.39
Go version: go1.10.8
Git commit: 0dd43dd87f
Built: Wed Jul 17 17:38:58 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.7
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 2d0083d
Built: Thu Jun 27 17:26:28 2019
OS/Arch: linux/amd64
Experimental: false
+ echo ==============docker-compose===========
==============docker-compose===========
+ docker-compose version
docker-compose version 1.23.2, build 1110ad01
docker-py version: 3.6.0
CPython version: 3.6.7
OpenSSL version: OpenSSL 1.1.0f 25 May 2017
+ echo =============kubectl=============
=============kubectl=============
+ kubectl get pods
NAME READY STATUS RESTARTS AGE
jenkins-575b84fb7b-59s5h 1/1 Running 0 29h
jnlp-slave-pk06f 1/1 Running 0 7s
Finished: SUCCESS
其中，你可以發現jenkins創建了jnlp-slave-pk06f的slave對象；而當該任務執行完之后，你在master上獲取pod就會發現 jnlp-slave-pk06f 這個slave自動消失了，這就是動態jenkins slave的簡單體現。
四、參考
本次實踐得益於諸多運維大神對於知識的不吝分享，十分感謝！！！
具體參考的博客或指南如下：
基於 Jenkins 的 CI/CD (一) 強烈推薦陽明老師的博客
kubernetes實踐：安裝jenkins slave
kubernetes Jenkins gitlab搭建CI/CD環境 (二)
01 [從這里開始]Jenkins CI解決方案