k8s交付實戰-交付jenkins到k8s集群
1 准備jenkins鏡像
准備鏡像的操作在7.200運維機上完成
1.1 下載官方鏡像
docker pull jenkins/jenkins:2.190.3
docker tag jenkins/jenkins:2.190.3 harbor.zq.com/public/jenkins:v2.190.3
docker push harbor.zq.com/public/jenkins:v2.190.3
1.2 修改官方鏡像
基於官方jenkins鏡像,編寫dockerfile做個性化配置
1.2.1 創建目錄
mkdir -p /data/dockerfile/jenkins/
cd /data/dockerfile/jenkins/
1.2.2 創建dockerfile
cat >/data/dockerfile/jenkins/Dockerfile <<'EOF'
FROM harbor.zq.com/public/jenkins:v2.190.3
#定義啟動jenkins的用戶
USER root
#修改時區為東八區
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
echo 'Asia/Shanghai' >/etc/timezone
#加載用戶密鑰，使用ssh拉取dubbo代碼需要
ADD id_rsa /root/.ssh/id_rsa
#加載運維主機的docker配置文件，里面包含登錄harbor倉庫的認證信息。
ADD config.json /root/.docker/config.json
#在jenkins容器內安裝docker客戶端，docker引擎用的是宿主機的docker引擎
ADD get-docker.sh /get-docker.sh
# 跳過ssh時候輸入yes的交互步驟，並執行安裝docker
RUN echo " StrictHostKeyChecking no" >/etc/ssh/ssh_config &&\
/get-docker.sh
EOF
1.2.3 准備dockerfile所需文件
創建秘鑰對:
ssh-keygen -t rsa -b 2048 -C "lg@126.com" -N "" -f /root/.ssh/id_rsa
cp /root/.ssh/id_rsa /data/dockerfile/jenkins/
郵箱請根據自己的郵箱自行修改
創建完成后記得把公鑰放到gitee的信任中
獲取docker.sh腳本:
curl -fsSL get.docker.com -o /data/dockerfile/jenkins/get-docker.sh
chmod u+x /data/dockerfile/jenkins/get-docker.sh
拷貝config.json文件:
cp /root/.docker/config.json /data/dockerfile/jenkins/
1.2.4 harbor中創建私有倉庫infra
1.2.5 構建自定義的jenkins鏡像
cd /data/dockerfile/jenkins/
docker build . -t harbor.zq.com/infra/jenkins:v2.190.3
docker push harbor.zq.com/infra/jenkins:v2.190.3
2 准備jenkins運行環境
2.1 專有名稱空間和secret資源
2.1.1 創建專有namespace
創建專有名詞空間infra的目錄是將jenkins等運維相關軟件放到同一個namespace下,便於統一管理以及和其他資源分開
kubectl create ns infra
2.1.2 創建訪問harbor的secret規則
Secret用來保存敏感信息，例如密碼、OAuth 令牌和 ssh key等,有三種類型:
-
Opaque：
base64 編碼格式的 Secret，用來存儲密碼、密鑰等,可以反解,加密能力弱 -
kubernetes.io/dockerconfigjson：
用來存儲私有docker registry的認證信息。 -
kubernetes.io/service-account-token：
用於被serviceaccount引用，serviceaccout 創建時Kubernetes會默認創建對應的secret
前面dashborad部分以及用過了
訪問docker的私有倉庫,必須要創建專有的secret類型,創建方法如下:
kubectl create secret docker-registry harbor \
--docker-server=harbor.zq.com \
--docker-username=admin \
--docker-password=Harbor12345 \
-n infra
# 查看結果
~]# kubectl -n infra get secrets
NAME TYPE DATA AGE
default-token-rkg7q kubernetes.io/service-account-token 3 19s
harbor kubernetes.io/dockerconfigjson 1 12s
解釋命令：
創建一條secret，資源類型是docker-registry，名字是 harbor
並指定docker倉庫地址、訪問用戶、密碼、倉庫名
2.2 創建NFS共享存儲
jenkins中一些數據需要持久化的，可以使用共享存儲進行掛載：
這里使用最簡單的NFS共享存儲，因為k8s默認支持nfs模塊
如果使用其他類型的共享存儲
2.2.1 運維機部署NFS
yum install nfs-utils -y
echo '/data/nfs-volume 10.4.7.0/24(rw,no_root_squash)' >>/etc/exports
mkdir -p /data/nfs-volume/jenkins_home
systemctl start nfs
systemctl enable nfs
# 查看結果
~]# showmount -e
Export list for hdss7-200:
/data/nfs-volume 10.4.7.0/24
2.2.2 node節點安裝nfs
yum install nfs-utils -y
2.3 運維機創建jenkins資源清單
mkdir /data/k8s-yaml/jenkins
2.3.1 創建depeloy清單
有兩個需要注意的地方:
-
掛載了宿主機的docker.sock
使容器內的docker客戶端可以直接與宿主機的docker引擎進行通信 -
在使用私有倉庫的時候，資源清單中，一定要聲明：
imagePullSecrets: - name: harbor
cat >/data/k8s-yaml/jenkins/dp.yaml <<EOF
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: jenkins
namespace: infra
labels:
name: jenkins
spec:
replicas: 1
selector:
matchLabels:
name: jenkins
template:
metadata:
labels:
app: jenkins
name: jenkins
spec:
volumes:
- name: data
nfs:
server: hdss7-200
path: /data/nfs-volume/jenkins_home
- name: docker
hostPath:
path: /run/docker.sock
type: ''
containers:
- name: jenkins
image: harbor.zq.com/infra/jenkins:v2.190.3
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
protocol: TCP
env:
- name: JAVA_OPTS
value: -Xmx512m -Xms512m
volumeMounts:
- name: data
mountPath: /var/jenkins_home
- name: docker
mountPath: /run/docker.sock
imagePullSecrets:
- name: harbor
securityContext:
runAsUser: 0
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 7
progressDeadlineSeconds: 600
EOF
2.3.2 創建service清單
cat >/data/k8s-yaml/jenkins/svc.yaml <<EOF
kind: Service
apiVersion: v1
metadata:
name: jenkins
namespace: infra
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
app: jenkins
EOF
2.3.3 創建ingress清單
cat >/data/k8s-yaml/jenkins/ingress.yaml <<EOF
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: jenkins
namespace: infra
spec:
rules:
- host: jenkins.zq.com
http:
paths:
- path: /
backend:
serviceName: jenkins
servicePort: 80
EOF
3 交付jenkins
3.1 應用jenkins資源清單
3.1.2 部署jenkins
任意node節點
kubectl create -f http://k8s-yaml.zq.com/jenkins/dp.yaml
kubectl create -f http://k8s-yaml.zq.com/jenkins/svc.yaml
kubectl create -f http://k8s-yaml.zq.com/jenkins/ingress.yaml
啟動時間很長,等待結果
kubectl get pod -n infra
3.1.2 驗證jenkins容器狀態
docker exec -it 8ff92f08e3aa /bin/bash
# 查看用戶
whoami
# 查看時區
date
# 查看是否能用宿主機的docker引擎
docker ps
# 看是否能免密訪問gitee
ssh -i /root/.ssh/id_rsa -T git@gitee.com
# 是否能訪問是否harbor倉庫
docker login harbor.zq.com
3.1.3 查看持久化結果和密碼
到運維機上查看持久化數據是否成功存放到共享存儲
~]# ll /data/nfs-volume/jenkins_home
total 36
-rw-r--r-- 1 root root 1643 May 5 13:18 config.xml
-rw-r--r-- 1 root root 50 May 5 13:13 copy_reference_file.log
-rw-r--r-- 1 root root 156 May 5 13:14 hudson.model.UpdateCenter.xml
-rw------- 1 root root 1712 May 5 13:14 identity.key.enc
-rw-r--r-- 1 root root 7 May 5 13:14 jenkins.install.UpgradeWizard.state
-rw-r--r-- 1 root root 171 May 5 13:14 jenkins.telemetry.Correlator.xml
drwxr-xr-x 2 root root 6 May 5 13:13 jobs
drwxr-xr-x 3 root root 19 May 5 13:14 logs
-rw-r--r-- 1 root root 907 May 5 13:14 nodeMonitors.xml
drwxr-xr-x 2 root root 6 May 5 13:14 nodes
drwxr-xr-x 2 root root 6 May 5 13:13 plugins
-rw-r--r-- 1 root root 64 May 5 13:13 secret.key
-rw-r--r-- 1 root root 0 May 5 13:13 secret.key.not-so-secret
drwx------ 4 root root 265 May 5 13:14 secrets
drwxr-xr-x 2 root root 67 May 5 13:19 updates
drwxr-xr-x 2 root root 24 May 5 13:14 userContent
drwxr-xr-x 3 root root 56 May 5 13:14 users
drwxr-xr-x 11 root root 4096 May 5 13:13 war
找到jenkins初始化的密碼
~]# cat /data/nfs-volume/jenkins_home/secrets/initialAdminPassword
02f69d78026d489e87b01332f1caa85a
3.1.4 替換jenkins插件源
cd /data/nfs-volume/jenkins_home/updates
sed -i 's#http:\/\/updates.jenkins-ci.org\/download#https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins#g' default.json
sed -i 's#http:\/\/www.google.com#https:\/\/www.baidu.com#g' default.json
3.2 解析jenkins
jenkins部署成功后后,需要給他添加外網的域名解析
vi /var/named/zq.com.zone
jenkins A 10.4.7.10
# 重啟服務
systemctl restart named
3.3 初始化jenkins
瀏覽器訪問http://jenkins.zq.com,使用前面的密碼進入jenkins
進入后操作:
- 跳過安裝自動安裝插件的步驟
- 在
manage jenkins->Configure Global Security菜單中設置
2.1 允許匿名讀：勾選allow anonymous read access
2.2 允許跨域：勾掉prevent cross site request forgery exploits - 搜索並安裝藍海插件
blue ocean - 設置用戶名密碼為
admin:admin123
3.4 給jenkins配置maven環境
因為jenkins的數據目錄已經掛載到了NFS中做持久化,因此可以直接將maven放到NFS目錄中,同時也就部署進了jenkins
3.4.1 下載並解壓
wget https://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz
tar -zxf apache-maven-3.6.1-bin.tar.gz -C /data/nfs-volume/jenkins_home/
mv /data/nfs-volume/jenkins_home/{apache-,}maven-3.6.1
cd /data/nfs-volume/jenkins_home/maven-3.6.1
3.4.2 初始化maven配置：
修改下載倉庫地址,除了<mirror>中是新增的阿里雲倉庫地址外,其他內容都是settings.xml中原有的配置(只是清除了注釋內容)
cat >conf/settings.xml <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<pluginGroups>
</pluginGroups>
<proxies>
</proxies>
<servers>
</servers>
<mirrors>
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>*</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
</mirrors>
<profiles>
</profiles>
</settings>
EOF