jenkins 如何發布k8s項目-1

一環境准備

1.1 外部環境准備

1 docker倉庫，harbor我這里用的是線上的hub https://hub.docker.com/ 自己注冊即可
2 准備git倉庫，我這里用的是碼雲 https://gitee.com/huningfei/java.git
3 准備好k8s環境，一主兩從，可以二進制安裝，也可以kubeadmin安裝，k8s版本1.15，

4 准備好jenkins環境，（我是利用jenkins.war包直接部署的，也可以用k8s部署jenkins）

1.2 jenkins環境准備

1 安裝Kubernetes,Kubernetes Continuous Deploy 和git，pipeline 安裝這幾個插件

2 在jenkins服務器上面安裝maven和docker環境。

yum install maven
# install docker
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
  yum -y install docker-ce-18.06.1.ce-3.el7
   curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
  systemctl enable docker && systemctl start docker
 
 # 還需要注意一點，如果你的jenkins是用普通用戶啟動的，當構建的時候可能會提示權限問題， 要改一下/var/run/docker.sock  這個文件的權限

二 設置jenkins

1 設置jenkins去連接k8s api

1.1 kubeadmin方式

Configure system
系統管理---系統設置-cloud

如下圖所示

1.1.1 kubernetes 服務證書從哪里來？

如果是kubeadmin部署的k8s,就比較麻煩，進入到 /etc/kubernetes目錄下面打開admin.conf配置文件，文件中有三個值 certificate-authority-data 、 client-key-data ,client-certificate-data 分別用這三部分生成證書，這里我把生成的證書都放到了tmp目錄下面,切記順序不能亂

echo LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1USXlOVEE0TVRFMU4xb1hEVEk1TVRJeU1qQTRNVEUxTjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBT0d6Ckl0RzFDMTMxL09ObXNJeERraTFrWDFyZ25kZ0hLRnBLTmJxNVl2R21jamh3a21SVjdnc3g2VGVIakpCY01welUKUFFGU3FkcjRldUJQTTM1MGhTSnkrODZpSWl5NWRzZTNONGw2bng1cjVyQzB1TEk1c1BLTXdLb3pGaHc4a0pFUwpLQnBpTFd2Q0VsTTljVlJKWnpiRENISWNHc09ZUklaWDBPcDJWNEh0U21FdlRuU0wrL1hEZmEzcUR1Y0NLVGNWCnNVWjNwdmRvMERCbEZzUUZPZDVQVDF4Rm13WmlSeDBneVJDS1VITDc0V1A3akk1eFZsYWMwczJ2QlRzaEp5L2IKTjBIcDM2RXQ2NUlLRHFrVjM4RS9ONEZEN3B2eWkxTGtxcWI3QXF5VG1VQlpsd2taclpPTnhuZFhUeGdWeXZVVApSRTBtZW1KNnAxaWp1MVFGKzZjQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFDUERiRXBzdlhVWVc2bW1LZjB2TjNkS0ZGN0kKbldUcUJBUXdTcnBFK0oxL0FueWRHOTFVT2NtQnZMdTBkTzhJbk56VXJvWnNWbEVqNjVpUDNMU2hEbzdqQzEzbQp5MFFrM2Zzc1ltd1JpY3NwcE9RMmpLZ1dETUF5dGVoRm1zTnBybmxQTWJueXVLQktHOTdDeWpYdExvam5jbGtQCnBqb1BYdFRNTmZSemJOdEVEeVNzcHhTaFZEaDQ1eWdBMk9yK3BTbGlaRXY4YUFJcWFTbFU2MU5rV3FuVGtlS0oKNzJ4SW9JRmpyV3ZrQzZsTDdMUVQrcjJxQlFjTDBGbDNiQS9zbnFoM2Irck1rTDZYQkZ6UmFnK1pwdEV3bktIdgordG85bVBaR0FVUldLakNvRHdXN05Da3NjdEZRTnBUbGZUMWppaUJXRHFNd1pqclkxdWlhcW5Lb3BuTT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= | base64 -d > /tmp/ca.crt

echo LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBelBKRllPQVdFZm94WDh6dVh2NWhjUE5DS1oxVUZ4RGF6T3lSYjNaZnRXTVNmUDk0Ck92dW5Ocmx5MGh1Yk9MMmxrWllXM3ZUb2dsbXhiaVNKdGw4SDM3cHlzbXJ6bjJSMk5LMzZkRmQ4YVgxeDkzYXoKS2xWK1YwMnhuUUVreFZlL3lRUmE0L0RVUzdVUHdOdU85Q2FDVWoxV2dzVStwcjA0UlBEYVhpaE9VTGR0cVEyagpKeVJkeGhTQVBBMDdSWGM5cmFKODMzMXFBcUVJQkl2bG56Vm9zM1d6WUlhUzI2d29OOHhIaStpTXJGT2pkVnV3CjM0OGlvTlhIUUZuaVk4c0ZJTXpua0dEQmljZ2N3R1IzeGRGM1NkdldudFZlQ0xlWFZBaS9iOStOVTBBZmlEdjkKNmsrRVJQUEJ5UFIrWXIxR1ZtTWNncG40Q0lwaUlRZG11SmhxQlFJREFRQUJBb0lCQVFERkhONk5aZmpVSVlZTApWZGFObHF1eExzV2JzSzJINGJZUEVoaHUwN01KaWRmbVVPOGFDNSt2aW4xY2h5Q3JXZnM4SXRhc2FYQ1dyaDF3CmljUEptMnkvU1Z1M0xoWlNYeFV4WllOVkVpK2VaZ0NPdWFOb1BBSGZoZUlsMTB1K1liOGJYd3pyM2x6UTA3dnIKQ1VCY29nMEZlK2tRclJTczJhQ2M3ZW1ZVmJ1ODlIUXlGV2dsK21VQ1FYVzFXWWo1OWlmT3hkRlNEdVlaUUo5bwo1OHhPOWRHQXJQRll6WmdmZ0g4UkU2dXhiRlJUNUlXQlg1aEpLL2hQMDE1WW04M0tPWk9pNGZLOEg5cVhyK0VjCkZob3VIU0ZoRng1TTVURjgyVHpXWjNLS2FFYmhOMWxoUXdnNWIyVHQ3QzNDSlNoL01VOGx4UUJmbUVxR24yZW0KZGJveVNtOEJBb0dCQVBwZEtacG9hNngzNTFKSytNeW5lck05MVBMQXRXMXZMc3V6Z1Z1dGE2MGZIZlhSYWcwLwpTTFhpem5nb3FIMzducE55T28yTGl3aEtYb0NQNXpXMVVwQkt4UlJmeS8yYk9rWFBHMk9Qb2U5anNpeHFUV3NrCnBFakpEaEp2WnZHbVJVaEIvdVhRaGxuSmEzNWpWMzdzMEVnU1lVd0V5RmlPeXZQSXRIWmhhRDdOQW9HQkFOR1AKWG5QcnBaTlVVdFpSTVVxUjV6Z2RySmZFWTU2MUpTRjQ4YUVWVzRVWThsT1V4T3ZYT2xuTGFWNnZ6TWx6T3VPYQo0RStVRW1TaFYvNkt5Qm5jV2R1dmtnMmVBSEo2U2lnZlhJeGhYcHp0b2o0NFNTNlJ3M2Z3UE5NVEF5VjNQd2VtCjBWaTFRa1NtS0JMY2pYVE5VZ1FOaU5zNk8rVFJFN0EydlEyZGcyZ1pBb0dCQU5xS3JnRXB1eXVPY3E3RDROQU0KTmEvMmZrYjBicW83RGpjajF2d21za0lwVW1hOFNQMk5TelB4NmxhNjd2RWh6ZmRaRThGd21JbEJHYUxSZW5ySQpiMnlpeDh1VUg2V2RkVTF3anJEbXRPMFpNbk5ReTRtR0w1MlZXeUkrZWpiZjg4UXlUNFZkODVpMm1JMTN1KzJBClhBTTlnQTd5Y2N2VHdWYngwSTB4VStUVkFvR0FmcXAzQUFCV2s0ZnJqaW1EYnJ4a0V4STBxWU9HWjM2OGs4L2oKMVdid2E3SFpmMVd6OVkyaGNuYzJHSXFRY00rYXI4cUVmUFZXYk1idFJpa2lyQ2I1bFlVNDljd2tIdzMxMXV5bAo1eUJQclBFaUdST3lnRmlRMnVVMkRxczJRcVlpVGdDeUZ6bHdkY2dzL3NHYkt4ZVQxR2xONkp6NWFPUkxUejYyCjVRenBTYmtDZ1lFQWllUlY4b0FUNUdKQlRGK3FnMFhPZFA3ZFJUZFZHek1wb05rWVpzOEZ3bzBrT2ZZRHE2eVoKOTdQV2VHeEkxNlJWS2V0N1U0NlpTUGNOUWdkMWdmcklmenZmQndaeVA3NkVtTkRGTDJCZWJNaVBkTGtXRUp3RgozbmliUUZxd1pkdkw1L2tuSklhdkpKY3RGZ2NxOGFvREtWSVNYU255YzRrUVBJZUlKbEZWWk5nPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=  | base64 -d > /tmp/client.key

echo LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4ekNDQWR1Z0F3SUJBZ0lJZVVLY0NoaXNBRTR3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB4T1RFeU1qVXdPREV4TlRkYUZ3MHlNREV5TWpRd09ERXlNREZhTURVeApGVEFUQmdOVkJBb1RESE41YzNSbGJUcHViMlJsY3pFY01Cb0dBMVVFQXhNVGMzbHpkR1Z0T201dlpHVTZjMkZ6CmN6QXdNVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNenlSV0RnRmhINk1WL00KN2w3K1lYRHpRaW1kVkJjUTJzenNrVzkyWDdWakVuei9lRHI3cHphNWN0SWJtemk5cFpHV0Z0NzA2SUpac1c0awppYlpmQjkrNmNySnE4NTlrZGpTdCtuUlhmR2w5Y2ZkMnN5cFZmbGROc1owQkpNVlh2OGtFV3VQdzFFdTFEOERiCmp2UW1nbEk5Vm9MRlBxYTlPRVR3Mmw0b1RsQzNiYWtOb3lja1hjWVVnRHdOTzBWM1BhMmlmTjk5YWdLaENBU0wKNVo4MWFMTjFzMkNHa3R1c0tEZk1SNHZvakt4VG8zVmJzTitQSXFEVngwQlo0bVBMQlNETTU1Qmd3WW5JSE1CawpkOFhSZDBuYjFwN1ZYZ2kzbDFRSXYyL2ZqVk5BSDRnNy9lcFBoRVR6d2NqMGZtSzlSbFpqSElLWitBaUtZaUVIClpyaVlhZ1VDQXdFQUFhTW5NQ1V3RGdZRFZSMFBBUUgvQkFRREFnV2dNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUYKQndNQ01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQkpaWlh5SEdhM1BMb1Rtb0I4RmtMRDN4OHRBTk9oRW8vMQozY0lBZ1NsMFd0SFU3Lzh0Wlo5alBIa3djNWZCeE94Uk5lZk4wWVdoSUR4UFFmSkZwWlEyL0JxTmw1dnpRNGlICko4ZGJZblJxTmcyYWJWUHQvdEtERGtKanBNQ2U4LzlJbFFZY3M5L0gxVUtpaW5WOHFPUmwvKzBvTFFDdkRMRzcKYWdXc1pMb3M1MWVWM3Z3WnBzSTZvSFNSVlFuaTZLVXc3RnJpUWNtYS94WkVweitPWlBlUFBrZVUzQXR2OXQyRQpWcHhDNlZudkMvaWJZU2xEWENmU1lYeEZNZFh2bzQxamgvSk11QU1KRGZqRDJOODM4TFhVMVd3d1hqUHRQUWFICllsbEtENjhoRmoycTJVR0xOMUZqSDZSMEJxeDV2cTkrRUNBQW5GOVgwYU9KYlpxdnVmNDgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= | base64 -d > /tmp/client.crt

然后到tmp目錄，在生成一個ca.crt證書和一個cert.pfx

openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
 生成一個pfx的證書，會提示你輸入密碼 111

然后將ca.crt里面的內容放到jenkisn服務證書key的地方即可。

1.1.2 添加憑據
點擊add,選擇kind類型，然后上傳你剛才生成的cert.pfx這個證書，並且輸入剛才的密碼

最終去測試jenkins能否連接成功，如圖

1.2 pod類型的jenkins(這個是最簡單的)

如果你的jenkins是直接部署在k8s之內的，就比較簡單，不需要填寫證書，直接這樣，如圖

1.3 二進制類安裝的k8s

如果你是用的二進制安裝的k8s,這個證書在安裝過程中，會產生，我測試了用ca.pem和server.pem發現都可以連接，並且不需要添加憑據。

2 添加憑據

找到這個，點擊進去

要添加的憑據有docker_hub,gitlab,k8s_auth
其中k8s_auth添加的辦法是，如下圖：

添加完成之后，去憑據里面查看即可

三 發布k8s項目-kubeconfigId方法

腳本內容如下:

// 公共
def registry = "huningfei"
// 項目
def project = "welcome"
def app_name = "demo"
def image_name = "${registry}/${project}:${BUILD_NUMBER}"
def git_address = "https://gitee.com/huningfei/java.git"
// 認證
def secret_name = "registry-pull-secret"
def docker_registry_auth = "450fb8b4-62c9-43fe-8a69-ec30705e724e"
def git_auth = "b363af0f-a96c-465c-a14e-60f7127727eb"
def k8s_auth = "a60b4b7f-2aef-4622-94a5-6efef67c4ac9"


  node(){
      // 第一步
      stage('拉取代碼'){
         checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
      }
      // 第二步
      stage('代碼編譯'){
          sh "mvn clean package -Dmaven.test.skip=true"
      }
      // 第三步
      stage('構建鏡像'){
          withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
            sh """
              echo '
                FROM lizhenliang/tomcat 
                RUN rm -rf /usr/local/tomcat/webapps/*
                ADD target/*.war /usr/local/tomcat/webapps/ROOT.war 
              ' > Dockerfile
              docker build -t ${image_name} .
              docker login -u ${username} -p '${password}'
              docker push ${image_name}
            """
            }
      }
      // 第四步
      stage('部署到K8S平台'){
          sh """
          sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yml
          sed -i 's#\$SECRET_NAME#${secret_name}#' deploy.yml
          """
          kubernetesDeploy configs: 'deploy.yml', kubeconfigId: "${k8s_auth}"
      }
  }

注意： 腳本里面的deploy.yml是部署k8s的文件，這個文件必須存到gitlab倉庫里面就是要跟代碼在同一級目錄里面，內容如下：

---
apiVersion: extensions/v1beta1 
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: java-demo
  template:
    metadata:
      labels:
        app: java-demo 
    spec:
      imagePullSecrets:
      - name: $SECRET_NAME 
      containers:
      - name: tomcat 
        image: $IMAGE_NAME 
        ports:
        - containerPort: 8080
          name: web
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12


---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: NodePort
  selector:
    app: java-demo 
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web 
spec:
  rules:
  - host: demo2.java.sanwenqian.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: web 
          servicePort: 80

四 構建項目

這里要說明一下，19年12月，按照此方法是成功的，但是最近再次做的時候，發現一直報錯，信息如下：

這個信息是說deploy.yml這個文件格式有錯誤，但是我手動apply 的時候卻是正常的，就是說只有用jenkins的時候會有這個錯誤，期間換過k8s和jenkins版本，但最終都是這個錯誤，因此並沒有解決，不知有哪位遇到過這個問題？

五 第二種辦法-用kubeconfig方法，這種方法就不用kubeconfig這種證書的方式了

1 安裝jekins插件-Kubernetes CLI Plugin

找到k8s的秘鑰證書-certificate-authority-data

首先去/root/.kube 打開config這個文件，找到下面的

然后echo xxxxx | base64 -d > /tmp/ca.crt 生成證書

用證書去生成連接k8s的kubeconfig,如下圖
點擊下面的generate pieline script 就會生成部署的語句。

3 pipeline語句

注意下面的serverUrl 這個，你的jenkins必須能和這個地址通信，

def registry = "huningfei"
// 項目
def project = "welcome"
def app_name = "demo"
def image_name = "${registry}/${project}:${BUILD_NUMBER}"
def git_address = "https://gitee.com/huningfei/java.git"
// 認證
def secret_name = "registry-pull-secret"
def docker_registry_auth = "e9a6d521-6b42-4892-96c5-f7b675039fb6"
def git_auth = "e043a097-5ad8-4ec4-a2d9-df27e8debfc4"
def k8s_auth = "4bb069f6-36ce-4257-bef3-1c3b2c86ceee"


  node(){
      // 第一步
      stage('拉取代碼'){
         checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
      }
      // 第二步
      stage('代碼編譯'){
          sh "mvn clean install -Dmaven.test.skip=true"
          
      }
      // 第三步
      stage('構建鏡像'){
          withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
           sh """
              echo '
                FROM lizhenliang/tomcat 
                RUN rm -rf /usr/local/tomcat/webapps/*
                ADD target/*.war /usr/local/tomcat/webapps/ROOT.war 
              ' > Dockerfile
              docker build -t ${image_name} .
              docker login -u ${username} -p '${password}'
              docker push ${image_name}
            """
            }
      }
      // 第四步
      stage('部署到K8S平台'){
      
                 	kubeconfig(caCertificate: '''-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIwMTIzMTA4NTIwN1oXDTMwMTIyOTA4NTIwN1owFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM1X
b1j9aX0rOuNpKTUQZ/Ai5xUDQqKSSpq4SkencOGS2r86H76eetIg+cK9n86Ag7Ds
DKtp4fqGo2VJLocJXNtlySHa0G9txAtiQXOC1hukRev0npJFLx9dU4csfLZyZFgt
uYI5UB7QaFSneCnX0kUhXCb0KxKOpIMB2G6kkzW7jMEUVYZmFf/esHGwyz4vsnr0
A/3+WDlloROCC91FoU/XTLA9bJqagMpjtIxtv39bwMfZ/VTO7TK393kYIQ66zj8S
ZsWZkJFXXXJOlf8BwQa7Cab6lSEfZZ8975FQnLwveUcWR7fOj6uRpi6lDSjs1y60
UI279dfNzM1t6n2rRB0CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAI4vEDTiK7JZNiAZRqzBmPu8gCIS
5J1Fye5+bJmtWU/Hc8a2U/R3PfZPKtGJyXHuIVAS5JYJUKTXiC9PoE+UHBBwzDK6
viRlbmri0kasmTerTksW8QCjfVHtJiqmo0Iluo7ytZHhDsOWwuhhGzmFU9FeP7S8
kMfxDLT39HMj7C8PrGgHtZtY6Dva2UNh/lBaMT/+/VrbGpfzIRPe3wh5nkxmzMIX
2gOw6TzAu8WfKo+Ne/kzO+dDNXTzcK/+E5MrsvMrDZoiRNsvjqWwl+fyQyPaHaQ0
x4OlXHG8cH8Z2iABNYRxgXtV323z2HguBrkovy3OsSTxMyu2ZOWWjrClg3c=
-----END CERTIFICATE-----''', credentialsId: '560a5481-119c-402f-b01b-60acc0efe216', serverUrl: 'https://172.29.251.182:6443') {
    // some block
    sh """
      sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yml
     kubectl apply -f deploy.yml
    """
}

       
         
        }
   }

最后構建即可

最后效果如下