k8s集群部署應用----gitlab-runner

環境：

 

# 打上節點標簽 kubectl label nodes k8s-node-ci1 system.devops=gitlab-runner kubectl label nodes k8s-node-ci2 system.devops=gitlab-runner kubectl label nodes k8s-node-ci3 system.devops=gitlab-runner # 防止不相關的pod調度到該節點,我們還需要打上污點 kubectl taint nodes k8s-node-ci1 ci/cd=:NoSchedule kubectl taint nodes k8s-node-ci2 ci/cd=:NoSchedule kubectl taint nodes k8s-node-ci3 ci/cd=:NoSchedule 

 

## 本次所有操作均以官方helm char配置提供

## 添加helm repo地址,並更新helm

 

helm repo add gitlab https://charts.gitlab.io
helm repo update

Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.

# 如果沒有制定版本,將獲取最新的char包
[root@k8s-master-01 gitlab] helm fetch gitlab/gitlab-runne

[root@k8s-master-01 gitlab] tar xf gitlab-runner-0.10.1.tgz

解壓得到gitlab-runner目錄

 

 

 

 ## 前期准備工作已經做完了,現在需要我們修改配置了,因為我們直接將chart包down了下來,所以可以直接使用gitlab-runner里面的values.yaml文件,如果你是直接想通過helm install安裝 可以重新創建該文件並 -f 指定即可

 # vim values.yaml

## The GitLab Server URL (with protocol) that want to register the runner against
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register
##
# 你gitlab的地址,在runenrs能夠獲取,如果走的外部tls請使用https協議
gitlabUrl: https://xxxxxxx/

## The registration token for adding new Runners to the GitLab server. This must
## be retrieved from your GitLab instance.
## ref: https://docs.gitlab.com/ee/ci/runners/
## 地址: https://docs.gitlab.com/ce/ci/runners/README.html#shared-specific-and-group-runners
# 你的注冊碼,如果在設置獲取的token則創建的runner則為共享runner,所有項目默認走該runner,如果有多個runnenr則隨機調度,如果不想這樣用可關閉auto devops，自行單獨開啟
# 也可以在創建groups時候在CI/CD處 runners也能獲取到token,顧名思義，創建出來的runner將轉為為這個groups使用
# 還有就是直接在project了,跟groups一樣 
#runnerRegistrationToken: "M_a5FsmY899u4rPsib2k"

#runnerRegistrationToken: "uNQy5pREsmmRX-XUsyE2"
## Set the certsSecretName in order to pass custom certificates for GitLab Runner to use
## Provide resource name for a Kubernetes Secret Object in the same namespace,
## this is used to populate the /etc/gitlab-runner/certs directory
## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates
##
#certsSecretName:

## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 100

## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 30

## For RBAC support:
# 開啟權限,否則無法創建pod,
rbac:
  create: true

  ## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs
  ## cluster-wide or only within namespace
  clusterWideAccess: false

  ## If RBAC is disabled in this Helm chart, use the following Kubernetes Service Account name.
  ## 
  # serviceAccountName: default

## Configuration for the Pods that the runner launches for each new job
##
runners:
  ## Default container image to use for builds when none is specified
  ## 如果你的構建腳本沒有指定鏡像,就會用這個
  image: ubuntu:18.04

  ## Run all containers with the privileged flag enabled
  ## This will allow the docker:stable-dind image to run if you need to run Docker
  ## commands. Please read the docs before turning this on:
  ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind
  ## 開啟特權模式,則啟動的容器具有root權限,若需要調用比如docker此類的命令或者掛在外部目錄,則需要開啟
  privileged: true

  ## Namespace to run Kubernetes jobs in (defaults to 'default')
  ##
  # namespace:

  ## Build Container specific configuration
  ##
  builds:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi

  ## Service Container specific configuration
  ##
  services:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi

  ## Helper Container specific configuration
  ##
  helpers:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi
  # nodeSelector:
  #   system.devops: gitlab-runner
  # 重點,開啟緩存服務,在docker下面創建runner服務時,容器往往只會創建一個,每個階段的cache都在當前容器下,但在k8s中,每個階段都會獨立創建一個pod,也就是說,當你在上一個階段緩存下來的文件,走到下一個階段下載下來是空的
  # 所以，我們需要開啟s3緩存,如果你的機器在谷歌或者AWS，則直接申請key，按照官方的提示即可,如果沒有,則自行創建s3服務,關於s3服務創建在后面會說到
  cache: 
    ## General settings
    cacheType: s3
    # s3 啟動時加載共享目錄的根目錄
    cachePath: "export"
    cacheShared: true

    ## S3 settings
    s3ServerAddress: minio.kube-ops:9000
    # 緩存到哪個目錄,需指定 因接口的請求方式 是 http://xxxxxx/cachePath/s3BucketName/后面的目錄隨機生成
    s3BucketName: runner
    s3BucketLocation: 
    s3CacheInsecure: true
    secretName: s3access
# 配置gitlab-runner node親和
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: "system.devops"
          operator: In
          values:
          - gitlab-runner

# 配置污點容忍
tolerations: 
- key: "ci/cd"
  operator: "Exists"

## 上面的配置說完了,我們的k8s集群是部署在國內的,所以需要自行創建s3緩存服務,如果你不想創建,則需要將工作目錄掛載為本地存儲或者NFS，pvc等,請自行研究

# s3 deployment配置文件，同時通過NFS掛載了遠程目錄和s3工作目錄 /cache  /minio

# 這里注意 服務啟動命令是  minio server - /export ，不要搞混了

#

# 參考文檔 https://docs.gitlab.com/runner/install/registry_and_cache_servers.html#install-your-own-cache-server

--- apiVersion: apps/v1beta1 kind: Deployment metadata: name: minio namespace: kube-ops spec: replicas: 1 template: metadata: labels: k8s-app: minio namespace: kube-ops spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "system.devops" operator: In values: - gitlab-runner tolerations: - key: "ci/cd" operator: "Exists" containers: - args: - server - /export name: minio image: minio/minio:latest imagePullPolicy: IfNotPresent resources: requests: memory: "2G" cpu: "500m" limits: memory: "2G" cpu: "1000m" ports: - containerPort: 9000 volumeMounts: - mountPath: /root/.minio name: minio - mountPath: /export name: cache volumes: - name: minio nfs: path: /minio server: 172.16.100.122
      - name: cache nfs: path: /cache server: 172.16.100.122
--- kind: Service apiVersion: v1 metadata: name: minio namespace: kube-ops spec: selector: k8s-app: minio namespace: kube-ops ports: - protocol: TCP port: 9000 name: http

 

# kubectl apply -f minio.yaml  啟動服務

# 如果遠程掛載或者掛載到本地請提前創建目錄 mkdir -p NFS:/cache/runner 
並給上權限 
chown -R 1000:1000 cache 
以及所有權 
chmod -R 777 cache
最后 kubectl apply -f minio.yaml

minio-74ff8c7f64-r2bkg                       1/1     Running   0          2d23h

#啟動成功以后會生成accesskey以及secretkey

[root@k8s-master-01 gitlab]# kubectl exec -n kube-ops minio-74ff8c7f64-r2bkg -- cat /export/.minio.sys/config/config.json | grep Key

#  創建secret

kubectl create secret generic s3access \ --from-literal=accesskey="616236ZD08IJJX1I9M02" \ --from-literal=secretkey="VUomlerWfJiBU8MnEx4ZE7w+zcqQRagzHbgE3fGt" -n kube-ops

# 關於gitlab-runner絕大多數服務就已結束了,但是當我們做流水線時,創建的pod如果需要執行docker等這些操作時,就會報錯, 官方提供了的vloumes字段,但一直不生效,參考了他人的文檔后,可以這樣,在即將啟動runner時候將需要的配置插入進去,同時包括pod的污點以及標簽

vim gitlab-runner/templates/configmap.yaml     cat >>/home/gitlab-runner/.gitlab-runner/config.toml <<EOF
          [[runners.kubernetes.volumes.host_path]]
            name = "docker-sock"
            mount_path = "/var/run/docker.sock"
          [[runners.kubernetes.volumes.host_path]]
            name = "docker-bin"
            mount_path = "/usr/bin/docker"
 # 如果你的git倉庫地址配置了外網+域名,那么在做CI時,每次克隆下來的代碼都會通過外網拉取,通過掛載宿主機hosts文件將域名指向內網IP
          [[runners.kubernetes.volumes.host_path]]
            name = "hosts"
            mount_path = "/etc/hosts"
            read_only = true
      [runners.kubernetes.node_selector]
          "system.devops" = "gitlab-runner"
      [runners.kubernetes.node_tolerations]
          "ci/cd" = ""
    EOF

 

 

 

 

 

# 最后,生成yaml文件

[root@k8s-master-01 gitlab]# helm template gitlab-runner --name gitlab-runner --namespace kube-ops -f values.yaml > gitlab-runner.yaml
[root@k8s-master-01 gitlab]# kubectl apply -f gitlab-runner.yaml -n kube-ops
secret/gitlab-runner-gitlab-runner created
configmap/gitlab-runner-gitlab-runner created
serviceaccount/gitlab-runner-gitlab-runner created
deployment.apps/gitlab-runner-gitlab-runner created
role.rbac.authorization.k8s.io/gitlab-runner-gitlab-runner created
rolebinding.rbac.authorization.k8s.io/gitlab-runner-gitlab-runner created

[root@k8s-master-01 gitlab]# kubectl get pods -n kube-ops -owide
NAME                                         READY   STATUS    RESTARTS   AGE     IP             NODE           NOMINATED NODE   READINESS GATES
gitlab-runner-gitlab-runner-64cbbdc4-85z7q   1/1     Running   0          50m     10.244.40.14   k8s-node-ci1   <none>           <none>
minio-74ff8c7f64-r2bkg                       1/1     Running   0          2d23h   10.244.42.11   k8s-node-ci3   <none>           <none>

都運行成功了,現在我們需要驗證是否生效

 

 

## 可以看到我的某個項目下已有該pod了,如果有共享runners，則記得先關閉

 

 

 

 

# vim .gitlab-ci.yml,驗證cache是否生效,驗證docker是否能夠使用

image: ccr.ccs.tencentyun.com/yzt_public/node_v10.16.0:v9

cache:
  key: '123'
  paths:
    - rbac/

stages:
  - init
  - setup
  - test

init:
  stage: init
  script:
    - ls rbac
    - docker info

prepare:
  stage: setup
  cache:
    key: '123'
    paths:
      - rbac/  
  script:
    - touch rbac/cache.txt

rspec:
  stage: test
  cache:
    key: '123'
    policy: pull
    paths:
      - rbac/    
  script:
    - ls rbac/

 

## 點擊提交

 

 

 

查看任務執行結果

 

 

 

 

 

##  首先查看第一個stage init ,

 

 

# docker 已掛載成功

在查看最后一個stage，查看cache是否生效

 

 緩存下載成功,並且下載地址正是我們的剛才自行部署的s3共享緩存服務!!