jenkins X實踐系列（3） —— jenkins X 安裝拾遺

jx是雲原生CICD，devops的一個最佳實踐之一，目前在快速的發展成熟中。最近調研了JX，這里為第3篇，介紹下如何安裝jenkins x。

前置條件

• 安裝K8S
• 安裝ceph集群（jx需要storage class創建pv）
• 申請一個域名（可選），可以修改hosts實現
• helm
• git私服

安裝ceph集群

這里的服務器為centos 7。

使用官方的ceph-deploy安裝即可，先安裝ceph-deploy，然后每台機器安裝運行環境

 pip install ceph-deploy
 export CEPH_DEPLOY_REPO_URL=http://mirrors.ustc.edu.cn/ceph/rpm-jewel/el7
 export CEPH_DEPLOY_GPG_URL=http://mirrors.ustc.edu.cn/ceph/keys/release.asc
 ceph-deploy install docker86-156 docker86-155 docker86-154

然后安裝集群

ceph-deploy new docker86-156 docker86-155 docker86-154

修改配置文件

cat <<EOF >>ceph.conf
#osd_journal_size = 10000
public network = 192.168.86.0/24
osd_pool_default_size = 2
osd_pool_default_min_size = 1
osd_crush_chooseleaf_type = 1
osd_crush_update_on_start = true
max_open_files = 131072
osd pool default pg num = 128
osd pool default pgp num = 128
mon_pg_warn_max_per_osd = 0
mon clock drift allowed = 2
mon clock drift warn backoff = 30
mon_pg_warn_max_per_osd = 300
EOF

分發配置文件：

  ceph-deploy --overwrite-conf config push docker86-155 docker86-154 docker86-156

安裝服務

ceph-deploy mon create-initial
ceph-deploy admin docker86-156 docker86-155 docker86-154

安裝osd

ceph-deploy disk zap  docker86-156:sdb docker86-155:sdb docker86-154:sdb
ceph-deploy osd prepare docker86-156:sdb docker86-155:sdb docker86-154:sdb
ceph-deploy osd activate docker86-156:sdb1 docker86-154:sdb1

添加pool

ceph osd pool create k8smeta 128
ceph osd pool create k8sdata 128
ceph fs new k8s k8smeta k8sdata
ceph osd pool ls detail

K8S使用ceph

生成 Ceph secret

grep key /etc/ceph/ceph.client.admin.keyring |awk '{printf "%s", $NF}'|base64

假如得到: $SECRET==

在k8s創建Secret

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
  namespace: default
type: "kubernetes.io/rbd"  
data:
  key: $SECRET==
EOF

創建StorageClass

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: ceph-web
provisioner: kubernetes.io/rbd
parameters:
  monitors: 192.168.86.156,192.168.86.155,192.168.86.154
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: default
  pool: rbd
  userId: admin
  userSecretName: ceph-secret
EOF

可以將ceph設為默認的storage class：

kubectl patch storageclass ceph-web -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

安裝git私服gitea(可選）

如果已經有git的，或者直接使用GitHub的跳過。

創建PV：

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: cephfs-github-pv
  namespace: gitea
  labels:
    name: cephfs-github-pv
spec:
  capacity:
    storage: 200Gi
  accessModes:
    - ReadWriteMany
  cephfs:
    monitors: 
    - 192.168.86.156:6789
    path: /github
    user: admin
    secretRef:
      name: ceph-secret
    readOnly: false
  persistentVolumeReclaimPolicy: Retain
EOF

PVC

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-github-pvc
  namespace: gitea
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 200Gi
  selector:
    matchLabels:
      name: cephfs-github-pv
EOF

gitea部署：

cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: giteamysql
  namespace: gitea
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: giteamysql
    spec:
      containers:
      - image: gitea/gitea:latest
        imagePullPolicy: IfNotPresent
        name: gitea
        resources: {}
        volumeMounts:
        - name: ceph
          mountPath: /data 
      volumes:
        - name: ceph
          persistentVolumeClaim:
            claimName: cephfs-github-pvc
EOF

創建服務：

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: giteamysql-service
  namespace: gitea
  labels:
    app: charts
spec:
  ports:
    - port: 80
      targetPort: 3000 
  selector:
    app: giteamysql
  type: NodePort
EOF

創建ingress

cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    kubernetes.io/tls-acme: 'true'
  name: giteamysql-ingress
  namespace: gitea
spec:
  rules:
  - host: github.youdomain.com
    http:
      paths:
      - backend:
          serviceName: giteamysql-service
          servicePort: 80
        path: /
EOF

一切正常的話，打開github.youdomain.com，按提示進行安裝，設置管理員密碼即可。

安裝好后創建一個token， $git_access_token

域名與tls

將域名的通配符，a記錄到k8s集群。

申請TLS證書，使用certbot

 $ yum -y install yum-utils
$ yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
 $ sudo yum install certbot

然后申請

  certbot certonly --manual -d *.domain.com --email [e-mail]

這個會要求創建一個A記錄，按提示創建即可。

一些OK的話，服務器/etc/letsencrypt/live/domain.com/ 會生成tls證書。

$ ll /etc/letsencrypt/live/iflyresearch.com/
total 4
lrwxrwxrwx. 1 root root  40 Oct 17 15:11 cert.pem -> ../../archive/iflyresearch.com/cert1.pem
drwxr-xr-x  2 root root  78 Nov 14 09:33 certs
lrwxrwxrwx. 1 root root  41 Oct 17 15:11 chain.pem -> ../../archive/iflyresearch.com/chain1.pem
lrwxrwxrwx. 1 root root  45 Oct 17 15:11 fullchain.pem -> ../../archive/iflyresearch.com/fullchain1.pem
lrwxrwxrwx. 1 root root  43 Oct 17 15:11 privkey.pem -> ../../archive/iflyresearch.com/privkey1.pem

要在k8s使用，需要創建secret：

kubectl create secret tls research-tls-secret --cert=cert.pem --key=./privkey.pem -n=kube-system

安裝helm

jx依賴helm，需要先安裝，可以參見本系列第一篇

安裝jenkins X

先創建一個namespace：incubation

寫入 ceph-secret：

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
  namespace: incubation
type: "kubernetes.io/rbd"  
data:
  key: $SECRET==
EOF

下載jx執行文件：

  wget https://github.com/jenkins-x/jx/releases/download/v1.3.380/jx-linux-amd64.tar.gz 
  tar xzv jx-linux-amd64.tar.gz  -C ~/.jx/bin
  export PATH=$PATH:~/.jx/bin
  echo 'export PATH=$PATH:~/.jx/bin' >> ~/.bashrc

然后執行安裝命令：

jx install --external-ip=192.168.86.214 --namespace='incubation'  --git-provider-url='http://github.iflyresearch.com' --git-username='jqpeng' --git-api-token=' $git_access_token' --domain='iflyresearch.com'  --provider=kubernetes

• $git_access_token 替換為你的token
• external-ip 填寫k8s的虛擬ip

然后按提示，輸出jenkins等access_token等參數。

注意:

• 安裝過程，會依賴一些gcr.io的鏡像，請參見《google gcr.io、k8s.gcr.io 國內鏡像》解決
• 如果使用gitea，安裝修改下vim gitAuth.yaml，修改kind為gitea

---

作者：Jadepeng
出處：jqpeng的技術記事本--http://www.cnblogs.com/xiaoqi
您的支持是對博主最大的鼓勵，感謝您的認真閱讀。
本文版權歸作者所有，歡迎轉載，但未經作者同意必須保留此段聲明，且在文章頁面明顯位置給出原文連接，否則保留追究法律責任的權利。