04-5-持續部署
1. 概念
1.1. 基於Kubernets生態的閉環
Kubernets集群的目標是為了構建一套Paas平台：
- 代碼提交：開發將代碼提交到Git倉庫
- 持續集成：通過流水線將開發提交的代碼克隆、編譯、構建鏡像並推到docker鏡像倉庫
- 持續部署：通過流水線配置Kubernetes中Pod控制器、service和ingress等，將docker鏡像部署到測試環境
- 生產發布：通過流水線配置Kubernetes中Pod控制器、service和ingress等，將通過測試的docker鏡像部署到生產環境
涉及到的功能組件：
- 持續集成用Jenkins實現
- 持續部署用Spinnaker實現
- 服務配置中心用Apollo實現
- 監控用Prometheus+Grafana實現
- 日志收集用ELK實現
- 通過外掛存儲方式實現數據持久化，甚至可以通過StoargeClass配合PV和PVC來實現自動分配和掛盤
- 數據庫屬於有狀態的服務，一般不會放大Kubernets集群中
1.2. Spinnaker
Spinnaker(https://www.spinnaker.io)是一個開源多雲持續交付的平台，主要提供了兩個功能：
1.2.1. 應用管理
Spinnaker使用應用程序管理功(Application management)能來查看和管理您的雲資源，常涉及到的雲資源有 Azure、AWS、Kubernetes等，不支持國內阿里雲、騰訊雲。
Applications, clusters, server groups是Spinnaker用來描述服務的關鍵概念。Load balancers and firewalls 描述了您的服務如何向用戶公開。
1.2.2. 應用部署
應用部署中核心功能有兩個，流水線和部署策略。流水線將CI和CD過程串聯起來，每個項目構建一個流水線，通過傳遞變化的參數(服務名、版本號、鏡像標簽等)，調用Jenkins中持續集成流水線完成構建，再通過提前部署方式(如kubernetes中deployment/service/ingress)方式來將構建好的鏡像發布到指定的環境中。部署策略是在通過測試環境測試之后，在生產環境中的升級策略，常用的有藍綠發布、金絲雀發布、滾動發布。
1.3. Spinnaker的常用組件
1.3.1. 組件
- Deck: 基於瀏覽器的UI界面
- Gate: 即Apigateway，UI和所有api調用程序都通過Gate與Spinnaker進行通信
- Orca: 編排引擎
- Clouddrive: 操縱雲環境資源的驅動
- Front50: 用於持久保存應用程序，管道，項目和通知的元數據，存放在桶中。本實驗采用Minio存儲(類似s3)
- Rosoc: 為雲廠商提供VM鏡像或者鏡像模板，Kubernetes集群中不涉及
- Igor: 提供流水線構建
- Echo: 它支持發送通知（例如，Slack，電子郵件，SMS），並處理來自Github之類的服務中傳入的Webhook。
- Fiat: 提供用戶認證，本實驗未涉及，后期可以考慮使用
- Kayenta: 提供金絲雀部署分析的，本實驗未涉及
- Halyard: 提供spinnaker集群部署、升級和配置的，本實驗未涉及
1.3.2. 架構圖
2. 部署Spinnaker
2.1. 部署Minio
2.1.1. 准備鏡像
[root@hdss7-200 ~]# docker pull minio/minio:latest
[root@hdss7-200 ~]# docker image tag minio/minio:latest harbor.od.com/public/minio:latest
[root@hdss7-200 ~]# docker image push harbor.od.com/public/minio:latest
2.1.2. 准備資源配置清單
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: minio
name: minio
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
name: minio
template:
metadata:
labels:
app: minio
name: minio
spec:
containers:
- name: minio
image: harbor.od.com/public/minio:latest
ports:
- containerPort: 9000
protocol: TCP
args:
- server
- /data
env:
- name: MINIO_ACCESS_KEY
value: duduniao
- name: MINIO_SECRET_KEY
value: duduniao12345
readinessProbe:
failureThreshold: 3
httpGet:
path: /minio/health/ready
port: 9000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
volumeMounts:
- mountPath: /data
name: data
volumes:
- nfs:
server: hdss7-200
path: /data/nfs-volume/minio
name: data
apiVersion: v1
kind: Service
metadata:
name: minio
namespace: armory
spec:
ports:
- port: 80
protocol: TCP
targetPort: 9000
selector:
app: minio
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: minio
namespace: armory
spec:
rules:
- host: minio.od.com
http:
paths:
- path: /
backend:
serviceName: minio
servicePort: 80
2.1.3. 交付minio到k8s
[root@hdss7-200 ~]# mkdir /data/nfs-volume/minio # 創建待共享的目錄
[root@hdss7-21 ~]# kubectl create namespace armory
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/minio/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/minio/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/minio/ingress.yaml
[root@hdss7-11 ~]# vim /var/named/od.com.zone
......
minio A 10.4.7.10
[root@hdss7-11 ~]# systemctl restart named
[root@hdss7-11 ~]# host minio.od.com
minio.od.com has address 10.4.7.10
2.2. 部署redis
Spinnaker中的redis僅僅是起到緩存作用，對Spinnaker的作用不是很大，即使宕機重啟也問題不大，且並發小。基於當前有限的資源條件下，考慮使用單個副本非持久化的方式部署redis。如果需要持久化，在啟動容器時，指定command和args，如改為： /usr/local/bin/redis-server /etc/myredis.conf
2.2.1. 准備鏡像
# 推薦使用4.x版本，高版本的在spinnaker中是否適暫不清楚
[root@hdss7-200 ~]# docker image pull redis:4.0.14
[root@hdss7-200 ~]# docker image tag redis:4.0.14 harbor.od.com/public/redis:v4.0.14
[root@hdss7-200 ~]# docker image push harbor.od.com/public/redis:v4.0.14
2.2.2. 准備資源配置清單
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: redis
name: redis
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
name: redis
template:
metadata:
labels:
app: redis
name: redis
spec:
containers:
- name: redis
image: harbor.od.com/public/redis:v4.0.14
ports:
- containerPort: 6379
protocol: TCP
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: armory
spec:
ports:
- port: 6379
protocol: TCP
targetPort: 6379
selector:
app: redis
2.2.3. 應用資源配置清單
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/redis/depolyment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/redis/service.yaml
[root@hdss7-21 ~]# kubectl get pod -n armory -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
minio-6cb7db494b-hjqvh 1/1 Running 0 33m 172.7.22.5 hdss7-22.host.com <none> <none>
redis-5886797648-mxjbm 1/1 Running 0 7m57s 172.7.22.6 hdss7-22.host.com <none> <none>
[root@hdss7-21 ~]# telnet 172.7.22.6 6379
Trying 172.7.22.6...
Connected to 172.7.22.6.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
[root@hdss7-21 ~]# kubectl get svc -n armory -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
minio ClusterIP 192.168.139.95 <none> 80/TCP 33m app=minio
redis ClusterIP 192.168.70.230 <none> 6379/TCP 8m16s app=redis
[root@hdss7-21 ~]# telnet 192.168.70.230 6379
Trying 192.168.70.230...
Connected to 192.168.70.230.
Escape character is '^]'.
^]
telnet> quit
2.3. 部署Spinnaker-clouddrive
2.3.1. 准備鏡像
# 當前版本的鏡像比較老，可以考慮新版本進行嘗試
[root@hdss7-200 ~]# docker pull armory/spinnaker-clouddriver-slim:release-1.8.x-14c9664
[root@hdss7-200 ~]# docker image tag armory/spinnaker-clouddriver-slim:release-1.8.x-14c9664 harbor.od.com/public/spinnaker-clouddriver-slim:v1.8.x-14c9664
[root@hdss7-200 ~]# docker image push harbor.od.com/public/spinnaker-clouddriver-slim:v1.8.x-14c9664
2.3.2. 制作secret
[root@hdss7-21 ~]# cat minio-login.secret # minio 賬號密碼，后期會掛載到spinnakr中
[default]
aws_access_key_id=duduniao
aws_secret_access_key=duduniao12345
[root@hdss7-21 ~]# kubectl create secret generic credentials --from-file=credentials=minio-login.secret -n armory
2.3.3. 制作kube-config文件
# 簽發證書，注意CN和用戶名一致
[root@hdss7-200 certs]# cat spinnaker-csr.json
{
"CN": "spinnake",
"hosts": [
"10.4.7.10"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client spinnake-csr.json | cfssl-json -bare spinnake
[root@hdss7-200 certs]# ls spinnake* -l
-rw-r--r-- 1 root root 1037 Feb 15 13:58 spinnake.csr
-rw-r--r-- 1 root root 296 Feb 15 13:58 spinnake-csr.json
-rw------- 1 root root 1675 Feb 15 13:58 spinnake-key.pem
-rw-r--r-- 1 root root 1391 Feb 15 13:58 spinnake.pem
[root@hdss7-200 certs]# scp spinnake* ca.pem hdss7-21:~/spinnaker/
# kube-config文件也是用於通過apiserver來操作k8s集群創建資源使用，一般給予cluster-admin權限
[root@hdss7-21 spinnaker]# kubectl config set-cluster myk8s --certificate-authority=ca.pem --embed-certs=true --server=https://10.4.7.10:7443 --kubeconfig=config
[root@hdss7-21 spinnaker]# kubectl config set-credentials spinnake --client-certificate=spinnake.pem --client-key=spinnake-key.pem --embed-certs=true --kubeconfig=config
[root@hdss7-21 spinnaker]# kubectl config set-context myk8s-context --cluster=myk8s --user=spinnake --kubeconfig=config
[root@hdss7-21 spinnaker]# kubectl config use-context myk8s-context --kubeconfig=config
[root@hdss7-21 spinnaker]# kubectl create clusterrolebinding spinnake --clusterrole=cluster-admin --user=spinnake
# 測試kube-config是否可以用，注意：dashborad只能用service account登陸
[root@hdss7-200 ~]# scp hdss7-21:~/spinnaker/config /tmp/
[root@hdss7-200 ~]# cat /etc/profile
export KUBECONFIG=/tmp/config
[root@hdss7-200 ~]# kubectl get pod -n armory --kubeconfig=/tmp/config
NAME READY STATUS RESTARTS AGE
minio-6cb7db494b-hjqvh 1/1 Running 0 3h23m
redis-5886797648-mxjbm 1/1 Running 0 178m
[root@hdss7-200 ~]# rm -f /tmp/config
# 創建configmap配置，spinnaker將kube-config文件掛載到容器中使用
[root@hdss7-21 spinnaker]# kubectl create configmap default-kubeconfig --from-file=default-kubeconfig=config -n armory
[root@hdss7-21 ~]# rm -fr spinnaker
2.3.4. 資源配置清單
Spinnaker 的配置比較繁瑣，其中有一個default-config.yaml的configmap非常復雜，一般不需要修改：
# init-env.yaml
# 包括redis地址、對外的API接口域名等
apiVersion: v1
kind: ConfigMap
metadata:
name: init-env
namespace: armory
data:
API_HOST: http://spinnaker.od.com/api
ARMORY_ID: c02f0781-92f5-4e80-86db-0ba8fe7b8544
ARMORYSPINNAKER_CONF_STORE_BUCKET: armory-platform
ARMORYSPINNAKER_CONF_STORE_PREFIX: front50
ARMORYSPINNAKER_GCS_ENABLED: "false"
ARMORYSPINNAKER_S3_ENABLED: "true"
AUTH_ENABLED: "false"
AWS_REGION: us-east-1
BASE_IP: 127.0.0.1
CLOUDDRIVER_OPTS: -Dspring.profiles.active=armory,configurator,local
CONFIGURATOR_ENABLED: "false"
DECK_HOST: http://spinnaker.od.com
ECHO_OPTS: -Dspring.profiles.active=armory,configurator,local
GATE_OPTS: -Dspring.profiles.active=armory,configurator,local
IGOR_OPTS: -Dspring.profiles.active=armory,configurator,local
PLATFORM_ARCHITECTURE: k8s
REDIS_HOST: redis://redis:6379
SERVER_ADDRESS: 0.0.0.0
SPINNAKER_AWS_DEFAULT_REGION: us-east-1
SPINNAKER_AWS_ENABLED: "false"
SPINNAKER_CONFIG_DIR: /home/spinnaker/config
SPINNAKER_GOOGLE_PROJECT_CREDENTIALS_PATH: ""
SPINNAKER_HOME: /home/spinnaker
SPRING_PROFILES_ACTIVE: armory,configurator,local
# custom-config.yaml
# 該配置文件指定訪問k8s、harbor、minio、Jenkins的訪問方式
# 其中部分地址可以根據是否在k8s內部，和是否同一個名稱空間來選擇是否使用短域名
apiVersion: v1
kind: ConfigMap
metadata:
name: custom-config
namespace: armory
data:
clouddriver-local.yml: |
kubernetes:
enabled: true
accounts:
- name: spinnake
serviceAccount: false
dockerRegistries:
- accountName: harbor
namespace: []
namespaces:
- dev
- fat
- pro
kubeconfigFile: /opt/spinnaker/credentials/custom/default-kubeconfig
primaryAccount: spinnake
dockerRegistry:
enabled: true
accounts:
- name: harbor
requiredGroupMembership: []
providerVersion: V1
insecureRegistry: true
address: http://harbor.od.com
username: admin
password: Harbor12345
primaryAccount: harbor
artifacts:
s3:
enabled: true
accounts:
- name: armory-config-s3-account
apiEndpoint: http://minio
apiRegion: us-east-1
gcs:
enabled: false
accounts:
- name: armory-config-gcs-account
custom-config.json: ""
echo-configurator.yml: |
diagnostics:
enabled: true
front50-local.yml: |
spinnaker:
s3:
endpoint: http://minio
igor-local.yml: |
jenkins:
enabled: true
masters:
- name: jenkins-admin
address: http://jenkins.infra
username: admin
password: admin123
primaryAccount: jenkins-admin
nginx.conf: |
gzip on;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/vnd.ms-fontobject application/x-font-ttf font/opentype image/svg+xml image/x-icon;
server {
listen 80;
location / {
proxy_pass http://armory-deck/;
}
location /api/ {
proxy_pass http://armory-gate:8084/;
}
rewrite ^/login(.*)$ /api/login$1 last;
rewrite ^/auth(.*)$ /api/auth$1 last;
}
spinnaker-local.yml: |
services:
igor:
enabled: true
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: armory-clouddriver
name: armory-clouddriver
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
app: armory-clouddriver
template:
metadata:
annotations:
artifact.spinnaker.io/location: '"armory"'
artifact.spinnaker.io/name: '"armory-clouddriver"'
artifact.spinnaker.io/type: '"kubernetes/deployment"'
moniker.spinnaker.io/application: '"armory"'
moniker.spinnaker.io/cluster: '"clouddriver"'
labels:
app: armory-clouddriver
spec:
containers:
- name: armory-clouddriver
image: harbor.od.com/public/spinnaker-clouddriver-slim:v1.8.x-14c9664
command:
- bash
- -c
args:
# 腳本在default-config.yaml中
- bash /opt/spinnaker/config/default/fetch.sh && cd /home/spinnaker/config
&& /opt/clouddriver/bin/clouddriver
ports:
- containerPort: 7002
protocol: TCP
env:
- name: JAVA_OPTS
# 生產中調大到2048-4096M
value: -Xmx1024M
envFrom:
- configMapRef:
name: init-env
livenessProbe:
failureThreshold: 5
httpGet:
path: /health
port: 7002
scheme: HTTP
initialDelaySeconds: 600
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 5
httpGet:
path: /health
port: 7002
scheme: HTTP
initialDelaySeconds: 180
periodSeconds: 3
successThreshold: 5
timeoutSeconds: 1
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /etc/podinfo
name: podinfo
- mountPath: /home/spinnaker/.aws
name: credentials
- mountPath: /opt/spinnaker/credentials/custom
name: default-kubeconfig
- mountPath: /opt/spinnaker/config/default
name: default-config
- mountPath: /opt/spinnaker/config/custom
name: custom-config
volumes:
- configMap:
defaultMode: 420
name: default-kubeconfig
name: default-kubeconfig
- configMap:
defaultMode: 420
name: custom-config
name: custom-config
- configMap:
defaultMode: 420
name: default-config
name: default-config
- name: credentials
secret:
defaultMode: 420
secretName: credentials
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: podinfo
apiVersion: v1
kind: Service
metadata:
name: armory-clouddriver
namespace: armory
spec:
ports:
- port: 7002
protocol: TCP
targetPort: 7002
selector:
app: armory-clouddriver
2.3.5. 應用資源配置清單
[root@hdss7-21 spinnaker]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/spinnake/init-env.yaml
[root@hdss7-21 spinnaker]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/spinnake/custon-config.yaml
[root@hdss7-21 spinnaker]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/spinnake/default-config.yaml
[root@hdss7-21 spinnaker]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/spinnake/deployment.yaml
[root@hdss7-21 spinnaker]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/spinnake/service.yaml
[root@hdss7-21 spinnaker]# kubectl get svc -n armory | grep armory-clouddriver
armory-clouddriver ClusterIP 192.168.109.209 <none> 7002/TCP 11m
[root@hdss7-21 spinnaker]# kubectl get pod -n armory -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
armory-clouddriver-5f8b5f4cbb-2qfgf 1/1 Running 0 8m1s 172.7.21.5 hdss7-21.host.com <none> <none>
# 通過curl測試，發現一個現象：通過pod的IP或者service的域名都可以訪問health，但是通過svc的IP卻不行
[root@hdss7-21 spinnaker]# curl 172.7.21.5:7002/health
{"status":"UP","kubernetes":{"status":"UP"},"redisHealth":{"status":"UP","maxIdle":100,"minIdle":25,"numActive":0,"numIdle":4,"numWaiters":0},"dockerRegistry":{"status":"UP"},"diskSpace":{"status":"UP","total":53659832320,"free":43917455360,"threshold":10485760}}[root@hdss7-21 spinnake]#
[root@hdss7-21 spinnaker]# kubectl exec minio-6cb7db494b-hjqvh -n armory -- curl -s armory-clouddriver:7002/health
{"status":"UP","kubernetes":{"status":"UP"},"redisHealth":{"status":"UP","maxIdle":100,"minIdle":25,"numActive":0,"numIdle":4,"numWaiters":0},"dockerRegistry":{"status":"UP"},"diskSpace":{"status":"UP","total":53659832320,"free":43917455360,"threshold":10485760}}
2.4. 部署front50
2.4.1. 准備鏡像
[root@hdss7-200 ~]# docker pull armory/spinnaker-front50-slim:release-1.8.x-93febf2
[root@hdss7-200 ~]# docker image tag armory/spinnaker-front50-slim:release-1.8.x-93febf2 harbor.od.com/public/spinnaker-front50-slim:v1.8.x-93febf2
[root@hdss7-200 ~]# docker image push harbor.od.com/public/spinnaker-front50-slim:v1.8.x-93febf2
2.4.2. 准備資源配置清單
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: armory-front50
name: armory-front50
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
app: armory-front50
template:
metadata:
annotations:
artifact.spinnaker.io/location: '"armory"'
artifact.spinnaker.io/name: '"armory-front50"'
artifact.spinnaker.io/type: '"kubernetes/deployment"'
moniker.spinnaker.io/application: '"armory"'
moniker.spinnaker.io/cluster: '"front50"'
labels:
app: armory-front50
spec:
containers:
- name: armory-front50
image: harbor.od.com/public/spinnaker-front50-slim:v1.8.x-93febf2
command:
- bash
- -c
args:
- bash /opt/spinnaker/config/default/fetch.sh && cd /home/spinnaker/config
&& /opt/front50/bin/front50
ports:
- containerPort: 8080
protocol: TCP
env:
- name: JAVA_OPTS
# 生產中給大一些，本實驗一開始給了512M，啟動后運行一會就宕了
value: -javaagent:/opt/front50/lib/jamm-0.2.5.jar -Xmx1024M
envFrom:
- configMapRef:
name: init-env
livenessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 600
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 180
periodSeconds: 5
successThreshold: 8
timeoutSeconds: 1
volumeMounts:
- mountPath: /etc/podinfo
name: podinfo
- mountPath: /home/spinnaker/.aws
name: credentials
- mountPath: /opt/spinnaker/config/default
name: default-config
- mountPath: /opt/spinnaker/config/custom
name: custom-config
volumes:
- configMap:
defaultMode: 420
name: custom-config
name: custom-config
- configMap:
defaultMode: 420
name: default-config
name: default-config
- name: credentials
secret:
defaultMode: 420
secretName: credentials
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: podinfo
apiVersion: v1
kind: Service
metadata:
name: armory-front50
namespace: armory
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
app: armory-front50
2.4.2. 應用資源配置清單
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/front50/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/front50/service.yaml
[root@hdss7-21 ~]# kubectl exec minio-6cb7db494b-hjqvh -n armory -- curl -s 'http://armory-front50:8080/health'
{"status":"UP"}
2.5. 部署Orca
2.5.1. 准備鏡像
[root@hdss7-200 ~]# docker pull armory/spinnaker-orca-slim:release-1.8.x-de4ab55
[root@hdss7-200 ~]# docker image tag armory/spinnaker-orca-slim:release-1.8.x-de4ab55 harbor.od.com/public/spinnaker-orca-slim:v1.8.x-de4ab55
[root@hdss7-200 ~]# docker image push harbor.od.com/public/spinnaker-orca-slim:v1.8.x-de4ab55
2.5.2. 准備資源配置清單
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: armory-orca
name: armory-orca
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
app: armory-orca
template:
metadata:
annotations:
artifact.spinnaker.io/location: '"armory"'
artifact.spinnaker.io/name: '"armory-orca"'
artifact.spinnaker.io/type: '"kubernetes/deployment"'
moniker.spinnaker.io/application: '"armory"'
moniker.spinnaker.io/cluster: '"orca"'
labels:
app: armory-orca
spec:
containers:
- name: armory-orca
image: harbor.od.com/public/spinnaker-orca-slim:v1.8.x-de4ab55
command:
- bash
- -c
args:
- bash /opt/spinnaker/config/default/fetch.sh && cd /home/spinnaker/config
&& /opt/orca/bin/orca
ports:
- containerPort: 8083
protocol: TCP
env:
- name: JAVA_OPTS
value: -Xmx512M
envFrom:
- configMapRef:
name: init-env
livenessProbe:
failureThreshold: 5
httpGet:
path: /health
port: 8083
scheme: HTTP
initialDelaySeconds: 600
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8083
scheme: HTTP
initialDelaySeconds: 180
periodSeconds: 3
successThreshold: 5
timeoutSeconds: 1
volumeMounts:
- mountPath: /etc/podinfo
name: podinfo
- mountPath: /opt/spinnaker/config/default
name: default-config
- mountPath: /opt/spinnaker/config/custom
name: custom-config
volumes:
- configMap:
defaultMode: 420
name: custom-config
name: custom-config
- configMap:
defaultMode: 420
name: default-config
name: default-config
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: podinfo
apiVersion: v1
kind: Service
metadata:
name: armory-orca
namespace: armory
spec:
ports:
- port: 8083
protocol: TCP
targetPort: 8083
selector:
app: armory-orca
2.5.3. 應用資源配置清單
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/orca/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/orca/service.yaml
[root@hdss7-21 ~]# kubectl exec minio-6cb7db494b-hjqvh -n armory -- curl -s 'http://armory-orca:8083/health'
{"status":"UP"}
2.6. 部署Echo
2.6.1. 准備鏡像
[root@hdss7-200 ~]# docker pull armory/echo-armory:c36d576-release-1.8.x-617c567
[root@hdss7-200 ~]# docker image tag armory/echo-armory:c36d576-release-1.8.x-617c567 harbor.od.com/public/echo-armory:v1.8.x-617c567
[root@hdss7-200 ~]# docker image push harbor.od.com/public/echo-armory:v1.8.x-617c567
2.6.2. 准備資源配置清單
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: armory-echo
name: armory-echo
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
app: armory-echo
template:
metadata:
annotations:
artifact.spinnaker.io/location: '"armory"'
artifact.spinnaker.io/name: '"armory-echo"'
artifact.spinnaker.io/type: '"kubernetes/deployment"'
moniker.spinnaker.io/application: '"armory"'
moniker.spinnaker.io/cluster: '"echo"'
labels:
app: armory-echo
spec:
containers:
- name: armory-echo
image: harbor.od.com/public/echo-armory:v1.8.x-617c567
command:
- bash
- -c
args:
- bash /opt/spinnaker/config/default/fetch.sh && cd /home/spinnaker/config
&& /opt/echo/bin/echo
ports:
- containerPort: 8089
protocol: TCP
env:
- name: JAVA_OPTS
value: -javaagent:/opt/echo/lib/jamm-0.2.5.jar -Xmx512M
envFrom:
- configMapRef:
name: init-env
livenessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8089
scheme: HTTP
initialDelaySeconds: 600
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8089
scheme: HTTP
initialDelaySeconds: 180
periodSeconds: 3
successThreshold: 5
timeoutSeconds: 1
volumeMounts:
- mountPath: /etc/podinfo
name: podinfo
- mountPath: /opt/spinnaker/config/default
name: default-config
- mountPath: /opt/spinnaker/config/custom
name: custom-config
imagePullSecrets:
- name: harbor
volumes:
- configMap:
defaultMode: 420
name: custom-config
name: custom-config
- configMap:
defaultMode: 420
name: default-config
name: default-config
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: podinfo
apiVersion: v1
kind: Service
metadata:
name: armory-echo
namespace: armory
spec:
ports:
- port: 8089
protocol: TCP
targetPort: 8089
selector:
app: armory-echo
2.6.3. 應用資源配置清單
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/echo/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/echo/service.yaml
[root@hdss7-21 ~]# kubectl exec minio-6cb7db494b-hjqvh -n armory -- curl -s 'http://armory-echo:8089/health'
{"status":"UP"}
2.7. 部署igor
2.7.1. 鏡像准備
[root@hdss7-200 ~]# docker image pull armory/spinnaker-igor-slim:release-1.8-x-new-install-healthy-ae2b329
[root@hdss7-200 ~]# docker image tag armory/spinnaker-igor-slim:release-1.8-x-new-install-healthy-ae2b329 harbor.od.com/public/igor:v1.8-x-ae2b329
[root@hdss7-200 ~]# docker image push harbor.od.com/public/igor:v1.8-x-ae2b329
2.7.2. 准備資源配置清單
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: armory-igor
name: armory-igor
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
app: armory-igor
template:
metadata:
annotations:
artifact.spinnaker.io/location: '"armory"'
artifact.spinnaker.io/name: '"armory-igor"'
artifact.spinnaker.io/type: '"kubernetes/deployment"'
moniker.spinnaker.io/application: '"armory"'
moniker.spinnaker.io/cluster: '"igor"'
labels:
app: armory-igor
spec:
containers:
- name: armory-igor
image: harbor.od.com/public/igor:v1.8-x-ae2b329
command:
- bash
- -c
args:
- bash /opt/spinnaker/config/default/fetch.sh && cd /home/spinnaker/config
&& /opt/igor/bin/igor
ports:
- containerPort: 8088
protocol: TCP
env:
- name: IGOR_PORT_MAPPING
value: -8088:8088
- name: JAVA_OPTS
value: -Xmx512M
envFrom:
- configMapRef:
name: init-env
livenessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8088
scheme: HTTP
initialDelaySeconds: 600
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8088
scheme: HTTP
initialDelaySeconds: 180
periodSeconds: 5
successThreshold: 5
timeoutSeconds: 1
volumeMounts:
- mountPath: /etc/podinfo
name: podinfo
- mountPath: /opt/spinnaker/config/default
name: default-config
- mountPath: /opt/spinnaker/config/custom
name: custom-config
securityContext:
runAsUser: 0
volumes:
- configMap:
defaultMode: 420
name: custom-config
name: custom-config
- configMap:
defaultMode: 420
name: default-config
name: default-config
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: podinfo
apiVersion: v1
kind: Service
metadata:
name: armory-igor
namespace: armory
spec:
ports:
- port: 8088
protocol: TCP
targetPort: 8088
selector:
app: armory-igor
2.7.3. 應用資源配置清單
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/igor/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/igor/service.yaml
[root@hdss7-21 ~]# kubectl exec minio-6cb7db494b-hjqvh -n armory -- curl -s 'http://armory-igor:8088/health'
{"status":"UP"}
2.8. 部署gate
2.8.1. 准備鏡像
[root@hdss7-200 ~]# docker pull armory/gate-armory:dfafe73-release-1.8.x-5d505ca
[root@hdss7-200 ~]# docker image tag armory/gate-armory:dfafe73-release-1.8.x-5d505ca harbor.od.com/public/gate-armory:v1.8.x-5d505ca
[root@hdss7-200 ~]# docker image push harbor.od.com/public/gate-armory:v1.8.x-5d505ca
2.8.2. 准備資源配置清單
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: armory-gate
name: armory-gate
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
app: armory-gate
template:
metadata:
annotations:
artifact.spinnaker.io/location: '"armory"'
artifact.spinnaker.io/name: '"armory-gate"'
artifact.spinnaker.io/type: '"kubernetes/deployment"'
moniker.spinnaker.io/application: '"armory"'
moniker.spinnaker.io/cluster: '"gate"'
labels:
app: armory-gate
spec:
containers:
- name: armory-gate
image: harbor.od.com/public/gate-armory:v1.8.x-5d505ca
command:
- bash
- -c
args:
- bash /opt/spinnaker/config/default/fetch.sh gate && cd /home/spinnaker/config
&& /opt/gate/bin/gate
ports:
- containerPort: 8084
name: gate-port
protocol: TCP
- containerPort: 8085
name: gate-api-port
protocol: TCP
env:
- name: GATE_PORT_MAPPING
value: -8084:8084
- name: GATE_API_PORT_MAPPING
value: -8085:8085
- name: JAVA_OPTS
value: -Xmx512M
envFrom:
- configMapRef:
name: init-env
livenessProbe:
exec:
command:
- /bin/bash
- -c
- wget -O - http://localhost:8084/health || wget -O - https://localhost:8084/health
failureThreshold: 5
initialDelaySeconds: 600
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
exec:
command:
- /bin/bash
- -c
- wget -O - http://localhost:8084/health?checkDownstreamServices=true&downstreamServices=true
|| wget -O - https://localhost:8084/health?checkDownstreamServices=true&downstreamServices=true
failureThreshold: 3
initialDelaySeconds: 180
periodSeconds: 5
successThreshold: 10
timeoutSeconds: 1
volumeMounts:
- mountPath: /etc/podinfo
name: podinfo
- mountPath: /opt/spinnaker/config/default
name: default-config
- mountPath: /opt/spinnaker/config/custom
name: custom-config
volumes:
- configMap:
defaultMode: 420
name: custom-config
name: custom-config
- configMap:
defaultMode: 420
name: default-config
name: default-config
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: podinfo
apiVersion: v1
kind: Service
metadata:
name: armory-gate
namespace: armory
spec:
ports:
- name: gate-port
port: 8084
protocol: TCP
targetPort: 8084
- name: gate-api-port
port: 8085
protocol: TCP
targetPort: 8085
selector:
app: armory-gate
2.8.3. 應用資源配清單
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/gate/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/gate/service.yaml
[root@hdss7-21 ~]# kubectl exec minio-6cb7db494b-hjqvh -n armory -- curl -s 'http://armory-gate:8084/health?checkDownstreamServices=true&downstreamServices=true'
{"status":"UP"}
2.9. 部署deck
2.9.1. 准備鏡像
[root@hdss7-200 ~]# docker image pull armory/deck-armory:d4bf0cf-release-1.8.x-0a33f94
[root@hdss7-200 ~]# docker image tag armory/deck-armory:d4bf0cf-release-1.8.x-0a33f94 harbor.od.com/public/deck-armory:v1.8.x-0a33f94
[root@hdss7-200 ~]# docker image push harbor.od.com/public/deck-armory:v1.8.x-0a33f94
2.9.2. 准備資源配置清單
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: armory-deck
name: armory-deck
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
app: armory-deck
template:
metadata:
annotations:
artifact.spinnaker.io/location: '"armory"'
artifact.spinnaker.io/name: '"armory-deck"'
artifact.spinnaker.io/type: '"kubernetes/deployment"'
moniker.spinnaker.io/application: '"armory"'
moniker.spinnaker.io/cluster: '"deck"'
labels:
app: armory-deck
spec:
containers:
- name: armory-deck
image: harbor.od.com/public/deck-armory:v1.8.x-0a33f94
command:
- bash
- -c
args:
- bash /opt/spinnaker/config/default/fetch.sh && /entrypoint.sh
ports:
- containerPort: 9000
protocol: TCP
envFrom:
- configMapRef:
name: init-env
livenessProbe:
failureThreshold: 3
httpGet:
path: /
port: 9000
scheme: HTTP
initialDelaySeconds: 180
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 5
httpGet:
path: /
port: 9000
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 3
successThreshold: 5
timeoutSeconds: 1
volumeMounts:
- mountPath: /etc/podinfo
name: podinfo
- mountPath: /opt/spinnaker/config/default
name: default-config
- mountPath: /opt/spinnaker/config/custom
name: custom-config
volumes:
- configMap:
defaultMode: 420
name: custom-config
name: custom-config
- configMap:
defaultMode: 420
name: default-config
name: default-config
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: podinfo
apiVersion: v1
kind: Service
metadata:
name: armory-deck
namespace: armory
spec:
ports:
- port: 80
protocol: TCP
targetPort: 9000
selector:
app: armory-deck
2.9.3. 應用資源配置清單
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/deck/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/deck/service.yaml
[root@hdss7-21 ~]# kubectl exec minio-6cb7db494b-hjqvh -n armory -- curl -Is 'http://armory-deck'
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Sat, 15 Feb 2020 10:07:24 GMT
Content-Type: text/html
Content-Length: 22031
Last-Modified: Tue, 17 Jul 2018 17:42:20 GMT
Connection: keep-alive
ETag: "5b4e2a7c-560f"
Accept-Ranges: bytes
2.10. 部署Nginx
2.10.1. 准備鏡像
[root@hdss7-200 ~]# docker image pull nginx:1.12.2
[root@hdss7-200 ~]# docker image tag nginx:1.12.2 harbor.od.com/public/nginx:v1.12.2
[root@hdss7-200 ~]# docker image push harbor.od.com/public/nginx:v1.12.2
2.10.2. 准備資源配置清單
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: armory-nginx
name: armory-nginx
namespace: armory
spec:
replicas: 1
selector:
matchLabels:
app: armory-nginx
template:
metadata:
annotations:
artifact.spinnaker.io/location: '"armory"'
artifact.spinnaker.io/name: '"armory-nginx"'
artifact.spinnaker.io/type: '"kubernetes/deployment"'
moniker.spinnaker.io/application: '"armory"'
moniker.spinnaker.io/cluster: '"nginx"'
labels:
app: armory-nginx
spec:
containers:
- name: armory-nginx
image: harbor.od.com/public/nginx:v1.12.2
command:
- bash
- -c
args:
- bash /opt/spinnaker/config/default/fetch.sh nginx && nginx -g 'daemon off;'
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 8085
name: api
protocol: TCP
livenessProbe:
failureThreshold: 3
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 180
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 3
successThreshold: 5
timeoutSeconds: 1
volumeMounts:
- mountPath: /opt/spinnaker/config/default
name: default-config
- mountPath: /etc/nginx/conf.d
name: custom-config
volumes:
- configMap:
defaultMode: 420
name: custom-config
name: custom-config
- configMap:
defaultMode: 420
name: default-config
name: default-config
apiVersion: v1
kind: Service
metadata:
name: armory-nginx
namespace: armory
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
- name: https
port: 443
protocol: TCP
targetPort: 443
- name: api
port: 8085
protocol: TCP
targetPort: 8085
selector:
app: armory-nginx
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
labels:
app: spinnaker
web: spinnaker.od.com
name: armory-nginx
namespace: armory
spec:
rules:
- host: spinnaker.od.com
http:
paths:
- backend:
serviceName: armory-nginx
servicePort: 80
2.10.3. 應用資源配置清單
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/nginx/deployment.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/nginx/service.yaml
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/devops/armory/nginx/ingress.yaml
