kubernetes實戰-配置中心（四）分環境使用apollo配置中心

要進行分環境，需要將現有實驗環境進行拆分

portal服務，可以各個環境共用，但是apollo-adminservice和apollo-configservice必須要分開。

1、zk環境拆分為test和prod環境

添加dns解析：

# vi /var/named/od.com.zone

zk-test.od.com       A       10.4.7.11

zk-prod.od.com      A       10.4.7.12

2、namespace 分環境，創建test 和prod

# kubectl create ns test
# kubectl create ns prod

創建secret：

# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=harbor --docker-password=Harbor12345 -n test

# kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=harbor --docker-password=Harbor12345 -n prod

3、數據庫進行拆分，因實驗資源有限，故使用分庫的形式模擬分環境

修改數據庫初始化腳本，分別創建ApolloConfigTestDB和ApolloConfigProdDB

 

 

 

 

 

 修改數據庫中eureka的地址，這里用到了兩個新的域名，自行在bind9中添加解析

> update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.Key="eureka.service.url";


> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

> update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.Key="eureka.service.url";


> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

 

 

修改portal數據，支持fat和pro環境：

> update ServerConfig set Value='fat,pro' where Id=1;

修改portal的cm資源配置清單：

# vi /data/k8s-yaml/apollo-portal/cm.yaml

 

 

 

# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/cm.yaml

 

 

分別創建修改兩個環境的資源配置文件：

test：

# cd /data/k8s-yaml

# mkdir -p test/{apollo-adminservice,apollo-configservice,dubbo-demo-server,dubbo-demo-consumer}


# mkdir -p prod/{apollo-adminservice,apollo-configservice,dubbo-demo-server,dubbo-demo-consumer}

將之前的資源配置清單cp到對應環境的目錄中，進行修改：

# cd  test/apollo-configservice/

# cp ../../apollo-configservice/* ./

1、cm.yaml 修改ns，數據庫庫名，eureka地址

apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-configservice-cm
  namespace: test
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config-test.od.com/eureka
  app.properties: |
    appId=100003171

2、dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-configservice
  namespace: test 
  labels: 
    name: apollo-configservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-configservice
  template:
    metadata:
      labels: 
        app: apollo-configservice 
        name: apollo-configservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-configservice-cm
      containers:
      - name: apollo-configservice
        image: harbor.od.com/infra/apollo-configservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-configservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

3、svc.yaml

kind: Service
apiVersion: v1
metadata: 
  name: apollo-configservice
  namespace: test 
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: apollo-configservice

4、ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-configservice
  namespace: test 
spec:
  rules:
  - host: config-test.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-configservice
          servicePort: 8080

 

 

 

 服務已經注冊進來了

接下來部署apollo-adminservice

修改apollo-adminservice的資源配置清單：

# cd /data/k8s-yaml/test/apollo-adminservice

# cp ../../apollo-adminservice/* ./

1、cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-adminservice-cm
  namespace: test
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config-test.od.com/eureka
  app.properties: |
    appId=100003172

2、dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-adminservice
  namespace: test 
  labels: 
    name: apollo-adminservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-adminservice
  template:
    metadata:
      labels: 
        name: apollo-adminservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-adminservice-cm
      containers:
      - name: apollo-adminservice
        image: harbor.od.com/infra/apollo-adminservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-adminservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

應用資源配置清單：

# kubectl apply -f http://k8s-yaml.od.com/test/apollo-adminservice/cm.yaml
# kubectl apply -f http://k8s-yaml.od.com/test/apollo-adminservice/dp.yaml

 

 

接下來部署prod環境的apollo-configservice，還是一樣的套路：

# cd ../../prod/apollo-configservice/
# cp ../../apollo-configservice/* ./

修改資源配置清單：

1、cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-configservice-cm
  namespace: prod
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config-prod.od.com/eureka
  app.properties: |
    appId=100003171

2、dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-configservice
  namespace: prod 
  labels: 
    name: apollo-configservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-configservice
  template:
    metadata:
      labels: 
        app: apollo-configservice 
        name: apollo-configservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-configservice-cm
      containers:
      - name: apollo-configservice
        image: harbor.od.com/infra/apollo-configservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-configservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

3、svc.yaml

kind: Service
apiVersion: v1
metadata:
  name: apollo-configservice
  namespace: prod
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app: apollo-configservice

4、ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: apollo-configservice
  namespace: prod
spec:
  rules:
  - host: config-prod.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: apollo-configservice
          servicePort: 8080

應用資源配置清單：

# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/cm.yaml
# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/dp.yaml
# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/svc.yaml
# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-configservice/ingress.yaml

修改apollo-adminservice的資源配置清單：

# cd ../apollo-adminservice/
# cp ../../apollo-adminservice/* ./

1、cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-adminservice-cm
  namespace: prod
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config-prod.od.com/eureka
  app.properties: |
    appId=100003172

2、dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-adminservice
  namespace: prod 
  labels: 
    name: apollo-adminservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-adminservice
  template:
    metadata:
      labels: 
        name: apollo-adminservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-adminservice-cm
      containers:
      - name: apollo-adminservice
        image: harbor.od.com/infra/apollo-adminservice:v1.5.1
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-adminservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

應用資源配置清單：

# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-adminservice/cm.yaml
# kubectl apply -f http://k8s-yaml.od.com/prod/apollo-adminservice/dp.yaml

兩個服務都已經注冊進來了，刪除portal數據庫中存儲的關於之前項目的配置，接下來啟動portal項目：

# mysql -uroot -p

> use ApolloPortalDB ;

> truncate table App;
> truncate table AppNamespace;

 

 

 

打開portal.od.com驗證，並且創建兩個項目：

首先創建dubbo-demo-service

 

 

 添加配置：兩個環境都添加上：注意連接地址一個是test.od.com,一個是prod.od.com

 

 

 接下來創建dubbo-demo-web項目：同樣是兩個環境都發布，注意一個是test.od.com,一個是prod.od.com

 

 

 

接下來交付dubbo服務分環境交付：

同樣操作，修改之前項目的資源配置清單：

# cd /data/k8s-yaml/test/dubbo-demo-server

# cp ../../dubbo-server/* ./

1、dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-service
  namespace: test 
  labels: 
    name: dubbo-demo-service
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-service
  template:
    metadata:
      labels: 
        app: dubbo-demo-service
        name: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:apollo_191211_1916
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://config-test.od.com
        - name: JAR_BALL
          value: dubbo-server.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

應用資源配置清單：

# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-server/dp.yaml

 

 

 接下來交付dubbo-demo-consumer:

# cd /data/k8s-yaml/test/dubbo-demo-consumer

# cp ../../dubbo-consumer/* ./

1、dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: test 
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-web:apollo_191212_1715
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://config-test.od.com
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

2、svc.yaml

kind: Service
apiVersion: v1
metadata:
  name: dubbo-demo-consumer
  namespace: test
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app: dubbo-demo-consumer

3、ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: test
spec:
  rules:
  - host: demo-test.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: dubbo-demo-consumer
          servicePort: 8080

這里會用到兩個新的域名，添加解析：

 

 

 應用test環境的dubbo-consumer資源配置清單：

# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/dp.yaml
# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/svc.yaml
# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/ingress.yaml

 

 

 

接下來交付prod環境的dubbo-demo-server和dubbo-demo-consumer服務：

同樣套路，復制資源配置清單，然后修改成prod環境：

cd /data/k8s-yaml/prod/dubbo-demo-server

# cp ../../dubbo-server/* ./

1、dp.yaml

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-service
  namespace: prod 
  labels: 
    name: dubbo-demo-service
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-service
  template:
    metadata:
      labels: 
        app: dubbo-demo-service
        name: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:apollo_191211_1916
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://config-prod.od.com
        - name: JAR_BALL
          value: dubbo-server.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

應用資源配置清單：

# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-server/dp.yaml

 

 

 接下來做dubbo-demo-consumer的資源配置清單：

# cd /data/k8s-yaml/prod/dubbo-demo-consumer

# cp ../../dubbo-consumer/* ./

1、dp.yaml   http://apollo-configservice:8080  這里可以直接使用svc資源名稱調用，不用走ingress，因為svc資源只在當前namespace中生效。

 

 

 

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: prod 
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-web:apollo_191212_1715
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=pro -Dapollo.meta=http://apollo-configservice:8080
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

 

2、svc.yaml

kind: Service
apiVersion: v1
metadata: 
  name: dubbo-demo-consumer
  namespace: prod 
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: dubbo-demo-consumer

 

3、ingress.yaml

kind: Service
apiVersion: v1
metadata: 
  name: dubbo-demo-consumer
  namespace: prod 
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: dubbo-demo-consumer
[root@hdss7-200 dubbo-demo-consumer]# cat ingress.yaml 
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: dubbo-demo-consumer
  namespace: prod 
spec:
  rules:
  - host: demo-prod.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: dubbo-demo-consumer
          servicePort: 8080

應用資源配置清單：

# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/dp.yaml
# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/svc.yaml
# kubectl apply -f http://k8s-yaml.od.com/prod/dubbo-demo-consumer/ingress.yaml

 

 

 

訪問兩個環境的不同url：

http://demo-test.od.com

 

 

 http://demo-prod.od.com

模擬發版：

使用jenkins構建新的鏡像：

 

 構建成功，然后我們在測試環境發布此版本鏡像：

 

 修改測試環境的dp.yaml

 

 應用修改后的資源配置清單：

# kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/dp.yaml

 

 

已經成功將新代碼上線到test環境，接下來上線到prod環境

同樣修改prod環境的dp.yaml,並且應用該資源配置清單：

 

 已經上線到生產環境，這樣一套完整的分環境使用apollo配置中心發布流程已經可以使用了，並且真正做到了一次構建，多平台使用。