采用Operator-sdk輕松將helm chart轉為Operator

去年就接觸Operator,從Oracle發布的WebLogic Operator到mySQL Operator,構建的源碼一大堆，但感覺一直缺少合適的開發框架能夠避免復雜性快速生成，

隨着技術的日益成熟，目前基於helm Operator輕松解決快速安裝的問題，值得嘗試一下。

下圖是Operator框架的成熟度模型，基於不同的階段，采用不同的技術滿足全生命周期管理的需求

本文主要是以tomcat為例來進行快速的生成一個tomcat Operator.

1.環境准備

• 安裝go

下載安裝，參考 https://golang.org/doc/install?download=go1.11.5.linux-amd64.tar.gz，不再詳訴。

• 安裝編譯環境

yum -y install gcc automake autoconf libtool make

• 安裝Operator CLI

mkdir -p $GOPATH/src/github.com/operator-framework
cd $GOPATH/src/github.com/operator-framework
git clone https://github.com/operator-framework/operator-sdk
cd operator-sdk
git checkout master
make install

 

2.構建Tomcat Operator

• 新建項目

cd /usr/local/go/src/github.com/operator-framework/operator-sdk
operator-sdk new tomcat-operator --cluster-scoped --api-version=example.com/v1alpha1 --kind=Tomcat --type=helm

這里建立的是cluster-scope,意思是全集群都可以用，缺省是當前的命名空間，看一下有什么東西

[root@master operator-sdk]# tree tomcat-operator
tomcat-operator
├── build
│   └── Dockerfile
├── deploy
│   ├── crds
│   │   ├── example_v1alpha1_tomcat_crd.yaml
│   │   └── example_v1alpha1_tomcat_cr.yaml
│   ├── operator.yaml
│   ├── role_binding.yaml
│   ├── role.yaml
│   └── service_account.yaml
├── helm-charts
│   └── tomcat
│       ├── charts
│       ├── Chart.yaml
│       ├── templates
│       │   ├── deployment.yaml
│       │   ├── _helpers.tpl
│       │   ├── ingress.yaml
│       │   ├── NOTES.txt
│       │   ├── service.yaml
│       │   └── tests
│       │       └── test-connection.yaml
│       └── values.yaml
└── watches.yaml

 

• 客戶化Operator的邏輯

主要是針對templates下面的deployment.yaml進行修改，因為缺省是以nginx為模板來做的，所以端口都是80,需要修改成8080

主要的修改就是deployment.yaml和values.yaml, 當然如果需要部署多個服務，同時多個服務由一定的依賴關系可以在helm中進行實現，鏡像及版本的修改在values.yaml里面，我貼一段

[root@master tomcat]# cat values.yaml 
# Default values for tomcat.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: registry.example.com/tomcat
  tag: 8-slim
  pullPolicy: IfNotPresent

nameOverride: ""
fullnameOverride: ""

service:
  type: ClusterIP
  port: 8080

ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  paths: []

我就修改了image和端口部分。

 example_v1alpha1_tomcat_cr.yaml這個文件主要用於部署tomcat類型的實例的yaml文件，

可以修改deploy/crds/example_v1alpha1_tomcat_cr.yaml,部署多個實例

[root@master crds]# cat example_v1alpha1_tomcat_cr.yaml
apiVersion: example.com/v1alpha1
kind: Tomcat
metadata:
  name: example-tomcat
spec:
  # Default values copied from <project_dir>/helm-charts/tomcat/values.yaml
  
  # Default values for tomcat.
  # This is a YAML-formatted file.
  # Declare variables to be passed into your templates.
  
  replicaCount: 2
  
  image:
    repository: registry.example.com/tomcat
    tag: 8-slim
    pullPolicy: IfNotPresent
  
  nameOverride: ""
  fullnameOverride: ""

 打開可以看到，基本就是指定了Kind為Tomcat,下面具體的值都可以從values.yaml中拷貝，並且可以覆蓋values.yaml的值。

 

3.部署Operator到集群

• 部署CRD

oc create -f deploy/crds/example_v1alpha1_nginx_crd.yaml

Openshift需要知道客戶化的資源定義，這個定義就通過這個腳本，指定了watch.

• 生成Operator的鏡像

首先部署的時候是基於build目錄下的Dockerfile,因為訪問不到，所以做了個跳轉，把Dockerfile的quay.io/operator-framework/helm-operator:v0.5.0路徑修改掉

[root@master build]# cat Dockerfile 
FROM docker.io/ericnie2017/helm-operator:latest

COPY helm-charts/ ${HOME}/helm-charts/
COPY watches.yaml ${HOME}/watches.yaml

然后運行

operator-sdk build registry.example.com/example/tomcat-operator:v0.0.1

[root@master tomcat-operator]# operator-sdk build registry.example.com/example/tomcat-operator:v0.0.1
INFO[0000] Building Docker image registry.example.com/example/tomcat-operator:v0.0.1 
Sending build context to Docker daemon 111.1 kB
Step 1/3 : FROM docker.io/ericnie2017/helm-operator:latest
 ---> f0d56774da3e
Step 2/3 : COPY helm-charts/ ${HOME}/helm-charts/
 ---> 9f77f7fba44d
Removing intermediate container efd44d601b0a
Step 3/3 : COPY watches.yaml ${HOME}/watches.yaml
 ---> 7469e31336af
Removing intermediate container 73189235ec15
Successfully built 7469e31336af
INFO[0001] Operator build complete.    

build語句會把我們定制的Operator生成一個鏡像，運行完push到鏡像倉庫讓全集群可以訪問。

[root@master tomcat-operator]# docker push registry.example.com/example/tomcat-operator:v0.0.1 
The push refers to a repository [registry.example.com/example/tomcat-operator]
ae10451a67a5: Pushed 
bebcddc5922f: Pushed 
e256e39f5897: Pushed 
d724046711d4: Pushed 
903dc29d7cf3: Pushed 
e79522dce35e: Pushed 
v0.0.1: digest: sha256:a439041a9de91f0fee04f4cd15c554d8a03ec37a286760415b015cbdce7f4315 size: 1569

• 建立相關的角色，權限和CRD的信息

執行幾個sed操作用於對生成模板的替換操作。

[root@master tomcat-operator]# sed -i 's|REPLACE_IMAGE|registry.example.com/example/tomcat-operator:v0.0.1|g' deploy/operator.yaml
[root@master tomcat-operator]# oc config view --minify -o jsonpath='{.contexts[0].context.namespace}'
default

[root@master tomcat-operator]# sed -i "s|REPLACE_NAMESPACE|default|g" deploy/role_binding.yaml

 

一切就緒，開始創建

oc create -f deploy/service_account.yaml
oc create -f deploy/role.yaml
oc create -f deploy/role_binding.yaml
oc create -f deploy/operator.yaml

因為openshift自己的權限要求比較嚴格，干脆直接加成集群管理員了。

[root@master tomcat-operator]# oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:default:tomcat-operator 
cluster role "cluster-admin" added: "system:serviceaccount:default:tomcat-operator"

查看一下，已經創建起來了。這時這個Operator類型就作為一個Pod在容器內運行。

[root@master crds]# oc get deployment
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
tomcat-operator   1         1         1            1           30m

 

4.創建和刪除tomcat Operator實例

建立實例，先查看一下這個創建的yaml文件，Kind就是我們指定的類型Tomcat，而下面的值就是從values.yaml而來，可以覆蓋也可以不用覆蓋。

[root@master crds]# cat example_v1alpha1_tomcat_cr.yaml
apiVersion: example.com/v1alpha1
kind: Tomcat
metadata:
  name: example-tomcat
spec:
  # Default values copied from <project_dir>/helm-charts/tomcat/values.yaml
  
  # Default values for tomcat.
  # This is a YAML-formatted file.
  # Declare variables to be passed into your templates.
  
  replicaCount: 2
  
  image:
    repository: registry.example.com/tomcat
    tag: 8-slim
    pullPolicy: IfNotPresent
  
  nameOverride: ""
  fullnameOverride: ""
  
  service:
    type: ClusterIP
    port: 8080
  
  ingress:
    enabled: false
    annotations: {}
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
    paths: []
    hosts:
      - chart-example.local
    tls: []
    #  - secretName: chart-example-tls
    #    hosts:
    #      - chart-example.local
  
  resources: {}
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    # limits:
    #  cpu: 100m
    #  memory: 128Mi
    # requests:
    #  cpu: 100m
    #  memory: 128Mi
  
  nodeSelector: {}
  tolerations: []
  
  affinity: {}
  

 

 

 

[root@master crds]# oc create  -f example_v1alpha1_tomcat_cr.yaml
tomcat.example.com/example-tomcat created
[root@master crds]# oc get pods 
NAME                                                        READY     STATUS    RESTARTS   AGE
docker-registry-1-gl8jh                                     1/1       Running   6          18d
example-tomcat-1xvukmzvgn1tijep2w61xgm56-69457d7456-fm49d   0/1       Running   0          11s
example-tomcat-1xvukmzvgn1tijep2w61xgm56-69457d7456-twjhk   0/1       Running   0          11s
registry-console-1-6m4cq                                    1/1       Running   2          8d
router-3-7gx4b                                              1/1       Running   2          9d
tomcat-operator-75dc656956-hhnfd                            1/1       Running   0          27m

看到已經運行起來了，但是沒有ready,沒有ready的原因是readness和liveness的端口在deployment.yaml里面設置錯了，沒有修改成8080.

進去查看Pod日志，已經正常運行。

 

查看一下自定義對象，有一個example-tomcat，包含了2個pod

[root@master crds]# oc get Tomcat
NAME             AGE
example-tomcat   10s

 

刪除實例

[root@master crds]# oc delete -f example_v1alpha1_tomcat_cr.yaml
tomcat.example.com "example-tomcat" deleted

 

好把，這是一個簡單的Helm Operator上手的實驗，隨着Operator的成熟，在OpenShift 4.0版本中已經有很多組件化的部署都基於Operator來實現。