目錄
operator-sdk 環境搭建與簡單實例
環境說明
系統：CentOS Linux release 7.5.1804 (Core)
golang：v1.15
operator-sdk：v1.7.0
docker：v1.19 # 因為 operator-sdk 使用了多階段構建功能，所以 docker 必須大於等於 v1.17 版本
k8s：k3s v1.20.6+k3s1 單節點
安裝 docker
略
安裝 golang
安裝
$ yum install epel-release -y
$ yum install golang -y
配置
$ mkdir -p /home/workspace/go/{src,bin,pkg}
$ vi /etc/profile
export GOPATH=/home/workspace/go
export GO111MODULE=on
export GOPROXY=https://goproxy.io,direct
export PATH=$PATH:$GOPATH/bin
$ . /etc/profile
驗證
$ go version
go version go1.15.5 linux/amd64
安裝 operator-sdk
operator-sdk 為 go 語言編寫，直接下載二進制執行文件安裝即可，下載地址：
https://github.com/operator-framework/operator-sdk/releases
安裝步驟略。
創建一個簡單實例
步驟1: init操作
$ mkdir -p $GOPATH/src/github.com/leffss/memcached-operator
$ cd $GOPATH/src/github.com/leffss/memcached-operator
$ operator-sdk init --domain=example.com --repo=github.com/leffss/memcached-operator
步驟2: 創建API
$ operator-sdk create api --group cache --version v1 --kind Memcached --resource=true --controller=true
步驟3：安裝 CRDs
$ make generate && make manifests && make install
步驟4: 構建鏡像
$ vi Dockerfile
# 在 RUN go mod download 前面新增 goproxy
ENV GOPROXY https://goproxy.io,direct
# 注釋 FROM gcr.io/distroless/static:nonroot（國外源，pull 不下來），新增
FROM kubeimages/distroless-static:latest
$ make docker-build IMG=leffss/memcache:v1
$ docker images |grep memcache
leffss/memcache v1 1eaa4b6a4781 About a minute ago 46.5MB
- 如遇到包下載錯誤（網絡問題），多嘗試幾次
步驟5: 運行Operator
准備鏡像：
# 因為 k3s 安裝的 k8s v1.20 默認使用的是 containerd，所以要導入鏡像
$ docker save leffss/memcache:v1 > memcache.tar
$ ctr image import memcache.tar
$ docker pull kubesphere/kube-rbac-proxy:v0.8.0
$ docker tag kubesphere/kube-rbac-proxy:v0.8.0 gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
$ docker save gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0 > kube-rabc-proxy.tar
$ ctr image import kube-rabc-proxy.tar
運行
$ make deploy IMG=leffss/memcache:v1
結果確認：
$ kubectl get service -A |grep memcache
memcached-operator-system memcached-operator-controller-manager-metrics-service ClusterIP 10.43.212.38 <none> 8443/TCP 83s
$ kubectl -n memcached-operator-system get pod
NAME READY STATUS RESTARTS AGE
memcached-operator-controller-manager-5cd7d4cdc7-tpsh2 2/2 Running 0 9m10s
$ kubectl -n memcached-operator-system get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
memcached-operator-controller-manager 1/1 1 1 26m
CRDs 確認
$ kubectl get crd
NAME CREATED AT
addons.k3s.cattle.io 2021-05-02T01:03:34Z
helmcharts.helm.cattle.io 2021-05-02T01:03:34Z
helmchartconfigs.helm.cattle.io 2021-05-02T01:03:34Z
memcacheds.cache.example.com 2021-05-02T06:55:23Z
詳細確認
$ kubectl describe crd memcacheds.cache.example.com
Name: memcacheds.cache.example.com
Namespace:
Labels: <none>
Annotations: controller-gen.kubebuilder.io/version: v0.4.1
API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2021-05-02T06:55:23Z
Generation: 1
Managed Fields:
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:acceptedNames:
f:listKind:
f:singular:
Manager: k3s
Operation: Update
Time: 2021-05-02T06:55:23Z
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:controller-gen.kubebuilder.io/version:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
f:conversion:
.:
f:strategy:
f:group:
f:names:
f:kind:
f:listKind:
f:plural:
f:singular:
f:scope:
f:versions:
f:status:
f:acceptedNames:
f:kind:
f:plural:
f:conditions:
f:storedVersions:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-05-02T06:55:26Z
Resource Version: 16326
UID: 361420cd-5785-4751-ab40-bb0e4482d0cf
Spec:
Conversion:
Strategy: None
Group: cache.example.com
Names:
Kind: Memcached
List Kind: MemcachedList
Plural: memcacheds
Singular: memcached
Scope: Namespaced
Versions:
Name: v1
Schema:
openAPIV3Schema:
Description: Memcached is the Schema for the memcacheds API
Properties:
API Version:
Description: APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
Type: string
Kind:
Description: Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
Type: string
Metadata:
Type: object
Spec:
Description: MemcachedSpec defines the desired state of Memcached
Properties:
Foo:
Description: Foo is an example field of Memcached. Edit memcached_types.go to remove/update
Type: string
Type: object
Status:
Description: MemcachedStatus defines the observed state of Memcached
Type: object
Type: object
Served: true
Storage: true
Subresources:
Status:
Status:
Accepted Names:
Kind: Memcached
List Kind: MemcachedList
Plural: memcacheds
Singular: memcached
Conditions:
Last Transition Time: 2021-05-02T06:55:23Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: 2021-05-02T06:55:23Z
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Stored Versions:
v1
Events: <none>
步驟6: 創建自定義資源
$ kubectl apply -f config/samples/cache_v1_memcached.yaml
memcached.cache.example.com/memcached-sample created
確認Operator相關日志信息
$ kubectl -n memcached-operator-system logs memcached-operator-controller-manager-6cf86db855-sqhpj -c manager
2021-05-02T07:11:32.316Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2021-05-02T07:11:32.317Z INFO setup starting manager
I0502 07:11:32.317614 1 leaderelection.go:243] attempting to acquire leader lease memcached-operator-system/2ef0186a.example.com...
2021-05-02T07:11:32.317Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
I0502 07:11:32.414648 1 leaderelection.go:253] successfully acquired lease memcached-operator-system/2ef0186a.example.com
2021-05-02T07:11:32.415Z INFO controller-runtime.manager.controller.memcached Starting EventSource {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "source": "kind source: /, Kind="}
2021-05-02T07:11:32.415Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"memcached-operator-system","name":"2ef0186a.example.com","uid":"b8b753e8-6400-4adc-96bd-365ce8d9dabc","apiVersion":"v1","resourceVersion":"17936"}, "reason": "LeaderElection", "message": "memcached-operator-controller-manager-6cf86db855-sqhpj_ac983311-240b-47da-a7cd-2b8753c8fe21 became leader"}
2021-05-02T07:11:32.415Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"Lease","namespace":"memcached-operator-system","name":"2ef0186a.example.com","uid":"1ee59e1e-4481-4b84-a3bd-0f0b1e97979d","apiVersion":"coordination.k8s.io/v1","resourceVersion":"17937"}, "reason": "LeaderElection", "message": "memcached-operator-controller-manager-6cf86db855-sqhpj_ac983311-240b-47da-a7cd-2b8753c8fe21 became leader"}
2021-05-02T07:11:32.515Z INFO controller-runtime.manager.controller.memcached Starting Controller {"reconciler group": "cache.example.com", "reconciler kind": "Memcached"}
2021-05-02T07:11:32.515Z INFO controller-runtime.manager.controller.memcached Starting workers {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "worker count": 1}
創建 crd 資源后可能 controller 會包 RBAC 權限錯誤，解決方法是修改部署時的權限配置，最簡單的方法是直接給 controller 綁定到 cluster-admin 集群管理員
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: app-controller-manager
namespace: app-system
步驟7: 刪除CR和相關資源
$ kubectl delete -f config/samples/cache_v1_memcached.yaml
memcached.cache.example.com "memcached-sample" deleted
步驟8：刪除 CRDs
$ make uninstall
步驟9：刪除 operator
$ make undeploy
總結
按照前面步驟的一系列的操作，我們使用了 sdk 創建的默認crd的屬性和controller代碼，實際開發中需要根據需求自行定義 crd 屬性，並編寫對應的 controller 邏輯代碼，最終實現一個完整的 operator。
補充：kubebuilder和operator-sdk對比
kubebuilder和operator-sdk都是開發crd的工具。其中kubebuilder是k8s官方提供工具，operator-sdk早期版本使用了其自己的邏輯實現，但是目前這兩個工具都是在controller-runtime基礎上做了一層封裝,底層邏輯代碼基本沒什么多大的區別，可以看個人習慣自行選擇。