1、常見的開發框架
- Charmed Operator Framework
- kubebuilder
- KubeOps (.NET operator SDK)
- KUDO (Kubernetes Universal Declarative Operator)
- Metacontroller along with WebHooks that you implement yourself
- Operator Framework
- shell-operator
2、kubebuilder安裝
kubebuilder book：https://book.kubebuilder.io/quick-start.html
kubebuilder GitHub：https://github.com/kubernetes-sigs/kubebuilder
kubebuilder安裝比較的簡單。由於對於windows支持不是很好，建議在Linux上安裝。
必要條件：
Prerequisites
查看go 、docker和kubernetes版本
[root@master ~]# go version
go version go1.17.7 linux/amd64
[root@master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
[root@master ~]# docker version
Client: Docker Engine - Community
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:45:22 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:44 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
[root@master ~]#
注：
（1）關於在centos8.2中安裝gov1.17，見：centos 8.2中安裝go 1.17
由於我們的go 版本為1.17，所以應該下載kubebuilder版本v3.3+
下載並安裝kuberbuilder
# download kubebuilder and install locally.
$ curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
$ chmod +x kubebuilder && mv kubebuilder /usr/local/bin/
3、kubebuilder初步實踐
初始化
[root@master ~]# mkdir -p $GOPATH/src/kubebuilder-demo
[root@master ~]# cd $GOPATH/src/kubebuilder-demo
[root@master kubebuilder-demo]# kubebuilder init --domain demo.kubebuilder.io
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.11.0
Update dependencies:
$ go mod tidy
go: downloading github.com/stretchr/testify v1.7.0
go: downloading github.com/onsi/gomega v1.17.0
go: downloading github.com/onsi/ginkgo v1.16.5
go: downloading github.com/go-logr/zapr v1.2.0
go: downloading go.uber.org/zap v1.19.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/Azure/go-autorest/autorest v0.11.18
go: downloading github.com/Azure/go-autorest v14.2.0+incompatible
go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.13
go: downloading go.uber.org/goleak v1.1.12
go: downloading go.uber.org/atomic v1.7.0
go: downloading go.uber.org/multierr v1.6.0
go: downloading github.com/benbjohnson/clock v1.1.0
go: downloading gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f
go: downloading github.com/Azure/go-autorest/logger v0.2.1
go: downloading github.com/Azure/go-autorest/tracing v0.6.0
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.1
go: downloading cloud.google.com/go v0.81.0
go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0
go: downloading github.com/form3tech-oss/jwt-go v3.2.3+incompatible
go: downloading golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
go: downloading github.com/nxadm/tail v1.4.8
go: downloading github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e
go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: downloading github.com/kr/text v0.2.0
Next: define a resource with:
注意：如果你不是在$GOPATH/src路徑下進行初始化，請添加上--repo參數作為項目MODULE的名稱
mkdir -p ~/projects/guestbook cd ~/projects/guestbook kubebuilder init --domain my.domain --repo my.domain/guestbookIf your project is initialized within
GOPATH, the implicitly calledgo mod initwill interpolate the module path for you. Otherwise--repo=must be set.Read the Go modules blogpost if unfamiliar with the module system.
創建API
[root@master kubebuilder-demo]# kubebuilder create api --group myapp --version v1 --kind Redis
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/redis_types.go
controllers/redis_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.8.0
go: downloading sigs.k8s.io/controller-tools v0.8.0
go: downloading github.com/spf13/cobra v1.2.1
go: downloading github.com/gobuffalo/flect v0.2.3
go: downloading golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff
go: downloading github.com/fatih/color v1.12.0
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/google/go-cmp v0.5.6
go: downloading golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e
go: downloading golang.org/x/mod v0.4.2
go get: installing executables with 'go get' in module mode is deprecated.
To adjust and download dependencies of the current module, use 'go get -d'.
To install using requirements of the current module, use 'go install'.
To install ignoring the current module, use 'go install' with a version,
like 'go install example.com/cmd@latest'.
For more information, see https://golang.org/doc/go-get-install-deprecation
or run 'go help get' or 'go help install'.
go get: added github.com/fatih/color v1.12.0
go get: added github.com/go-logr/logr v1.2.0
go get: added github.com/gobuffalo/flect v0.2.3
go get: added github.com/gogo/protobuf v1.3.2
go get: added github.com/google/go-cmp v0.5.6
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.12
go get: added github.com/mattn/go-colorable v0.1.8
go get: added github.com/mattn/go-isatty v0.0.12
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.2
go get: added github.com/spf13/cobra v1.2.1
go get: added github.com/spf13/pflag v1.0.5
go get: added golang.org/x/mod v0.4.2
go get: added golang.org/x/net v0.0.0-20210825183410-e898025ed96a
go get: added golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e
go get: added golang.org/x/text v0.3.7
go get: added golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff
go get: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.4.0
go get: added gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
go get: added k8s.io/api v0.23.0
go get: added k8s.io/apiextensions-apiserver v0.23.0
go get: added k8s.io/apimachinery v0.23.0
go get: added k8s.io/klog/v2 v2.30.0
go get: added k8s.io/utils v0.0.0-20210930125809-cb0fa318a74b
go get: added sigs.k8s.io/controller-tools v0.8.0
go get: added sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6
go get: added sigs.k8s.io/structured-merge-diff/v4 v4.1.2
go get: added sigs.k8s.io/yaml v1.3.0
/root/gopath/src/kubebuilder-demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
[root@master kubebuilder-demo]#
make install
[root@master kubebuilder-demo]# make install
/root/gopath/src/kubebuilder-demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7
go: downloading sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
go: downloading k8s.io/client-go v0.18.10
go: downloading github.com/spf13/cobra v1.0.0
go: downloading sigs.k8s.io/kustomize/api v0.6.5
go: downloading sigs.k8s.io/kustomize/cmd/config v0.8.5
go: downloading github.com/evanphx/json-patch v4.9.0+incompatible
go: downloading k8s.io/apimachinery v0.18.10
go: downloading sigs.k8s.io/kustomize/kyaml v0.9.4
go: downloading github.com/gogo/protobuf v1.3.1
go: downloading k8s.io/klog v1.0.0
go: downloading sigs.k8s.io/structured-merge-diff/v3 v3.0.0
go: downloading sigs.k8s.io/structured-merge-diff v0.0.0-20190525122527-15d366b2352e
go: downloading k8s.io/kube-openapi v0.0.0-20200410145947-61e04a5be9a6
go: downloading k8s.io/api v0.18.10
go: downloading github.com/olekukonko/tablewriter v0.0.4
go: downloading github.com/go-errors/errors v1.0.1
go: downloading gopkg.in/yaml.v2 v2.3.0
go: downloading github.com/yujunz/go-getter v1.4.1-lite
go: downloading github.com/json-iterator/go v1.1.8
go: downloading github.com/googleapis/gnostic v0.1.0
go: downloading github.com/Azure/go-autorest/autorest v0.9.0
go: downloading github.com/Azure/go-autorest/autorest/adal v0.5.0
go: downloading golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
go: downloading github.com/gophercloud/gophercloud v0.1.0
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go: downloading github.com/stretchr/testify v1.6.1
go: downloading github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
go: downloading github.com/go-openapi/strfmt v0.19.5
go: downloading github.com/go-openapi/validate v0.19.8
go: downloading github.com/mattn/go-runewidth v0.0.7
go: downloading github.com/hashicorp/go-multierror v1.1.0
go: downloading golang.org/x/net v0.0.0-20200625001655-4c5254603344
go: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
go: downloading github.com/hashicorp/go-cleanhttp v0.5.0
go: downloading github.com/hashicorp/go-safetemp v1.0.0
go: downloading github.com/hashicorp/go-version v1.1.0
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/mitchellh/go-testing-interface v1.0.0
go: downloading github.com/ulikunitz/xz v0.5.5
go: downloading github.com/Azure/go-autorest/logger v0.1.0
go: downloading github.com/Azure/go-autorest/tracing v0.5.0
go: downloading github.com/Azure/go-autorest/autorest/date v0.1.0
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: downloading cloud.google.com/go v0.38.0
go: downloading google.golang.org/appengine v1.5.0
go: downloading github.com/qri-io/starlib v0.4.2-0.20200213133954-ff2e8cd5ef8d
go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
go: downloading github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go: downloading github.com/go-openapi/errors v0.19.2
go: downloading github.com/mitchellh/mapstructure v1.1.2
go: downloading go.mongodb.org/mongo-driver v1.1.2
go: downloading github.com/golang/protobuf v1.3.2
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/hashicorp/errwrap v1.0.0
go: downloading github.com/go-openapi/analysis v0.19.5
go: downloading github.com/go-openapi/loads v0.19.4
go: downloading github.com/go-openapi/runtime v0.19.4
go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go: downloading golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go: downloading k8s.io/utils v0.0.0-20200324210504-a9aa75ae1b89
go: downloading golang.org/x/text v0.3.2
go: downloading github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633
go: downloading golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
go: downloading github.com/go-stack/stack v1.8.0
go get: installing executables with 'go get' in module mode is deprecated.
To adjust and download dependencies of the current module, use 'go get -d'.
To install using requirements of the current module, use 'go install'.
To install ignoring the current module, use 'go install' with a version,
like 'go install example.com/cmd@latest'.
For more information, see https://golang.org/doc/go-get-install-deprecation
or run 'go help get' or 'go help install'.
go get: added cloud.google.com/go v0.38.0
go get: added github.com/Azure/go-autorest/autorest v0.9.0
go get: added github.com/Azure/go-autorest/autorest/adal v0.5.0
go get: added github.com/Azure/go-autorest/autorest/date v0.1.0
go get: added github.com/Azure/go-autorest/logger v0.1.0
go get: added github.com/Azure/go-autorest/tracing v0.5.0
go get: added github.com/PuerkitoBio/purell v1.1.1
go get: added github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go get: added github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go get: added github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
go get: added github.com/davecgh/go-spew v1.1.1
go get: added github.com/dgrijalva/jwt-go v3.2.0+incompatible
go get: added github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633
go get: added github.com/evanphx/json-patch v4.9.0+incompatible
go get: added github.com/go-errors/errors v1.0.1
go get: added github.com/go-openapi/analysis v0.19.5
go get: added github.com/go-openapi/errors v0.19.2
go get: added github.com/go-openapi/jsonpointer v0.19.3
go get: added github.com/go-openapi/jsonreference v0.19.3
go get: added github.com/go-openapi/loads v0.19.4
go get: added github.com/go-openapi/runtime v0.19.4
go get: added github.com/go-openapi/spec v0.19.5
go get: added github.com/go-openapi/strfmt v0.19.5
go get: added github.com/go-openapi/swag v0.19.5
go get: added github.com/go-openapi/validate v0.19.8
go get: added github.com/go-stack/stack v1.8.0
go get: added github.com/gogo/protobuf v1.3.1
go get: added github.com/golang/protobuf v1.3.2
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go get: added github.com/googleapis/gnostic v0.1.0
go get: added github.com/gophercloud/gophercloud v0.1.0
go get: added github.com/hashicorp/errwrap v1.0.0
go get: added github.com/hashicorp/go-cleanhttp v0.5.0
go get: added github.com/hashicorp/go-multierror v1.1.0
go get: added github.com/hashicorp/go-safetemp v1.0.0
go get: added github.com/hashicorp/go-version v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.8
go get: added github.com/mailru/easyjson v0.7.0
go get: added github.com/mattn/go-runewidth v0.0.7
go get: added github.com/mitchellh/go-homedir v1.1.0
go get: added github.com/mitchellh/go-testing-interface v1.0.0
go get: added github.com/mitchellh/mapstructure v1.1.2
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.1
go get: added github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go get: added github.com/olekukonko/tablewriter v0.0.4
go get: added github.com/pkg/errors v0.9.1
go get: added github.com/pmezard/go-difflib v1.0.0
go get: added github.com/qri-io/starlib v0.4.2-0.20200213133954-ff2e8cd5ef8d
go get: added github.com/spf13/cobra v1.0.0
go get: added github.com/spf13/pflag v1.0.5
go get: added github.com/stretchr/testify v1.6.1
go get: added github.com/ulikunitz/xz v0.5.5
go get: added github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
go get: added github.com/yujunz/go-getter v1.4.1-lite
go get: added go.mongodb.org/mongo-driver v1.1.2
go get: added go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
go get: added golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go get: added golang.org/x/net v0.0.0-20200625001655-4c5254603344
go get: added golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
go get: added golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
go get: added golang.org/x/text v0.3.2
go get: added golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go get: added google.golang.org/appengine v1.5.0
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.3.0
go get: added gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go get: added k8s.io/api v0.18.10
go get: added k8s.io/apimachinery v0.18.10
go get: added k8s.io/client-go v0.18.10
go get: added k8s.io/klog v1.0.0
go get: added k8s.io/kube-openapi v0.0.0-20200410145947-61e04a5be9a6
go get: added k8s.io/utils v0.0.0-20200324210504-a9aa75ae1b89
go get: added sigs.k8s.io/kustomize/api v0.6.5
go get: added sigs.k8s.io/kustomize/cmd/config v0.8.5
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
go get: added sigs.k8s.io/kustomize/kyaml v0.9.4
go get: added sigs.k8s.io/structured-merge-diff/v3 v3.0.0
go get: added sigs.k8s.io/yaml v1.2.0
/root/gopath/src/kubebuilder-demo/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/redis.myapp.demo.kubebuilder.io created
[root@master kubebuilder-demo]
查看所創建的CRD
[root@master kubebuilder-demo]# kubectl get crds |grep redis
redis.myapp.demo.kubebuilder.io 2022-02-13T11:32:20Z
[root@master kubebuilder-demo]#
4、業務開發
本次實踐中，我們將創建自定義資源Redis，實現的功能
（1）創建CRD
（2）創建Kind ：Redis
（3）創建POD，支持副本數控制，驗證
（3）創建多副本POD，支持副本收縮
為了開發方便，可以將上面的生成的“kubebuilder-demo”文件拷貝到windows下，並導入到goland中。並配置SFTP自動上傳到$GOPATH/src/kubebuilder-demo下
另外也可以直接將文件拷貝到本地后，直接在本地運行
- （1）將Kubernetes的~/.kube/config文件，拷貝到windows的$ %HOME% /.kube/下
如：C:\Users\Administrator\.kube，之所以要這個做是因為kubebuilder中加載kubeconfig文件的順序所決定的。詳情見：Kubebuilder認證配置文件的加載 - （2）針對於_types.go文件的修改，需要重新編譯生成“zz_generated.deepcopy.go”文件，可以在Linux上make install后再拷貝回來
驗證
Kubebuilder中的驗證是通過標簽的方式來完成，標簽的基本形式為：//+some-tag 或//+some-other-tag=value
標簽分為兩種：
- 聲明在package行上的全局標簽
- 在類型聲明前的局部標簽（如在一個struct定義前）
（1）redis_types.go，修改Foo為Port
// +kubebuilder:validation:Maximum:=6380
// +kubebuilder:validation:Minimum:=6370
Port int `json:"port,omitempty"`
這里我們只是針對於port的范圍做驗證，有關Kubernetes的驗證見：https://book.kubebuilder.io/reference/markers/crd-validation.html
（2）修改“myapp_v1_redis.yaml”文件，增加端口字段：
apiVersion: myapp.demo.kubebuilder.io/v1
kind: Redis
metadata:
name: redis-sample
spec:
# TODO(user): Add fields here
port: 6379
由於這里我們修改了Redis struct的字段定義，所以需要重新make install
此時，我們可以看到自己的CRD中關於port字段的驗證規則：
[root@master ~]# kubectl get crds |grep redis
redis.myapp.demo.kubebuilder.io 2022-02-13T11:32:20Z
[root@master ~]# kubectl get crds/redis.myapp.demo.kubebuilder.io -oyaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.8.0
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.8.0"},"creationTimestamp":null,"name":"redis.myapp.demo.kubebuilder.io"},"spec":{"group":"myapp.demo.kubebuilder.io","names":{"kind":"Redis","listKind":"RedisList","plural":"redis","singular":"redis"},"scope":"Namespaced","versions":[{"name":"v1","schema":{"openAPIV3Schema":{"description":"Redis is the Schema for the redis API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"RedisSpec defines the desired state of Redis","properties":{"num":{"type":"integer"},"port":{"maximum":6380,"minimum":6370,"type":"integer"}},"type":"object"},"status":{"description":"RedisStatus defines the observed state of Redis","type":"object"}},"type":"object"}},"served":true,"storage":true,"subresources":{"status":{}}}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
creationTimestamp: "2022-02-13T11:32:20Z"
generation: 4
name: redis.myapp.demo.kubebuilder.io
resourceVersion: "510294"
uid: 91122c22-e52e-46cb-bcde-0a5b20578bc2
spec:
conversion:
strategy: None
group: myapp.demo.kubebuilder.io
names:
kind: Redis
listKind: RedisList
plural: redis
singular: redis
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: Redis is the Schema for the redis API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: RedisSpec defines the desired state of Redis
properties:
num:
type: integer
port:
maximum: 6380 #驗證規則
minimum: 6370
type: integer
type: object
status:
description: RedisStatus defines the observed state of Redis
type: object
type: object
served: true
storage: true
subresources:
status: {}
status:
acceptedNames:
kind: Redis
listKind: RedisList
plural: redis
singular: redis
conditions:
- lastTransitionTime: "2022-02-13T11:32:20Z"
message: no conflicts found
reason: NoConflicts
status: "True"
type: NamesAccepted
- lastTransitionTime: "2022-02-13T11:32:20Z"
message: the initial names have been accepted
reason: InitialNamesAccepted
status: "True"
type: Established
storedVersions:
- v1
[root@master ~]
（3）應用“myapp_v1_redis.yaml”文件，創建Kind Redis
[root@master kubebuilder-demo]# kubectl apply -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io/redis-sample configured
[root@master ~]# kubectl get Redis
NAME AGE
redis-sample 95m
上面就顯示了NAME和AGE，實際上這里的顯示格式，也是可以自定義的。
（4）修改“myapp_v1_redis.yaml”的port為6382，能夠看到報錯
[root@master kubebuilder-demo]# kubectl apply -f config/samples/myapp_v1_redis.yaml
The Redis "redis-sample" is invalid: spec.port: Invalid value: 6382: spec.port in body should be less than or equal to 6380
創建POD
創建“controllers/redis_helper.go”文件：
package controllers
import (
"context"
coreV1 "k8s.io/api/core/v1"
v1 "kubebuilder-demo/api/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
)
func CreateRedis(client client.Client, redisConfig *v1.Redis) error {
newPod := &coreV1.Pod{}
newPod.Name = redisConfig.Name
newPod.Namespace = redisConfig.Namespace
newPod.Spec.Containers = []coreV1.Container{
{
Name: redisConfig.Name,
Image: "redis:5-alpine",
ImagePullPolicy: coreV1.PullIfNotPresent,
Ports: []coreV1.ContainerPort{
{
ContainerPort: int32(redisConfig.Spec.Port),
},
},
},
}
return client.Create(context.Background(), newPod)
}
修改“controllers/redis_controller.go”文件的Reconcile，增加如下邏輯：
func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// TODO(user): your logic here
redis := &myappv1.Redis{}
err := r.Get(ctx, req.NamespacedName, redis)
if err != nil {
fmt.Println("error:", err)
} else {
fmt.Println("redis Obj:", redis)
err := CreateRedis(r.Client, redis)
fmt.Println("create pod failue,", err)
return ctrl.Result{}, err
}
return ctrl.Result{}, nil
}
此時make run，並打開另外一個窗口執行“kubectl apply -f config/samples/myapp_v1_redis.yaml”
注意：為了不影響結果，需要刪除前面一個步驟生成的Kind；kubectl delete -f
[root@master kubebuilder-demo]# kubectl get Redis
No resources found in default namespace.
[root@master kubebuilder-demo]# kubectl get Redis
No resources found in default namespace.
[root@master kubebuilder-demo]# kubectl apply -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io/redis-sample created
[root@master kubebuilder-demo]# kubectl get Redis
NAME AGE
redis-sample 21s
[root@master kubebuilder-demo]# kubectl get pods #已經創建了pod
NAME READY STATUS RESTARTS AGE
redis-sample 1/1 Running 0 16s
[root@master kubebuilder-demo]#
注意：這里存在一個問題，如果我們將創建資源的CRD文件刪掉，則POD依舊存在：
[root@master kubebuilder-demo]# kubectl delete -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io "redis-sample" deleted
[root@master kubebuilder-demo]# kubectl get pods #pod依舊存在
NAME READY STATUS RESTARTS AGE
redis-sample 1/1 Running 0 72s
[root@master kubebuilder-demo]# kubectl get Redis
No resources found in default namespace.
[root@master kubebuilder-demo]#
這就造成了所創建POD的孤立存在。
為此，我們可以使用finalizers來限制，如果POD沒有刪除，不能刪除對應POD。
有關finalizers的使用見：https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/
另外需要說明的是，POD究竟是如何被創建出來的，追蹤代碼能夠發現實際上調用的就是Kubernetes的API：
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#create-pod-v1-core
HTTP Request
POST /api/v1/namespaces/{namespace}/pods
Parameter Description namespaceobject name and auth scope, such as for teams and projects Query Parameters
Parameter Description prettyIf 'true', then the output is pretty printed. dryRunWhen present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed fieldManagerfieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. Body Parameters
Parameter Description bodyPodResponse
Code Description 200 Pod OK 201 Pod Created 202 Pod Accepte
跟蹤代碼“client.Create(context.Background(), newPod)”，
sigs.k8s.io/controller-runtime@v0.11.0/pkg/client/client.go
// Create implements client.Client.
func (c *client) Create(ctx context.Context, obj Object, opts ...CreateOption) error {
switch obj.(type) {
case *unstructured.Unstructured:
return c.unstructuredClient.Create(ctx, obj, opts...)
case *metav1.PartialObjectMetadata:
return fmt.Errorf("cannot create using only metadata")
default:
return c.typedClient.Create(ctx, obj, opts...)//
}
}
// Create implements client.Client.
func (c *typedClient) Create(ctx context.Context, obj Object, opts ...CreateOption) error {
o, err := c.cache.getObjMeta(obj)
if err != nil {
return err
}
createOpts := &CreateOptions{}
createOpts.ApplyOptions(opts)
return o.Post().
NamespaceIfScoped(o.GetNamespace(), o.isNamespaced()).
Resource(o.resource()).
Body(obj).
VersionedParams(createOpts.AsCreateOptions(), c.paramCodec).
Do(ctx).
Into(obj)
}
// request connects to the server and invokes the provided function when a server response is
// received. It handles retry behavior and up front validation of requests. It will invoke
// fn at most once. It will return an error if a problem occurred prior to connecting to the
// server - the provided function is responsible for handling server errors.
func (r *Request) request(ctx context.Context, fn func(*http.Request, *http.Response)) error {
//Metrics for total request latency
start := time.Now()
defer func() {
metrics.RequestLatency.Observe(ctx, r.verb, r.finalURLTemplate(), time.Since(start))
}()
if r.err != nil {
klog.V(4).Infof("Error in request: %v", r.err)
return r.err
}
if err := r.requestPreflightCheck(); err != nil {
return err
}
client := r.c.Client
if client == nil {
client = http.DefaultClient
}
// Throttle the first try before setting up the timeout configured on the
// client. We don't want a throttled client to return timeouts to callers
// before it makes a single request.
if err := r.tryThrottle(ctx); err != nil {
return err
}
if r.timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, r.timeout)
defer cancel()
}
// Right now we make about ten retry attempts if we get a Retry-After response.
var retryAfter *RetryAfter
for {
req, err := r.newHTTPRequest(ctx)
if err != nil {
return err
}
r.backoff.Sleep(r.backoff.CalculateBackoff(r.URL()))
if retryAfter != nil {
// We are retrying the request that we already send to apiserver
// at least once before.
// This request should also be throttled with the client-internal rate limiter.
if err := r.tryThrottleWithInfo(ctx, retryAfter.Reason); err != nil {
return err
}
retryAfter = nil
}
resp, err := client.Do(req) //在這里調用http.client發送請求報文
updateURLMetrics(ctx, r, resp, err)
....
}
}
最終在這里完成發送請求，將POD放入到請求荷載中：
net/http/client.go:725
創建多副本的POD
（1）修改“api/v1/redis_types.go”文件，增加副本Num字段：
// RedisSpec defines the desired state of Redis
type RedisSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
// Foo is an example field of Redis. Edit redis_types.go to remove/update
// +kubebuilder:validation:Maximum:=6380
// +kubebuilder:validation:Minimum:=6370
Port int `json:"port,omitempty"`
Num int `json:"num,omitempty"`
}
（2）修改“controllers/redis_helper.go”：
package controllers
import (
"context"
"fmt"
coreV1 "k8s.io/api/core/v1"
v1 "kubebuilder-demo/api/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
)
//判斷對應名稱的pod在finalizer中是否存在
func isExist(podName string, redis *v1.Redis) bool {
for _, finalizer := range redis.Finalizers {
if podName == finalizer {
return true
}
}
return false
}
//實際創建POD
func CreateRedis(podName string, client client.Client, redisConfig *v1.Redis) (string, error) {
if isExist(podName, redisConfig) {
return "", nil
}
newPod := &coreV1.Pod{}
newPod.Name = podName
newPod.Namespace = redisConfig.Namespace
newPod.Spec.Containers = []coreV1.Container{
{
Name: redisConfig.Name,
Image: "redis:5-alpine",
ImagePullPolicy: coreV1.PullIfNotPresent,
Ports: []coreV1.ContainerPort{
{
ContainerPort: int32(redisConfig.Spec.Port),
},
},
},
}
return podName, client.Create(context.Background(), newPod)
}
//為多副本POD命名
func GetRedisPodNames(redis *v1.Redis) []string {
podNames := make([]string, redis.Spec.Num)
for i := 0; i < redis.Spec.Num; i++ {
podNames[i] = fmt.Sprintf("%s-%d", redis.Name, i)
}
return podNames
}
（3）修改“controllers/redis_controller.go”：
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.0/pkg/reconcile
func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// TODO(user): your logic here
redis := &myappv1.Redis{}
err := r.Get(ctx, req.NamespacedName, redis)
if err != nil {
return ctrl.Result{}, err
}
fmt.Println("redis Obj:", redis)
//取得需要創建的POD副本的名稱
podNames := GetRedisPodNames(redis)
fmt.Println("podNames,", podNames)
updateFlag := false
//刪除POD，刪除Kind:Redis過程中，會自動加上DeletionTimestamp字段，據此判斷是否刪除了自定義資源
if !redis.DeletionTimestamp.IsZero() {
return ctrl.Result{}, r.clearRedis(ctx, redis)
}
//創建POD
for _, podName := range podNames {
finalizerPodName, err := CreateRedis(podName, r.Client, redis)
if err != nil {
fmt.Println("create pod failue,", err)
return ctrl.Result{}, err
}
if finalizerPodName == "" {
continue
}
//若該pod已經不在finalizers中，則添加
redis.Finalizers = append(redis.Finalizers, finalizerPodName)
updateFlag = true
}
//更新Kind Redis狀態
if updateFlag {
err = r.Client.Update(ctx, redis)
if err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}
//刪除POD邏輯
func (r *RedisReconciler) clearRedis(ctx context.Context, redis *myappv1.Redis) error {
//從finalizers中取出podName，然后執行刪除
for _, finalizer := range redis.Finalizers {
//刪除pod
err := r.Client.Delete(ctx, &v1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: finalizer,
Namespace: redis.Namespace,
},
})
if err != nil {
return err
}
}
//清空finializers，只要它有值，就無法刪除Kind
redis.Finalizers = []string{}
return r.Client.Update(ctx, redis)
}
（4）修改“config/samples/myapp_v1_redis.yaml”，增加“num”
apiVersion: myapp.demo.kubebuilder.io/v1
kind: Redis
metadata:
name: redis-sample
spec:
# TODO(user): Add fields here
port: 6379
num: 2
（5）make install后make run，在另外一個窗口中執行kubectl apply -f
注意：每次修改type類型定義文件，都需要進行make install
[root@master kubebuilder-demo]# kubectl apply -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io/redis-sample created
[root@master kubebuilder-demo]# kubectl get Redis
NAME AGE
redis-sample 6s
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 10s
redis-sample-1 1/1 Running 0 10s
[root@master kubebuilder-demo]#
此時查看Kind:Redis的yaml文件，能夠看到finalizers字段上填充了兩個POD的名字
[root@master kubebuilder-demo]# kubectl get Redis/redis-sample -oyaml
apiVersion: myapp.demo.kubebuilder.io/v1
kind: Redis
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"myapp.demo.kubebuilder.io/v1","kind":"Redis","metadata":{"annotations":{},"name":"redis-sample","namespace":"default"},"spec":{"num":2,"port":6379}}
creationTimestamp: "2022-02-14T15:13:09Z"
finalizers: #字段填充了POD名稱
- redis-sample-0
- redis-sample-1
generation: 1
name: redis-sample
namespace: default
resourceVersion: "561478"
uid: 46a36e7c-2af9-47b8-a882-1a9c11ab7110
spec:
num: 2
port: 6379
（6）此時嘗試刪除Kind：Redis，對應的POD也會被刪除
[root@master kubebuilder-demo]# kubectl delete -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io "redis-sample" deleted
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 0/1 Terminating 0 88s
redis-sample-1 0/1 Terminating 0 88s
[root@master kubebuilder-demo]# kubectl get Redis
No resources found in default namespace.
[root@master kubebuilder-demo]
（7）修改“config/samples/myapp_v1_redis.yaml”的num為1，並再次應用
查看POD，能夠發現POD數，並未發生變化：
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 4m27s
redis-sample-1 1/1 Running 0 4m27s
[root@master kubebuilder-demo]#
副本收縮
（1）修改“controllers/redis_controller.go”，優化副本刪除邏輯：
func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// TODO(user): your logic here
redis := &myappv1.Redis{}
err := r.Get(ctx, req.NamespacedName, redis)
if err != nil {
return ctrl.Result{}, err
}
fmt.Println("redis Obj:", redis)
//（1）刪除Kind Redis時，刪除所創建的POD
//（2）副本收縮時，縮減POD
if !redis.DeletionTimestamp.IsZero() || len(redis.Finalizers) > redis.Spec.Num {
return ctrl.Result{}, r.clearRedis(ctx, redis)
}
//創建POD
podNames := GetRedisPodNames(redis)
fmt.Println("podNames,", podNames)
updateFlag := false
for _, podName := range podNames {
finalizerPodName, err := CreateRedis(podName, r.Client, redis)
if err != nil {
fmt.Println("create pod failue,", err)
return ctrl.Result{}, err
}
if finalizerPodName == "" {
continue
}
redis.Finalizers = append(redis.Finalizers, finalizerPodName)
updateFlag = true
}
//更新Kind Redis狀態
if updateFlag {
err = r.Client.Update(ctx, redis)
if err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}
func (r *RedisReconciler) clearRedis(ctx context.Context, redis *myappv1.Redis) error {
//副本數和當前的num數量的差值，要刪除的就是這個差值部分，
//如果兩則相等，則刪除全部
//情形如下：
//（1）finalizers > num 可能出現，刪除差值部分
//（2）finalizers = num 可能出現，刪除全部
//（3）finalizers < num 不可能出現
var deletedPodNames []string
//后項刪除，即：刪除finalizers切片的最后的指定元素
position := redis.Spec.Num
if (len(redis.Finalizers) - redis.Spec.Num) != 0 {
deletedPodNames = redis.Finalizers[position:]
redis.Finalizers = redis.Finalizers[:position]
} else {
deletedPodNames = redis.Finalizers[:]
redis.Finalizers = []string{}
}
fmt.Println("deletedPodNames", deletedPodNames)
fmt.Println("redis.Finalizers", redis.Finalizers)
for _, finalizer := range deletedPodNames {
err := r.Client.Delete(ctx, &v1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: finalizer,
Namespace: redis.Namespace,
},
})
if err != nil {
return err
}
}
return r.Client.Update(ctx, redis)
}
（2）執行make run，修改“config/samples/myapp_v1_redis.yaml”的num值為5，創建5個副本的POD
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 4m27s
redis-sample-1 1/1 Running 0 4m27s
[root@master kubebuilder-demo]# vi config/samples/myapp_v1_redis.yaml
[root@master kubebuilder-demo]# kubectl apply -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io/redis-sample configured
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 12m
redis-sample-1 1/1 Running 0 12m
redis-sample-2 0/1 ContainerCreating 0 2s
redis-sample-3 0/1 ContainerCreating 0 2s
redis-sample-4 0/1 ContainerCreating 0 2s
[root@master kubebuilder-demo]#
等待POD創建就緒后，再次修改num為2，查看：
[root@master kubebuilder-demo]# kubectl apply -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io/redis-sample configured
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 14m
redis-sample-1 1/1 Running 0 14m
redis-sample-2 0/1 Terminating 0 2m5s
redis-sample-3 0/1 Terminating 0 2m5s
redis-sample-4 0/1 Terminating 0 2m5s
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 15m
redis-sample-1 1/1 Running 0 15m
[root@master kubebuilder-demo]#
刪除kind:Redis自定義資源：
[root@master kubebuilder-demo]# kubectl delete -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io "redis-sample" deleted
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 0/1 Terminating 0 16m
redis-sample-1 0/1 Terminating 0 16m
[root@master kubebuilder-demo]# kubectl get pods
No resources found in default namespace.
[root@master kubebuilder-demo]# kubectl get Redis
No resources found in default namespace.
[root@master kubebuilder-demo]#
POD刪除和重建
在前面的POD創建時，我們沒有使用Deployment來管理POD，一旦有些POD被刪除掉，則不會自動創建。這里我們借助於Watches來實現，同時引入Owner Referenc概念，從中取出Kind和Name，據此來判斷刪除的是我們POD，然后在等待隊列中，增加“reconcile.Request”，以此來實現對於副本數量的控制。
關於Owner Reference：
https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/
Owner References
Owner references describe how groups of objects are related. They are properties on resources that specify the relationship to one another, so entire trees of resources can be deleted.
Finalizer rules are processed when there are owner references. An owner reference consists of a name and a UID. Owner references link resources within the same namespace, and it also needs a UID for that reference to work. Pods typically have owner references to the owning replica set. So, when deployments or stateful sets are deleted, then the child replica sets and pods are deleted in the process.
（1）修改“controllers/redis_controller.go”，監聽POD：
//POD刪除時的回調
func (r *RedisReconciler) poddeleteHandler(event event.DeleteEvent, limitingInterface workqueue.RateLimitingInterface) {
fmt.Println("deleted pod name is :", event.Object.GetName())
//取得OwnerReference，如果Kind和Name匹配，則觸發reconcile.Request，並加入到等待隊列
for _, ownerReference := range event.Object.GetOwnerReferences() {
if ownerReference.Kind == "Redis" && ownerReference.Name == "redis-sample" {
limitingInterface.Add(reconcile.Request{
NamespacedName: types.NamespacedName{
Namespace: event.Object.GetNamespace(),
Name: ownerReference.Name,
},
})
}
}
}
// SetupWithManager sets up the controller with the Manager.
func (r *RedisReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&myappv1.Redis{}).
Watches(&source.Kind{
Type: &v1.Pod{},
}, handler.Funcs{
DeleteFunc: r.poddeleteHandler,
}).
Complete(r)
}
（2）修改“controllers/redis_helper.go”
//優化判斷POD存在邏輯
func isExistPod(podName string, redis *v1.Redis, client client.Client) bool {
err := client.Get(context.Background(), types.NamespacedName{
Name: podName,
Namespace: redis.Namespace,
}, &coreV1.Pod{})
if err != nil {
return false
}
return true
}
func CreateRedis(podName string, client client.Client, redisConfig *v1.Redis, scheme *runtime.Scheme) (string, error) {
if isExistPod(podName, redisConfig, client) {
return podName, nil
}
newPod := &coreV1.Pod{}
newPod.Name = podName
newPod.Namespace = redisConfig.Namespace
newPod.Spec.Containers = []coreV1.Container{
{
Name: redisConfig.Name,
Image: "redis:5-alpine",
ImagePullPolicy: coreV1.PullIfNotPresent,
Ports: []coreV1.ContainerPort{
{
ContainerPort: int32(redisConfig.Spec.Port),
},
},
},
}
err := controllerutil.SetControllerReference(redisConfig, newPod, scheme) //為POD增加OwnerReference
if err != nil {
return podName, err
}
return podName, client.Create(context.Background(), newPod)
}
（3）Make run
#刪除舊的自定義資源
[root@master kubebuilder-demo]# kubectl delete -f config/samples/myapp_v1_redis.yaml
[root@master kubebuilder-demo]# kubectl apply -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io/redis-sample created
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 13s
redis-sample-1 1/1 Running 0 13s
[root@master kubebuilder-demo]# kubectl delete pod redis-sample-1
pod "redis-sample-1" deleted
#在另外一個窗口查看pod情況
[root@master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 19s
redis-sample-1 0/1 Terminating 0 90s
[root@master ~]
#pod又被重新創建
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 37s
redis-sample-1 1/1 Running 0 8s
[root@master kubebuilder-demo]#
查看POD的yaml，觀察OwnerReference字段：
[root@master kubebuilder-demo]# kubectl get pod redis-sample-0 -oyaml
apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/containerID: 00e0b8f104de05271f3b0b9aa2f72e8bf6e919ce5139da10ad1c76e215fd4365
cni.projectcalico.org/podIP: 10.233.70.64/32
cni.projectcalico.org/podIPs: 10.233.70.64/32
creationTimestamp: "2022-02-15T09:11:21Z"
name: redis-sample-0
namespace: default
ownerReferences:
- apiVersion: myapp.demo.kubebuilder.io/v1
blockOwnerDeletion: true
controller: true
kind: Redis
name: redis-sample
uid: ed99405e-df1e-4d24-90f8-5994aa4c7f41
resourceVersion: "639304"
uid: 3173af40-498e-4d79-a006-cebaccdf86a9
spec:
...
其中“ownerReferences”中“ uid: ed99405e-df1e-4d24-90f8-5994aa4c7f41”為自定義資源Redis的UID：
[root@master kubebuilder-demo]# kubectl get Redis -oyaml
apiVersion: v1
items:
- apiVersion: myapp.demo.kubebuilder.io/v1
kind: Redis
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"myapp.demo.kubebuilder.io/v1","kind":"Redis","metadata":{"annotations":{},"name":"redis-sample","namespace":"default"},"spec":{"num":2,"port":6379}}
creationTimestamp: "2022-02-15T09:09:38Z"
finalizers:
- redis-sample-0
- redis-sample-1
generation: 1
name: redis-sample
namespace: default
resourceVersion: "639070"
uid: ed99405e-df1e-4d24-90f8-5994aa4c7f41
spec:
num: 2
port: 6379
kind: List
metadata:
resourceVersion: ""
selfLink: ""
添加事件支持
目前在自定義資源Kind:Redis中Events:
[root@master kubebuilder-demo]# kubectl describe Redis/redis-sample
Name: redis-sample
Namespace: default
Labels: <none>
Annotations: <none>
API Version: myapp.demo.kubebuilder.io/v1
Kind: Redis
Metadata:
Creation Timestamp: 2022-02-15T09:09:38Z
Finalizers:
redis-sample-0
redis-sample-1
Generation: 1
Managed Fields:
API Version: myapp.demo.kubebuilder.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:num:
f:port:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-02-15T09:09:38Z
API Version: myapp.demo.kubebuilder.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"redis-sample-0":
v:"redis-sample-1":
Manager: main
Operation: Update
Time: 2022-02-15T09:09:38Z
Resource Version: 639070
UID: ed99405e-df1e-4d24-90f8-5994aa4c7f41
Spec:
Num: 2
Port: 6379
Events: <none>
[root@master kubebuilder-demo]#
現在需要增加對於增加和刪除POD的事件進行記錄。
（1）修改redis_controller.go
// RedisReconciler reconciles a Redis object
type RedisReconciler struct {
client.Client
Scheme *runtime.Scheme
EventRecorder record.EventRecorder //增加事件記錄
}
func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
...
//更新Kind Redis狀態
if updateFlag {
r.EventRecorder.Event(redis,v1.EventTypeNormal,"Upgrade","scale replicates")//創建POD時，記錄事件
err = r.Client.Update(ctx, redis)
if err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}
func (r *RedisReconciler) clearRedis(ctx context.Context, redis *myappv1.Redis) error {
...
r.EventRecorder.Event(redis,v1.EventTypeNormal,"Updated","scale replicates")//刪除POD時，記錄事件
return r.Client.Update(ctx, redis)
}
（2）修改main.go，創建RedisReconciler時，為EventRecorder屬性賦值
func main() {
...
if err = (&controllers.RedisReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
EventRecorder: mgr.GetEventRecorderFor("demo.kubebuilder.io"), //名稱任意
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "Redis")
os.Exit(1)
}
...
}
（3）make install & make run ，然后測試效果
[root@master kubebuilder-demo]# kubectl apply -f config/samples/myapp_v1_redis.yaml
redis.myapp.demo.kubebuilder.io/redis-sample created
[root@master kubebuilder-demo]# kubectl get Redis
NAME AGE
redis-sample 29s
[root@master kubebuilder-demo]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-sample-0 1/1 Running 0 80s
redis-sample-1 1/1 Running 0 80s
[root@master kubebuilder-demo]#
[root@master kubebuilder-demo]# kubectl describe Redis/redis-sample
Name: redis-sample
Namespace: default
Labels: <none>
Annotations: <none>
API Version: myapp.demo.kubebuilder.io/v1
Kind: Redis
Metadata:
Creation Timestamp: 2022-02-15T10:14:38Z
Finalizers:
redis-sample-0
redis-sample-1
Generation: 1
Managed Fields:
API Version: myapp.demo.kubebuilder.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:num:
f:port:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-02-15T10:14:38Z
API Version: myapp.demo.kubebuilder.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"redis-sample-0":
v:"redis-sample-1":
Manager: main
Operation: Update
Time: 2022-02-15T10:14:38Z
Resource Version: 643906
UID: aae17d51-0889-40d0-838d-d6f18d411956
Spec:
Num: 2
Port: 6379
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Upgrade 40s demo.kubebuilder.io scale replicates
修改副本數為1：
[root@master kubebuilder-demo]# kubectl describe Redis/redis-sample
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Upgrade 4m59s (x2 over 7m39s) demo.kubebuilder.io scale replicates
Normal Updated 18s (x2 over 65s) demo.kubebuilder.io scale replicates
額外信息的輸出
默認情況下，我們查看自定義資源Redis時，輸出的是這樣的：
[root@master kubebuilder-demo]# kubectl get Redis
NAME AGE
redis-sample 21s
如果我們想要輸出中包含副本的數量，即：
[root@master kubebuilder-demo]# kubectl get Redis
NAME REPLICAS
redis-sample 5
關於Addition Printer Columns
Additional Printer Columns
Starting with Kubernetes 1.11,
kubectl getcan ask the server what columns to display. For CRDs, this can be used to provide useful, type-specific information withkubectl get, similar to the information provided for built-in types.The information that gets displayed can be controlled with the additionalPrinterColumns field on your CRD, which is controlled by the
+kubebuilder:printcolumnmarker on the Go type for your CRD.For instance, in the following example, we add fields to display information about the knights, rank, and alias fields from the validation example:
// +kubebuilder:printcolumn:name="Alias",type=string,JSONPath=`.spec.alias` // +kubebuilder:printcolumn:name="Rank",type=integer,JSONPath=`.spec.rank` // +kubebuilder:printcolumn:name="Bravely Run Away",type=boolean,JSONPath=`.spec.knights[?(@ == "Sir Robin")]`,description="when danger rears its ugly head, he bravely turned his tail and fled",priority=10 // +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp" type Toy struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec ToySpec `json:"spec,omitempty"` Status ToyStatus `json:"status,omitempty"` }
這里需要使用到“+kubebuilder:printcolumn”，關於它的字段描述：
https://book.kubebuilder.io/reference/markers/crd.html
（1）修改“api/v1/redis_types.go”
type RedisStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Important: Run "make" to regenerate code after modifying this file
Replicas int `json:"replicas,omitempty"` //添加副本顯示
}
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:printcolumn:JSONPath=".status.replicas",name=Replicas,type=integer
// Redis is the Schema for the redis API
type Redis struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec RedisSpec `json:"spec,omitempty"`
Status RedisStatus `json:"status,omitempty"`
}
（2）修改“controllers/redis_controller.go”
func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
...
//更新Kind Redis狀態
if updateFlag {
...
//更新status狀態值
redis.Status.Replicas = len(redis.Finalizers)
err := r.Status().Update(ctx, redis)
if err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}
func (r *RedisReconciler) clearRedis(ctx context.Context, redis *myappv1.Redis) error {
...
//更新status狀態值
redis.Status.Replicas = len(redis.Finalizers)
err = r.Status().Update(ctx, redis)
if err != nil {
return err
}
return nil
}
（3）驗證
make install 后make run
[root@master kubebuilder-demo]# kubectl get Redis
NAME REPLICAS
redis-sample 5
[root@master kubebuilder-demo]# kubectl describe Redis/redis-sample
Name: redis-sample
Namespace: default
Labels: <none>
Annotations: <none>
API Version: myapp.demo.kubebuilder.io/v1
Kind: Redis
...
Spec:
Num: 5
Port: 6379
Status:
Replicas: 5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Upgrade 28s demo.kubebuilder.io scale replicates