使用minukube部署kubernetes admission webhook實現etcd pod安全刪除

本需求來自於一道面試題😂（本環境使用centos 7）

最好使用阿里雲ec2服務器安裝minikube，若使用本地pc的vmware可能會出現網絡方面的問題。

使用如下命令安裝minikube，參見install minikube

# curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube
# sudo cp minikube /usr/local/bin && rm minikube

啟動minikube可能會遇到docker版本過舊導致啟動失敗的問題。默認centos下面yum安裝的docker版本比較舊，需要安裝最新docker，更新docker參見Get Docker CE for CentOS

安裝特定版本的kubectl

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x kubectl
mv kubectl /usr/local/bin

如下命令移除已經安裝的docker

yum remove docker \
                  docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine

如下命令安裝存儲驅動和設置docker stable版本的庫

yum install -y yum-utils \
  device-mapper-persistent-data \ lvm2 yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo

使用如下命令即可更新為最新的docker

yum install docker-ce docker-ce-cli containerd.io

 

管理kubernetes上的服務最好使用helm(本次未用到，如無需要可忽略本節)，helm安裝如下：

使用如下方式獲取helm的二進制版本

Download your desired version
Unpack it (tar -zxvf helm-v2.0.0-linux-amd64.tgz) Find the helm binary in the unpacked directory, and move it to its desired destination (mv linux-amd64/helm /usr/local/bin/helm)

 tiller安裝給出了在不同scope下面安裝tiller的方法，最簡單的是在cluster-admin下面安裝即可（生產環境下建議參考helm安裝文檔將權限最小化）

# cat rbac-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system

使用如下命令創建即可

kubectl create -f rbac-config.yaml
serviceaccount "tiller" created clusterrolebinding "tiller" created $ helm init --service-account tiller

admission webhook原理

kubernetes的認證和授權是對客戶端進行認證以及對資源進行授權，但在資源的使用處理上不夠細化。admission webhook是在一種在改變資源的持久化之前（比如某些資源的創建或刪除，修改等之前）的機制。參見官方資料

如下圖，在部署了admission webhook之后，apiserver會發送一個AdmissionReview的json數據到webhook，其中主要包含一個AdmissionRequest的請求，AdmissionRequest.RawExtension.Raw包含了需要處理的kubernetes組件(如pod，deployment等)的詳細信息。webhook在接收到該請求之后會根據自定義邏輯進行處理，並返回處理結果AdmissionResponse。admission webhook有兩種處理方式：

MutatingAdmissionWebhook：可以修改自定義的策略，MutatingAdmissionWebhook的處理一般優先於ValidatingAdmissionWebhook，這樣前者修改的內容就可以由后者進行校驗

ValidatingAdmissionWebhook: 允許或拒絕客戶自定義的策略

type AdmissionReview struct {
    metav1.TypeMeta `json:",inline"` // Request describes the attributes for the admission request. // +optional Request *AdmissionRequest `json:"request,omitempty" protobuf:"bytes,1,opt,name=request"` // Response describes the attributes for the admission response. // +optional Response *AdmissionResponse `json:"response,omitempty" protobuf:"bytes,2,opt,name=response"` }

 

 處理流程圖如下，可以看到MutatingAdmissionWebhook的處理優先於ValidatingAdmissionWebhook

 

使用如下命令啟動minikube

minikube start --vm-driver=none --extra-config=apiserver.enable-admission-plugins="NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota"

 使用kubectl api-versions查看是否支持admissionregistration.k8s.io/v1alpha1 API

In-depth introduction to Kubernetes admission webhooks是個很好的例子，里面詳細記錄了創建admission webhook的方式。這里參照In-depth introduction to Kubernetes admission webhooks實現了根據etcd pod角色(leader和非leader)來刪除pod。

首先使用webhook-create-signed-cert.sh文件來生成自簽證書，后續ValidatingWebhookConfiguration中會用到

# ./deployment/webhook-create-signed-cert.sh

部署admission webhook

# kubectl create -f deployment/deployment.yaml # kubectl create -f deployment/service.yaml

生成待CA bundle的config文件以及validatingwebhook.yaml原始文件如下

# cat ./deployment/validatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/validatingwebhook-ca-bundle.yaml

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validation-webhook-example-cfg labels: app: admission-webhook-example webhooks: - name: required-labels.banzaicloud.com clientConfig: service: name: admission-webhook-example-svc #service名稱 namespace: default path: "/validate" #訪問的后綴路徑 caBundle: ${CA_BUNDLE} #上述命令生成的認證字段 rules: - operations: [ "DELETE" ] #操作的動作 apiGroups: ["apps", ""] #api groups apiVersions: ["v1"] #api version resources: ["pods"] #操作的資源 namespaceSelector: matchLabels: admission-webhook-example: enabled # 限制default的命名空間

給default namespace打上標簽

$ kubectl label namespace default admission-webhook-example=enabled

創建ValidatingWebhookConfiguration

$ kubectl create -f deployment/validatingwebhook-ca-bundle.yaml

admission-webhook-example/build用於編譯和生成容器鏡像

etcd的部署直接使用admission-webhook-example\deployment\etcd中的配置文件即可，這樣在刪除etcd的leader時會顯示如下內容

admission webhook的實現只有2個文件，main.go和webhook.go。main.go中通過mux.HandleFunc("/validate", whsvr.serve)來啟動一個http服務處理路徑/validate(對應validatingwebhook.yaml的webhooks.clientConfig.service.path)的請求。主要邏輯是現在函數func (whsvr *WebhookServer) validate(ar *v1beta1.AdmissionReview) *v1beta1.AdmissionRespons中

 

FAQ：

• 執行yum install -y socat可解決如下問題：

an error occurred forwarding 40546 -> 44134: error forwarding port 44134 to pod 3ea221f842e1446a5fd9da9fc29e7e415a1ecb6dd6d07c56f64fb4788f2c3915, uid : unable to do port forwarding: socat not found.

 

TIPS:

•  使用如下方式可以安裝etcdctl命令行工具

# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/coreos/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL} rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1 rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz /tmp/etcd-download-test/etcd --version ETCDCTL_API=3 /tmp/etcd-download-test/etcdctl version # start a local etcd server /tmp/etcd-download-test/etcd # write,read to etcd ETCDCTL_API=3 /tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 put foo bar ETCDCTL_API=3 /tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 get foo

• go get -u github.com/golang/dep/cmd/dep 安裝dep
• go安裝方式如下：

安裝包地址：https://golang.google.cn/dl/ 
安裝go方式：tar -C /usr/local -xzf go1.11.2.linux-amd64.tar.gz 注意要安裝完整包，不能只是bin目錄
環境變量配置：
export GOPATH=/root/go
export GOROOT/usr/local/go
export PATH=$PATH:$GOROOT/bin/:$GOPATH/bin

• etcdv3的API使用可以參考package clientv3
• mutate webhook的使用參見k8s-mutate-webhook

 

參考：

https://github.com/helm/helm/blob/master/docs/install.md

https://container-solutions.com/some-admission-webhook-basics/

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/statefulset/etcd

https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/medium-article.md