1、什么是K8s的彈性伸縮？
答：Hpa（全稱叫做Horizontal Pod Autoscaler），Horizontal Pod Autoscaler的操作對象是Replication Controller、ReplicaSet或者Deployment對應的Pod（k8s中可以控制Pod的是rc、rs、deployment），根據觀察到的CPU使用量與用戶的閾值進行比對，做出是否需要增加或者減少實例數量的決策。controller目前使用heapSter來檢測CPU使用量，檢測周期默認是30秒。
2、K8s的彈性伸縮的工作原理？
答：Horizontal Pod Autoscaler的工作原理，主要是監控一個Pod，監控這個Pod的資源CPU使用率，一旦達到了設置的閾值，就做策略來決定它是否需要增加，做策略的時候還需要一個周期，比如，持續五分鍾都發現CPU使用率高，就抓緊增加Pod的數量來減輕它的壓力。當然也有一個策略，就是持續五分鍾之后，壓力一直都很低，那么會減少Pod的數量。這就是k8s的彈性伸縮的工作原理，主要是監控CPU的使用率，然后來決定是否增加或者減少Pod的數量。
3、K8s的彈性伸縮的實踐，為了演示效果，這里對rc進行cpu資源的進行限制，方便壓力測試效果。
1 [root@k8s-master ~]# cd k8s/ 2 [root@k8s-master k8s]# ls 3 book-master.war dashboard dashboard.zip deploy health heapster namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip 4 [root@k8s-master k8s]# mkdir hpa 5 [root@k8s-master k8s]# cd hpa/ 6 [root@k8s-master hpa]# ls 7 [root@k8s-master hpa]# cp ../rc/nginx_rc.yaml . 8 [root@k8s-master hpa]# ls 9 nginx_rc.yaml 10 [root@k8s-master hpa]# vim nginx_rc.yaml
配置內容，如下所示：
1 apiVersion: v1 2 kind: Pod 3 metadata: 4 name: myweb 5 labels: 6 app: web 7 env: myweb 8 spec: 9 containers: 10 - name: myweb 11 image: 192.168.110.133:5000/nginx:1.13 12 ports: 13 - containerPort: 80 14 resources: 15 # 最大可以使用的資源，100m的cpu時間片，50Mi的內存。 16 limits: 17 cpu: 100m 18 memory: 50Mi 19 # requests代表資源Pod需求的資源，100m的cpu時間片，50Mi的內存。 20 requests: 21 cpu: 100m 22 memory: 50Mi
開始創建這個RC，如下所示：
1 [root@k8s-master hpa]# kubectl create -f nginx_rc.yaml 2 replicationcontroller "myweb" created 3 [root@k8s-master hpa]#
查看初始的數量是兩個，如下所示：
1 [root@k8s-master hpa]# kubectl get all 2 NAME DESIRED CURRENT READY AGE 3 rc/myweb 2 2 2 31s 4 5 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE 6 svc/kubernetes 10.254.0.1 <none> 443/TCP 23d 7 8 NAME READY STATUS RESTARTS AGE 9 po/myweb-c0rs7 1/1 Running 0 31s 10 po/myweb-jkqc7 1/1 Running 0 31s 11 [root@k8s-master hpa]# kubectl get all -o wide 12 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 13 rc/myweb 2 2 2 35s myweb 192.168.110.133:5000/nginx:1.13 app=myweb 14 15 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 16 svc/kubernetes 10.254.0.1 <none> 443/TCP 23d <none> 17 18 NAME READY STATUS RESTARTS AGE IP NODE 19 po/myweb-c0rs7 1/1 Running 0 35s 172.16.16.4 k8s-node3 20 po/myweb-jkqc7 1/1 Running 0 35s 172.16.94.3 k8s-node2 21 [root@k8s-master hpa]#
4、開始創建hpa，可以通過配置文件或者命令進行聲明，如下所示：
1 kubectl autoscale 資源類型（rc、deployment） 資源的名稱 最大的pod數量 最小的pod數量 pod設定的閾值，cpu達到%多少使用率的時候就會觸發hpa。
pod設定的閾值，cpu達到%多少使用率的時候就會觸發hpa，這里測試的時候設置的比較低，生產環境可以設置60%、70%這些，較高些。
1 [root@k8s-master hpa]# kubectl autoscale replicationcontroller myweb --max=8 --min=1 --cpu-percent=5 2 replicationcontroller "myweb" autoscaled 3 [root@k8s-master hpa]#
此時，查看啟動的資源，如下所示：
1 [root@k8s-master hpa]# kubectl get all 2 NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE 3 hpa/myweb ReplicationController/myweb 5% <waiting> 1 8 53s 4 5 NAME DESIRED CURRENT READY AGE 6 rc/myweb 2 2 2 7m 7 8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE 9 svc/kubernetes 10.254.0.1 <none> 443/TCP 23d 10 11 NAME READY STATUS RESTARTS AGE 12 po/myweb-c0rs7 1/1 Running 0 7m 13 po/myweb-jkqc7 1/1 Running 0 7m 14 [root@k8s-master hpa]# kubectl get all -o wide 15 NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE 16 hpa/myweb ReplicationController/myweb 5% <waiting> 1 8 56s 17 18 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 19 rc/myweb 2 2 2 7m myweb 192.168.110.133:5000/nginx:1.13 app=myweb 20 21 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 22 svc/kubernetes 10.254.0.1 <none> 443/TCP 23d <none> 23 24 NAME READY STATUS RESTARTS AGE IP NODE 25 po/myweb-c0rs7 1/1 Running 0 7m 172.16.16.4 k8s-node3 26 po/myweb-jkqc7 1/1 Running 0 7m 172.16.94.3 k8s-node2 27 [root@k8s-master hpa]#
這里可以看到由hpa控制rc，rc來控制pod的數量，現在開始進行壓力測試，這里使用的ab的命令，首先查詢到這個pod的ip地址，如下所示：
1 [root@k8s-master hpa]# kubectl get all -o wide 2 NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE 3 hpa/myweb ReplicationController/myweb 5% <waiting> 1 8 3m 4 5 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 6 rc/myweb 2 2 2 10m myweb 192.168.110.133:5000/nginx:1.13 app=myweb 7 8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 9 svc/kubernetes 10.254.0.1 <none> 443/TCP 23d <none> 10 11 NAME READY STATUS RESTARTS AGE IP NODE 12 po/myweb-c0rs7 1/1 Running 0 10m 172.16.16.4 k8s-node3 13 po/myweb-jkqc7 1/1 Running 0 10m 172.16.94.3 k8s-node2 14 [root@k8s-master hpa]#
首先安裝一下httpd，安裝了這個就包含了ab的命令了，如下所示：
1 [root@k8s-master ~]# yum install httpd-tools.x86_64 -y 2 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager 3 4 This system is not registered with an entitlement server. You can use subscription-manager to register. 5 6 Determining fastest mirrors 7 * base: mirrors.tuna.tsinghua.edu.cn 8 * extras: mirrors.bfsu.edu.cn 9 * updates: mirrors.bfsu.edu.cn 10 base | 3.6 kB 00:00:00 11 extras | 2.9 kB 00:00:00 12 updates | 2.9 kB 00:00:00 13 updates/7/x86_64/primary_db | 2.9 MB 00:00:03 14 Resolving Dependencies 15 --> Running transaction check 16 ---> Package httpd-tools.x86_64 0:2.4.6-93.el7.centos will be installed 17 --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-tools-2.4.6-93.el7.centos.x86_64 18 --> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-tools-2.4.6-93.el7.centos.x86_64 19 --> Running transaction check 20 ---> Package apr.x86_64 0:1.4.8-5.el7 will be installed 21 ---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed 22 --> Finished Dependency Resolution 23 24 Dependencies Resolved 25 26 ================================================================================================================================================================================================================= 27 Package Arch Version Repository Size 28 ================================================================================================================================================================================================================= 29 Installing: 30 httpd-tools x86_64 2.4.6-93.el7.centos base 92 k 31 Installing for dependencies: 32 apr x86_64 1.4.8-5.el7 base 103 k 33 apr-util x86_64 1.5.2-6.el7 base 92 k 34 35 Transaction Summary 36 ================================================================================================================================================================================================================= 37 Install 1 Package (+2 Dependent packages) 38 39 Total download size: 288 k 40 Installed size: 584 k 41 Downloading packages: 42 (1/3): httpd-tools-2.4.6-93.el7.centos.x86_64.rpm | 92 kB 00:00:00 43 (2/3): apr-util-1.5.2-6.el7.x86_64.rpm | 92 kB 00:00:00 44 (3/3): apr-1.4.8-5.el7.x86_64.rpm | 103 kB 00:00:00 45 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 46 Total 273 kB/s | 288 kB 00:00:01 47 Running transaction check 48 Running transaction test 49 Transaction test succeeded 50 Running transaction 51 Installing : apr-1.4.8-5.el7.x86_64 1/3 52 Installing : apr-util-1.5.2-6.el7.x86_64 2/3 53 Installing : httpd-tools-2.4.6-93.el7.centos.x86_64 3/3 54 Verifying : apr-1.4.8-5.el7.x86_64 1/3 55 Verifying : httpd-tools-2.4.6-93.el7.centos.x86_64 2/3 56 Verifying : apr-util-1.5.2-6.el7.x86_64 3/3 57 58 Installed: 59 httpd-tools.x86_64 0:2.4.6-93.el7.centos 60 61 Dependency Installed: 62 apr.x86_64 0:1.4.8-5.el7 apr-util.x86_64 0:1.5.2-6.el7 63 64 Complete! 65 [root@k8s-master ~]#
壓力測試，總共發起500000次請求，每次發起30個請求，如下所示：
1 [root@k8s-master ~]# ab -n 500000 -c 30 http://172.16.94.3/index.html/
Kubernetes dashboard的界面，現在，如下所示：
由於我的是啟動了兩個Pod，這里我也同時壓力測試兩個Pod，第二個Pod的壓力測試如下所示：
1 [root@k8s-master hpa]# ab -n 500000 -c 30 http://172.16.16.4/index.html/ 2 This is ApacheBench, Version 2.3 <$Revision: 1430300 $> 3 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ 4 Licensed to The Apache Software Foundation, http://www.apache.org/ 5 6 Benchmarking 172.16.16.4 (be patient)
此時的CPU使用情況，如下所示：
可以使用命令查看hpa調度的情況，如下所示：
注意：我后來觀察了一下下面的信息，發現沒有進行Pod的伸縮的報錯原因，原來是下面的錯誤，如下所示：
1 FailedGetMetrics unable to get metrics for resource cpu: failed to get heapster service: the server could not find the requested resource (get services http:heapster:)
具體查看詳情信息，如下所示：
1 [root@k8s-master rc]# kubectl describe hpa myweb 2 Name: myweb 3 Namespace: default 4 Labels: <none> 5 Annotations: <none> 6 CreationTimestamp: Sun, 28 Jun 2020 19:47:19 +0800 7 Reference: ReplicationController/myweb 8 Target CPU utilization: 5% 9 Current CPU utilization: <unset> 10 Min replicas: 1 11 Max replicas: 8 12 ReplicationController pods: 2 current / 2 desired 13 Events: 14 FirstSeen LastSeen Count From SubObjectPath Type Reason Message 15 --------- -------- ----- ---- ------------- -------- ------ ------- 16 16m 14m 8 {horizontal-pod-autoscaler } Normal MetricsNotAvailableYet unable to get metrics for resource cpu: failed to get heapster service: the server could not find the requested resource (get services http:heapster:) 17 13m 17s 28 {horizontal-pod-autoscaler } Warning FailedGetMetrics unable to get metrics for resource cpu: failed to get heapster service: the server could not find the requested resource (get services http:heapster:) 18 [root@k8s-master rc]#
我對兩個Pod同時壓力測試都沒有伸縮，沒有給我擴容Pod，更別提縮減Pod。
我這里一直測試不出，這里將值調整的更低些，看看效果，如何，先將rc，hpa刪除掉哈，然后再創建即可，如下所示：
1 [root@k8s-master ~]# kubectl get all -o wide 2 NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE 3 hpa/myweb ReplicationController/myweb 5% <waiting> 1 8 38m 4 5 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 6 rc/myweb 2 2 2 45m myweb 192.168.110.133:5000/nginx:1.13 app=myweb 7 8 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 9 svc/kubernetes 10.254.0.1 <none> 443/TCP 23d <none> 10 11 NAME READY STATUS RESTARTS AGE IP NODE 12 po/myweb-c0rs7 1/1 Running 0 45m 172.16.16.4 k8s-node3 13 po/myweb-jkqc7 1/1 Running 0 45m 172.16.94.3 k8s-node2 14 [root@k8s-master ~]# kubectl get rc -o wide 15 NAME DESIRED CURRENT READY AGE CONTAINER(S) IMAGE(S) SELECTOR 16 myweb 2 2 2 45m myweb 192.168.110.133:5000/nginx:1.13 app=myweb 17 [root@k8s-master ~]# kubectl delete rc myweb 18 replicationcontroller "myweb" deleted 19 [root@k8s-master ~]# kubectl get rc -o wide 20 No resources found. 21 [root@k8s-master ~]# kubectl get all -o wide 22 NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE 23 hpa/myweb ReplicationController/myweb 5% <waiting> 1 8 38m 24 25 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 26 svc/kubernetes 10.254.0.1 <none> 443/TCP 23d <none> 27 [root@k8s-master ~]# kubectl get hpa -o wide 28 NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE 29 myweb ReplicationController/myweb 5% <waiting> 1 8 39m 30 [root@k8s-master ~]# kubectl delete hpa myweb 31 horizontalpodautoscaler "myweb" deleted 32 [root@k8s-master ~]# kubectl get hpa -o wide 33 No resources found. 34 [root@k8s-master ~]# kubectl get all -o wide 35 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 36 svc/kubernetes 10.254.0.1 <none> 443/TCP 23d <none> 37 [root@k8s-master ~]#
測不出來，k8s的hpa伸縮和縮減Pod的現象，不管修改內存、CPU參數，還是請求參數，先放到這里吧，以后有機會再來看看。這里沒有完成伸縮的具體原因是因為下面這段話。
metrics-server，從 Kubernetes 1.8開始，資源使用指標，例如容器 CPU 和內存使用率，可通過 Metrics API 在 Kubernetes 中獲得。此 API 不存儲指標值，因此想要獲取某個指定節點10分鍾前的資源使用量是不可能的。在此之前，kubernetes對容器的監控是通過hepater來完成的。之所以Metrics-server會替換掉hepater成為新的k8s 監控組件，在於其使得kubernetes在監控方面與其他功能保持了一致，不再像是一個割裂開的功能，比如風格統一的監控指標api，kubectl top命令等。這里順便提一下另外一個組件cadvisor，該監控組件本身並非kubernetes內置，但kubelete內置了部分cadvisor功能，因此kubelete可以獲取每個節點的容器監控信息。
5、查看一下我的k8s的版本，如下所示：
1 [root@k8s-master ~]# kubectl version 2 Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"269f928217957e7126dc87e6adfa82242bfe5b1e", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} 3 Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"269f928217957e7126dc87e6adfa82242bfe5b1e", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} 4 [root@k8s-master ~]#
由於heapster停止更新，這里也使用metrics-server來進行監控了。
1 [root@k8s-master ~]# cd k8s/ 2 [root@k8s-master k8s]# ls 3 book-master.war dashboard dashboard.zip deploy health heapster hpa namespace pod rc skydns skydns.zip svc tomcat_demo tomcat_demo.zip 4 [root@k8s-master k8s]# mkdir metrics 5 [root@k8s-master k8s]# cd metrics/ 6 [root@k8s-master metrics]# ls 7 [root@k8s-master metrics]# wget -c https://github.com/kubernetes-sigs/metrics-server/archive/v0.3.6.zip 8 --2020-06-29 10:24:47-- https://github.com/kubernetes-sigs/metrics-server/archive/v0.3.6.zip 9 Resolving github.com (github.com)... 13.250.177.223 10 Connecting to github.com (github.com)|13.250.177.223|:443... connected. 11 HTTP request sent, awaiting response... 302 Found 12 Location: https://codeload.github.com/kubernetes-sigs/metrics-server/zip/v0.3.6 [following] 13 --2020-06-29 10:24:49-- https://codeload.github.com/kubernetes-sigs/metrics-server/zip/v0.3.6 14 Resolving codeload.github.com (codeload.github.com)... 54.251.140.56 15 Connecting to codeload.github.com (codeload.github.com)|54.251.140.56|:443... connected. 16 HTTP request sent, awaiting response... 200 OK 17 Length: unspecified [application/zip] 18 Saving to: ‘v0.3.6.zip’ 19 20 [ <=> ] 9,374,036 183KB/s in 69s 21 22 2020-06-29 10:26:01 (132 KB/s) - ‘v0.3.6.zip’ saved [9374036] 23 24 [root@k8s-master metrics]#
查看下載的zip包並進行解壓縮操作，如下所示：
1 [root@k8s-master metrics]# unzip v0.3.6.zip 2 3 [root@k8s-master metrics]# ls 4 metrics-server-0.3.6 v0.3.6.zip 5 [root@k8s-master metrics]# ll 6 total 9156 7 drwxr-xr-x 8 root root 335 Oct 14 2019 metrics-server-0.3.6 8 -rw-r--r-- 1 root root 9374036 Jun 29 10:26 v0.3.6.zip 9 [root@k8s-master metrics]#
查看自己的jdk版本，並進入到指定的目錄，如下所示：
1 [root@k8s-master metrics]# ls 2 metrics-server-0.3.6 v0.3.6.zip 3 [root@k8s-master metrics]# cd metrics-server-0.3.6/deploy/1. 4 1.7/ 1.8+/ 5 [root@k8s-master metrics]# cd metrics-server-0.3.6/deploy/1. 6 1.7/ 1.8+/ 7 [root@k8s-master metrics]# cd metrics-server-0.3.6/deploy/1.8+/ 8 [root@k8s-master 1.8+]# ls 9 aggregated-metrics-reader.yaml auth-delegator.yaml auth-reader.yaml metrics-apiservice.yaml metrics-server-deployment.yaml metrics-server-service.yaml resource-reader.yaml 10 [root@k8s-master 1.8+]# java -version 11 openjdk version "1.8.0_181" 12 OpenJDK Runtime Environment (build 1.8.0_181-b13) 13 OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) 14 [root@k8s-master 1.8+]#
現在開始修改鏡像地址，如下所示：
首先將需要的鏡像下載下來，並上傳到自己的私有倉庫里面，如下所示：
1 [root@k8s-master ~]# docker pull docker.io/htcfive/metrics-server-amd64 2 Using default tag: latest 3 Trying to pull repository docker.io/htcfive/metrics-server-amd64 ... 4 sha256:0122b32b24dcb04ac5131cecdefd8abce0c8a3359605dd17b394acc9fc49de1d: Pulling from docker.io/htcfive/metrics-server-amd64 5 e8d8785a314f: Pull complete 6 b2f4b24bed0d: Pull complete 7 Digest: sha256:0122b32b24dcb04ac5131cecdefd8abce0c8a3359605dd17b394acc9fc49de1d 8 Status: Downloaded newer image for docker.io/htcfive/metrics-server-amd64:latest 9 [root@k8s-master ~]# docker images docker.io/htcfive/metrics-server-amd64 10 REPOSITORY TAG IMAGE ID CREATED SIZE 11 docker.io/htcfive/metrics-server-amd64 latest 77da73af4258 6 months ago 39.9 MB 12 [root@k8s-master ~]# docker tag docker.io/htcfive/metrics-server-amd64:latest 192.168.110.133:5000/docker.io/htcfive/metrics-server-amd64:latest 13 [root@k8s-master ~]# docker push 192.168.110.133:5000/docker.io/htcfive/metrics-server-amd64:latest 14 The push refers to a repository [192.168.110.133:5000/docker.io/htcfive/metrics-server-amd64] 15 7bf3709d22bb: Pushed 16 932da5156413: Pushed 17 latest: digest: sha256:0122b32b24dcb04ac5131cecdefd8abce0c8a3359605dd17b394acc9fc49de1d size: 738 18 [root@k8s-master ~]#
修改鏡像地址，metrics-server默認使用node的主機名，但是coredns里面沒有物理機主機名的解析，部署的時候添加一個參數--kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP。這里直接通過InternalIP進行訪問，忽略客戶端證書kubelet-insecure-tls。
1 [root@k8s-master 1.8+]# cat metrics-server-deployment.yaml 2 --- 3 apiVersion: v1 4 kind: ServiceAccount 5 metadata: 6 name: metrics-server 7 namespace: kube-system 8 --- 9 apiVersion: apps/v1 10 kind: Deployment 11 metadata: 12 name: metrics-server 13 namespace: kube-system 14 labels: 15 k8s-app: metrics-server 16 spec: 17 selector: 18 matchLabels: 19 k8s-app: metrics-server 20 template: 21 metadata: 22 name: metrics-server 23 labels: 24 k8s-app: metrics-server 25 spec: 26 serviceAccountName: metrics-server 27 volumes: 28 # mount in tmp so we can safely use from-scratch images and/or read-only containers 29 - name: tmp-dir 30 emptyDir: {} 31 containers: 32 - name: metrics-server 33 # image: k8s.gcr.io/metrics-server-amd64:v0.3.6 34 image: 192.168.110.133:5000/docker.io/htcfive/metrics-server-amd64:latest 35 # imagePullPolicy: Always 36 imagePullPolicy: IfNotPresent 37 command: 38 - /metrics-server 39 - --kubelet-preferred-address-types=InternalIP 40 - --kubelet-insecure-tls 41 volumeMounts: 42 - name: tmp-dir 43 mountPath: /tmp
創建metrics-server報錯了，先放置吧，搜了一堆也沒有解決，后續再來看看吧。
1 [root@k8s-master 1.8+]# kubectl create -f . 2 Error from server (BadRequest): error when creating "aggregated-metrics-reader.yaml": ClusterRole in version "v1" cannot be handled as a ClusterRole: no kind "ClusterRole" is registered for version "rbac.authorization.k8s.io/v1" 3 Error from server (BadRequest): error when creating "auth-delegator.yaml": ClusterRoleBinding in version "v1beta1" cannot be handled as a ClusterRoleBinding: no kind "ClusterRoleBinding" is registered for version "rbac.authorization.k8s.io/v1beta1" 4 Error from server (BadRequest): error when creating "auth-reader.yaml": RoleBinding in version "v1beta1" cannot be handled as a RoleBinding: no kind "RoleBinding" is registered for version "rbac.authorization.k8s.io/v1beta1" 5 Error from server (AlreadyExists): error when creating "metrics-server-deployment.yaml": serviceaccounts "metrics-server" already exists 6 Error from server (AlreadyExists): error when creating "metrics-server-service.yaml": services "metrics-server" already exists 7 Error from server (BadRequest): error when creating "resource-reader.yaml": ClusterRole in version "v1" cannot be handled as a ClusterRole: no kind "ClusterRole" is registered for version "rbac.authorization.k8s.io/v1" 8 Error from server (BadRequest): error when creating "resource-reader.yaml": ClusterRoleBinding in version "v1" cannot be handled as a ClusterRoleBinding: no kind "ClusterRoleBinding" is registered for version "rbac.authorization.k8s.io/v1" 9 [unable to recognize "metrics-apiservice.yaml": no matches for apiregistration.k8s.io/, Kind=APIService, unable to recognize "metrics-server-deployment.yaml": no matches for apps/, Kind=Deployment] 10 [root@k8s-master 1.8+]#