k8s安裝metrics-server

Kubernetes Metrics Server：

• Kubernetes Metrics Server 是 Cluster 的核心監控數據的聚合器，kubeadm 默認是不部署的。

• Metrics Server 供 Dashboard 等其他組件使用，是一個擴展的 APIServer，依賴於 API Aggregator。所以，在安裝 Metrics Server 之前需要先在 kube-apiserver 中開啟 API Aggregator。

• Metrics API 只可以查詢當前的度量數據，並不保存歷史數據。

• Metrics API URI 為 /apis/metrics.k8s.io/，在 k8s.io/metrics 下維護。

• 必須部署 metrics-server 才能使用該 API，metrics-server 通過調用 kubelet Summary API 獲取數據。

一、前提條件

注意：使用 Metrics Server 有必備兩個條件：

1、API Server 啟用 Aggregator Routing 支持。否則 API Server 不識別請求：

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

2、API Server 能訪問 Metrics Server Pod IP。否則 API Server 無法訪問 Metrics Server：

E1223 07:23:04.330206       1 available_controller.go:420] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: Get https://10.171.248.214:4443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

3、啟用API Aggregator，API Aggregation 允許在不修改 Kubernetes 核心代碼的同時擴展 Kubernetes API，即：將第三方服務注冊到 Kubernetes API 中，這樣就可以通過 Kubernetes API 來訪問第三方服務了，例如：Metrics Server API。注：另外一種擴展 Kubernetes API 的方法是使用 CRD（Custom Resource Definition，自定義資源定義）。

二、修改配置

1、檢查 API Server 是否開啟了 Aggregator Routing：查看 API Server 是否具有 --enable-aggregator-routing=true 選項。

[root@master1 ~]# ps -ef | grep apiserver
root      12721  10738  0 20:46 pts/0    00:00:00 grep --color=auto apiserver
root     111575 111557  6 16:42 ?        00:16:51 kube-apiserver --advertise-address=192.168.200.3 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

2、修改每個 API Server 的 kube-apiserver.yaml 配置開啟 Aggregator Routing：修改 manifests 配置后 API Server 會自動重啟生效。

cat /etc/kubernetes/manifests/kube-apiserver.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.200.3:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.200.3
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --enable-aggregator-routing=true            # 添加本行
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.18.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.200.3
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
status: {}

github地址：

https://github.com/kubernetes-sigs/metrics-server/releases

三、安裝

1、下載yaml文件

wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml

2、修改components.yaml文件

cat components.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP   # 刪掉 ExternalIP,Hostname這兩個，這里已經改好了，你那邊要自己核對一下
        - --kubelet-use-node-status-port
        - --kubelet-insecure-tls                    #   加上該啟動參數
        image: k8s.gcr.io/metrics-server/metrics-server:v0.4.1                 # 鏡像地址根據情況修改
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          periodSeconds: 10
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

3、安裝

kubectl apply -f components.yaml

4、查看metrics-server服務狀態

[root@master1 ~]# kubectl get pod -n kube-system | grep metrics-server
metrics-server-6f9f86ddf9-zphlw   1/1     Running   0          11s

5、檢查 API Server 是否可以連通 Metrics Server

[root@master1 ~]# kubectl describe svc metrics-server -n kube-system
Name:              metrics-server
Namespace:         kube-system
Labels:            kubernetes.io/cluster-service=true
                   kubernetes.io/name=Metrics-server
Annotations:       Selector:  k8s-app=metrics-server
Type:              ClusterIP
IP:                10.107.86.150
Port:              <unset>  443/TCP
TargetPort:        main-port/TCP
Endpoints:         10.244.2.20:4443
Session Affinity:  None
Events:            <none>


# 在其他幾個節點ping一下Endpoints的地址
[root@master3 ~]# ping 10.244.2.20
PING 10.244.2.20 (10.244.2.20) 56(84) bytes of data.
64 bytes from 10.244.2.20: icmp_seq=1 ttl=64 time=0.122 ms
64 bytes from 10.244.2.20: icmp_seq=2 ttl=64 time=0.032 ms

6、執行以下命令，檢查節點占用性能情況。

[root@master1 ~]# kubectl top nodes
NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master1   223m         5%     1433Mi          52%       
master2   148m         3%     972Mi           35%       
master3   183m         4%     1123Mi          41%       
node1     225m         5%     1389Mi          50%      


[root@master1 ~]# kubectl top pods
NAME                                      CPU(cores)   MEMORY(bytes)   
details-v1-6fc55d65c9-l97cs               4m           36Mi            
nfs-client-provisioner-6d4469b5b5-ndg5x   4m           20Mi            
productpage-v1-9cf6c76b5-hc2nr            10m          71Mi            
ratings-v1-6f855c5fff-v9ht2               3m           28Mi            
reviews-v1-697f45b7dd-w9fgc               7m           117Mi           
reviews-v2-8c85b467b-jgx6w                6m           113Mi           
reviews-v3-76ccf79776-74vz6               5m           117Mi