kubernetes之配置Metrics Server

Kubernetes 1.8 關於資源使用情況的 metrics，可以通過 Metrics API 獲取到， Kubernetes 1.11 已經廢棄 heapster。這里我們基於 Kubernetes 1.14.1 版本安裝 Metrics Server。

首先，先說明下集群環境：

[root@node-01]# kubectl get nodes
NAME      STATUS   ROLES    AGE    VERSION
node-01   Ready    master   2d1h   v1.14.1
node-02   Ready    master   2d1h   v1.14.1
node-03   Ready    master   2d1h   v1.14.1
node-04   Ready    <none>   2d1h   v1.14.1
node-05   Ready    <none>   2d1h   v1.14.1
node-06   Ready    <none>   2d1h   v1.14.1

當整個集群部署完成后，kubectl top 命令不會返回任何內容，因為 Heapster 和 metrics-server 都沒有安裝，但是自 Kubernetes 1.11版本后 heapster已經被廢棄了，取而代之的是更豐富的 metrics-server。

配置 /etc/kubernetes/manifests/kube-controller-manager.yaml

--horizontal-pod-autoscaler-use-rest-clients=true

kubedam 創建的集群，修改配置文件后會自動加載。如果手動創建的集群，需要重啟kube-controller-manager服務。

准備部署 Metrics Server 的 yaml文件

[root@node-01]# git clone https://github.com/kubernetes-incubator/metrics-server

下載完成后還需要對 metrics-server/deploy/1.8+/resource-reader.yaml文件進行修改，需要修改的內容如下：

[root@node-01 1.8+]# cat resource-reader.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - namespaces      # 增加此行
  - nodes/stats
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

修改 metrics-server/deploy/1.8+/metrics-server-deployment.yaml文件：

[root@node-01 1.8+]# cat metrics-server-deployment.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.2
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP   # 如果不配置此項，會報錯找不到node
        imagePullPolicy: Always
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

上面如果報錯是因為 node-01 和 node-02 是一個獨立的 Kubernetes 演示環境，只是修改了這兩個節點系統的 /etc/hosts文件，而並沒有內網的 DNS 服務器，所以 metrics-server 中不認識 node-01 和 node-02 的名字。

修改完成就可以正式部署了：

[root@node-01 1.8+]# kubectl apply -f .
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.extensions/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

Metrics Server 相關 pod 、service 默認部署在 kube-system的 NAMESPACE 下：

[root@node-01 1.8+]# kubectl get pods -n kube-system | grep metrics
metrics-server-5845cc8fd4-kkq6b         1/1     Running   0          18m

[root@node-01 1.8+]# kubectl get svc -n kube-system | grep metrics
metrics-server            ClusterIP   10.245.141.103   <none>        443/TCP                   20m

部署完成后使用如下命令查看node相關指標,需要等30s左右的時間：

[root@node-01 1.8+]# kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[
{"metadata":{"name":"node-02","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/node-02","creationTimestamp":"2019-05-08T08:17:11Z"},"timestamp":"2019-05-08T08:17:01Z","window":"30s","usage":{"cpu":"221367011n","memory":"1914616Ki"}},
{"metadata":{"name":"node-03","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/node-03","creationTimestamp":"2019-05-08T08:17:11Z"},"timestamp":"2019-05-08T08:17:08Z","window":"30s","usage":{"cpu":"198021879n","memory":"1809160Ki"}},
{"metadata":{"name":"node-04","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/node-04","creationTimestamp":"2019-05-08T08:17:11Z"},"timestamp":"2019-05-08T08:17:03Z","window":"30s","usage":{"cpu":"55570780n","memory":"719012Ki"}},
{"metadata":{"name":"node-05","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/node-05","creationTimestamp":"2019-05-08T08:17:11Z"},"timestamp":"2019-05-08T08:17:01Z","window":"30s","usage":{"cpu":"60116633n","memory":"851180Ki"}},
{"metadata":{"name":"node-06","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/node-06","creationTimestamp":"2019-05-08T08:17:11Z"},"timestamp":"2019-05-08T08:16:59Z","window":"30s","usage":{"cpu":"51157291n","memory":"677532Ki"}},
{"metadata":{"name":"node-01","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/node-01","creationTimestamp":"2019-05-08T08:17:11Z"},"timestamp":"2019-05-08T08:17:02Z","window":"30s","usage":{"cpu":"263183209n","memory":"2460972Ki"}}]}

Metrics API

Metrics Server 從 Kubernetes 集群中每個 Node 上 kubelet 的 API 收集 metrics 數據。通過 Metrics API 可以獲取Kubernetes 資源的 Metrics 指標，Metrics API 掛載/apis/metrics.k8s.io/下。 可以使用kubectl top命令訪問 Metrics API，例如:

[root@node-01 ~]# kubectl top pods
NAME                                CPU(cores)   MEMORY(bytes)
my-nginx-6785b88976-7rrll           0m           1Mi
nginx-deployment-6d6fdc59f7-pfcfj   1m           1Mi
nginx-deployment-6d6fdc59f7-vcclz   1m           1Mi
[root@node-01 ~]# kubectl top nodes
NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-01   276m         6%     2403Mi          31%
node-02   245m         6%     1868Mi          24%
node-03   206m         5%     1766Mi          22%
node-04   74m          1%     703Mi           9%
node-05   77m          1%     832Mi           10%
node-06   56m          1%     661Mi           8%

至此，Kubernetes 集群中的 Metrics Server 就配置完成了。但是在dashboard中看不到內存和CPU信息，而如果使用heapster則能看到。

所有yaml文件如下

# cat aggregated-metrics-reader.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods"]
  verbs: ["get", "list", "watch”]


# cat auth-delegator.yaml
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

# cat auth-reader.yaml
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
—————

# cat metrics-apiservice.yaml
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

# cat metrics-server-deployment.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.2
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        imagePullPolicy: Always
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

# cat metrics-server-service.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443


# cat resource-reader.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - namespaces      # 增加此行
  - nodes/stats
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system