[k8s]metricbeat的kubernetes模塊&kube-metric模塊

正確姿勢啟動metricbeat

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - filesystem
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']
  cpu_ticks: false
  
output.elasticsearch:
    hosts: ["http://192.168.x.x:9200"]

setup.template.name: "metricbeat"
setup.template.fields: "fields.yml"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1

setup.kibana.host: "192.168.x.x:5601"
setup.dashboards.enabled: true

./metricbeat -e -c metricbeat.yml -d "publish"

metricbeat的Kubernetes插件

metricbeat.modules:
# Node metrics, from kubelet:
- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  hosts: ["192.168.x.132:10255","192.168.x.133:10255"]

# State metrics from kube-state-metrics service:  #這部分需要kube-state-metrics支撐(Nodes/deploy個數),因此集群里需要安裝kube-state-metrics組件.默認情況下這里沒有的話圖中上半部分沒有數據
- module: kubernetes
  enabled: false  ## 這里一定要是true
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_pod
    - state_container
  period: 10s
  hosts: ["192.168.x.x:8080"]

# Kubernetes events
- module: kubernetes
  enabled: false
  metricsets:
    - event
    
output.elasticsearch:
    hosts: ["http://192.168.x.x:9200"]

既然需要kube-state-metrics了,那就研究下吧.

監控思路

• vm思路:遵從大道至簡的原則
  

• 容器監控思路:
  pod的:
  top cpu
  top mem

• 別人家的監控-只能流口水了
  

kube-state-metrics提供監控k8s集群的metric

kube-state-metrics本質上是提供了k8s集群的metric
部署參考:
https://github.com/kubernetes/kube-state-metrics

克隆代碼: create這幾個即可

訪問:

報錯解決:
kubectl -n kube-system logs -f kube-state-metrics-215312065-1vzg2 -c kube-state-metrics
時候發現一些api錯誤

解決: 給apiserver啟動添加這個參數: 它github上有說

--runtime-config=batch/v2alpha1=true

即目前完整的apiserver參數如下

 kube-apiserver     --service-cluster-ip-range=10.254.0.0/16     --etcd-servers=http://127.0.0.1:2379     --insecure-bind-address=0.0.0.0     --admission-control=ServiceAccount     --service-account-key-file=/root/ssl/ca.key --client-ca-file=/root/ssl/ca.crt    --tls-cert-file=/root/ssl/server.crt --tls-private-key-file=/root/ssl/server.key --allow-privileged=true --runtime-config=batch/v2alpha1=true --v=2

k8s插件監控集群效果圖

vm監控效果圖

貼上kube-metric的yaml

參考: https://github.com/kubernetes/kube-state-metrics

修改點:

• deploy的api版本

• svc為nodeport模式

• 鏡像拉取

docker pull lanny/gcr.io_google_containers_kube-state-metrics_v1.1.0
dokcer pull lanny/gcr.io_google_containers_addon-resizer_1.0
docker tag lanny/gcr.io_google_containers_kube-state-metrics_v1.1.0 gcr.io/google_containers/kube-state-metrics:v1.1.0 
docker tag lanny/gcr.io_google_containers_addon-resizer_1.0 gcr.io/google_containers/addon-resizer:1.0 

$ cat kube-state-metrics-deployment.yaml 
apiVersion: apps/v1beta1 # for versions before 1.8.0 use apps/v1beta1  ## 這里默認是v1beat1,我用的是1.7.8,因此apiVersion: apps/v1beta2是不適用的.
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: quay.io/coreos/kube-state-metrics:v1.1.0
        ports:
        - name: http-metrics
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
      - name: addon-resizer
        image: gcr.io/google_containers/addon-resizer:1.0
        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 30Mi
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        command:
          - /pod_nanny
          - --container=kube-state-metrics
          - --cpu=100m
          - --extra-cpu=1m
          - --memory=100Mi
          - --extra-memory=2Mi
          - --threshold=5
          - --deployment=kube-state-metrics

$ cat kube-state-metrics-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
  annotations:
    prometheus.io/scrape: 'true'
spec:
  type: NodePort ## 這里需要改下,外界通過nodeport方式訪問metric api
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
    protocol: TCP
  selector:
    k8s-app: kube-state-metrics

$ cat kube-state-metrics-service-account.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system