helm部署Filebeat + ELK

系統架構圖：

 

1) 多個Filebeat在各個Node進行日志采集，然后上傳至Logstash

2) 多個Logstash節點並行（負載均衡，不作為集群），對日志記錄進行過濾處理，然后上傳至Elasticsearch集群

3) 多個Elasticsearch構成集群服務，提供日志的索引和存儲能力

4) Kibana負責對Elasticsearch中的日志數據進行檢索、分析

1. Elasticsearch部署

官方chart地址：https://github.com/elastic/helm-charts/tree/master/elasticsearch

創建logs命名空間

kubectl create ns logs

添加elastic helm charts 倉庫

helm repo add elastic https://helm.elastic.co

安裝

helm install --name elasticsearch elastic/elasticsearch --namespace logs

參數說明

image: "docker.elastic.co/elasticsearch/elasticsearch"

imageTag: "7.2.0"

imagePullPolicy: "IfNotPresent"

podAnnotations: {}

esJavaOpts: "-Xmx1g -Xms1g"

resources:

  requests:

    cpu: "100m"

    memory: "2Gi"

  limits:

    cpu: "1000m"

    memory: "2Gi"

volumeClaimTemplate:

  accessModes: [ "ReadWriteOnce" ]

  storageClassName: "nfs-client"

  resources:

    requests:

      storage: 50Gi

2. Filebeat部署

官方chart地址：https://github.com/elastic/helm-charts/tree/master/filebeat

Add the elastic helm charts repo

helm repo add elastic https://helm.elastic.co

Install it

helm install --name filebeat elastic/filebeat --namespace logs

參數說明：

image: "docker.elastic.co/beats/filebeat"

imageTag: "7.2.0"

imagePullPolicy: "IfNotPresent"

resources:

  requests:

    cpu: "100m"

    memory: "100Mi"

  limits:

    cpu: "1000m"

    memory: "200Mi"

那么問題來了，filebeat默認收集宿主機上docker的日志路徑：/var/lib/docker/containers。如果我們修改了docker的安裝路徑要怎么收集呢，很簡單修改chart里的DaemonSet文件里邊的hostPath參數：

      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers   #改為docker安裝路徑

對java程序的報錯異常log實現多行合並，用multiline定義正則來匹配。

filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: docker
      containers.ids:
      - '*'
      multiline.pattern: '^[0-9]'
      multiline.negate: true
      multiline.match: after
      processors:
      - add_kubernetes_metadata:
          in_cluster: true

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'

3. Kibana部署

官方chart地址：https://github.com/elastic/helm-charts/tree/master/kibana

Add the elastic helm charts repo

helm repo add elastic https://helm.elastic.co

Install it

helm install --name kibana elastic/kibana --namespace logs

參數說明：

elasticsearchHosts: "http://elasticsearch-master:9200"

replicas: 1

image: "docker.elastic.co/kibana/kibana"

imageTag: "7.2.0"

imagePullPolicy: "IfNotPresent"

resources:

  requests:

    cpu: "100m"

    memory: "500m"

  limits:

    cpu: "1000m"

    memory: "1Gi"

4. Logstash部署

官方chart地址：https://github.com/helm/charts/tree/master/stable/logstash

安裝

$ helm install --name logstash stable/logstash --namespace logs

參數說明：

image:

  repository: docker.elastic.co/logstash/logstash-oss

  tag: 7.2.0

  pullPolicy: IfNotPresent

persistence:

  enabled: true

  storageClass: "nfs-client"

  accessMode: ReadWriteOnce

  size: 2Gi

匹配label：json的pod日志，沒有的話正常收集。

filebeatConfig:
  filebeat.yml: |
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition:
                equals:
                  kubernetes.labels.logFormat: "json"
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  json.keys_under_root: true
                  json.overwrite_keys: true
                  json.add_error_key: true
            - config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  processors:
                    - add_kubernetes_metadata:
                        in_cluster: true
    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'

5. Elastalert部署

官方chart地址：https://github.com/helm/charts/tree/master/stable/elastalert

安裝

helm install -n elastalert ./elastalert --namespace logs

　效果圖：