K8S-ELK日志系統之四：filebeat

1、filebeat介紹

Filebeat是用於轉發和集中日志數據的輕量級傳送工具。Filebeat監視您指定的日志文件或位置，收集日志事件，並將它們轉發到Elasticsearch或 Logstash或者kafka等
Filebeat的工作方式如下：啟動Filebeat時，它將啟動一個或多個輸入，這些輸入將在為日志數據指定的位置中查找。對於Filebeat所找到的每個日志，Filebeat都會啟動收集器。每個收集器都讀取單個日志以獲取新內容，並將新日志數據發送到libbeat，libbeat將聚集事件，並將聚集的數據發送到為Filebeat配置的輸出。

如圖：

　　

 

2、部署方式

      k8s上可以部署方式：

filebeat和應用容器運行在一個pod，作為sidercar模式，搜集日志，這樣將產生較多的sidercar容器 

filebeat作為daemonSet運行在各node節點，搜集docker日志，配置簡單，但是日志不好分類

filebeat和應用運行同一容器，本次使用的方式

3、制作Dockerfile

使用tomcat官方最新鏡像：tomcat:latest

filebeat版本：7.16.2，下載地址 https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.16.2-amd64.deb

FROM tomcat:latest
ENV TIME_ZONE Asia/Shanghai
COPY  filebeat-7.16.2-amd64.deb /
COPY sources.list /etc/apt/
COPY entrypoint.sh /
RUN apt-get update -y &&  \
    apt-get install vim inetutils-ping net-tools telnet -y && \
    echo "${TIME_ZONE}" > /etc/timezone && \
    ln -sf /usr/share/zoneinfo/${TIME_ZONE} /etc/localtime && \
    chmod +x /entrypoint.sh && \
    dpkg -i /filebeat-7.16.2-amd64.deb
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/bin/filebeat","-c","/etc/filebeat/filebeat.yml","-e"]
EXPOSE 8080

#!/bin/bash
/usr/local/tomcat/bin/catalina.sh run 1>/dev/null 2>&1 &
exec "$@"

# 默認注釋了源碼鏡像以提高 apt update 速度，如有需要可自行取消注釋
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye main contrib non-free
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-updates main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-updates main contrib non-free

deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-backports main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-backports main contrib non-free

deb https://mirrors.tuna.tsinghua.edu.cn/debian-security bullseye-security main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian-security bullseye-security main contrib non-free

4、yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: filebeat-configmap
  namespace: es
data:
  filebeat-config: |
    filebeat.inputs:
    - type: log
      enabled: true
      paths:
      - /var/log/*.log
      fields:
        app_id: system
    - type: log
      enabled: true
      paths:
      - /usr/local/tomcat/logs/*access*
      fields:
        app_id: tomcat
    filebeat.conf.modules:
      path: ${path.conf}/modules.d/*yml
      reload.enabled: false
    setup.template.setting:
      index.number_of_shards: 1
    output.kafka:
      hosts: ["10.0.8.111:30209","10.0.8.112:30209","10.0.8.113:30209"]
      enable: true
      required_acks: 1
      topic: "%{[fields.app_id]}"
      partition.round_robin:
        reachable_only: false
      keep_alive: 10s

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: tomcat
  namespace: es
spec:
  replicas: 1
  selector:
    matchLabels: 
      app: tomcat
  template:
    metadata:
      name: tomcat
      namespace: es
      labels:
        app: tomcat
    spec:
      containers:
      - name: tomcat
        image: harbor.myland.com/baseimages/tomcat-filebeat:7.16.0
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 300m
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: file-beatconfig-file
          mountPath: /etc/filebeat/

        ports:
        - name: tomcat
          containerPort: 8080
          protocol: TCP
      volumes:
      - name: file-beatconfig-file
        configMap: 
          name: filebeat-configmap
          items:
          - key: filebeat-config
            path: filebeat.yml

5、驗證

　　可以看到，kafka上產生system和tomcat兩個topic，搜集到最新日志