k8s filebeat sidecar模式收集ingress nginx日志並可視化展示

本文默認k8s環境以及已經部署ingress controller

公司所用ingress監控是由prometheus+grafana進行，但是監控不夠全面，故使用filebeat去采集ingress日志，並自主進行可視化展示

1、ingress nginx日志數據落盤

在ingress controller中將configmap改為

kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-nginx-controller
data:
  access-log-path: /var/log/nginx/access.log
  compute-full-forwarded-for: 'true'
  enable-vts-status: 'true'
  error-log-path: /var/log/nginx/error.log
  forwarded-for-header: X-Forwarded-For
  log-format-upstream: >-
    {"@timestamp":
    "$time_iso8601","remote_addr":"$remote_addr","x-forward-for":"$http_x_forwarded_for","request_id":"$req_id","remote_user":"$remote_user","bytes_sent":$bytes_sent,"request_time":$request_time,"status":$status,"vhost":"$host","request_proto":"$server_protocol","path":"$uri","request_query":"$args","request_length":$request_length,"duration":$request_time,"method":"$request_method","http_referrer":"$http_referer","http_user_agent":"$http_user_agent","upstream-sever":"$proxy_upstream_name","proxy_alternative_upstream_name":"$proxy_alternative_upstream_name","upstream_addr":"$upstream_addr","upstream_response_length":$upstream_response_length,"upstream_response_time":$upstream_response_time,"upstream_status":$upstream_status}
  use-forwarded-headers: 'true'

2、生成filebeat鏡像

新建目錄，目錄如下

 

 

dockerfile

FROM  million12/centos-supervisor:4.0.2
WORKDIR /usr/local
ADD filebeat-7.5.0-linux-x86_64.tar.gz .
RUN ln -s filebeat-7.5.0-linux-x86_64  filebeat \
 && cd filebeat       \
 && mkdir  config     \
 && chmod +x filebeat \
 && cp filebeat.yml config/ \ 
 && yum -y install logrotate crontabs


COPY supervisord.conf /etc/supervisord.conf

RUN mkdir -p /var/log/supervisor
EXPOSE 22 80
CMD ["/usr/bin/supervisord"]

因為需要使用logrotate進行日志輪轉，需要安裝

logrotate crontabs

supervisord.conf配置如下

[supervisord]
nodaemon=true

[program:cron]
command=/usr/sbin/crond -i

[program:filebeat]
command=/usr/local/filebeat/filebeat -c /usr/local/filebeat/config/filebeat.yml

 

3、修改原有ingress controller depl，將filebeat與ingress controller放到同一pod中，使用emptydir卷共享ingress日志，使filebeat能夠讀取，另外一個是面對日志的持續正常如何處理，這里使用logrotate，將logrotate在filebeat中配置，盡量對ingress影響小點，首先增加filebeat configmap

 

kind: ConfigMap
apiVersion: v1
metadata:
  name: filebeat-config
data:
  filebeat.yml: |
    filebeat.inputs:
      - type: log
        enabled: true 
        paths:
          - /var/log/nginx/access.log
        json.keys_under_root: true
        json.overwrite_keys: true
        json.add_error_key: true      
        json.ignore_decoding_error: true      
        tags: ["access"]
      - type: log
        enabled: true 
        paths:
          - /var/log/nginx/error.log
        json.keys_under_root: true
        json.overwrite_keys: true
        json.add_error_key: true      
        json.ignore_decoding_error: true      
        tags: ["error"]
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    setup.template.settings:
      index.number_of_shards: 3
    output.elasticsearch:
      hosts: ["es-local.nxgp.svc.cluster.local:9200"]
      index: "nginx_log-%{+yyyy.MM.dd}"
      indices:
        - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}"
          when.contains:
            tags: "access"
        - index: "nginx_error-%{[beat.version]}-%{+yyyy.MM.dd}"
          when.contains:
            tags: "error"
    setup.template.name: "nginx_log"
    setup.template.pattern: "nginx_*"
    setup.template.enabled: true
    setup.ilm.enabled: false
    setup.template.overwrite: false

 

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-ingress-logrotate
data:
  nginx: |
    /var/log/nginx/*.log {
      su root root
      size 50M
      notifempty
      copytruncate
      rotate 3
      missingok
      compress
      dateext
      dateformat .%Y%m%d-%H
  }

然后進行depl更新，只展示新增部分

      volumes:
        - name: ingress-log
          emptyDir: {}
        - name: filebeat-config
          configMap:
            name: filebeat-config
            defaultMode: 420
        - name: logrotateconf
          configMap:
            name: nginx-ingress-logrotate
            items:
              - key: nginx
                path: nginx
            defaultMode: 420
      containers:
        - name: controller
          volumeMounts:
            - name: ingress-log
              mountPath: /var/log/nginx/
        - name: filebeat
          image: 'xxx/filebeat:7.5.0'
          resources:
            limits:
              cpu: '2'
              memory: 2Gi
            requests:
              cpu: '1'
              memory: 1Gi
          volumeMounts:
            - name: filebeat-config
              mountPath: /usr/local/filebeat/config/
            - name: ingress-log
              mountPath: /var/log/nginx/
            - name: logrotateconf
              mountPath: /etc/logrotate.d/nginx
              subPath: nginx
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 300
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      serviceAccount: ingress-nginx
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

 

 

logrotate是按天更新，更新時間不定

 

 

輪轉效果內存占用高的是還沒有進行打包的







4、es可視化展示

(1)PV

(2)UV

(3)Top10(接口訪問量)

(4)Top10(客戶端IP訪問占比)

(5)Top10(最慢接口)

(6)后端upstream占比

(7)實時流量

(8)客戶端訪問占比

(9)平均並發數

(10)異常狀態碼統計

 

(11)總流量

(12)接口異常響應碼

(13)接口訪問耗時占比

(14)每10秒接口訪問平均耗時

(15)每10秒接口訪問最大耗時

(16)狀態碼統計

(17)訪問量趨勢圖

(18)超過30秒以上的接口

(19)超過30秒以上的接口出現次數