Elasticsearch+Fluentd+Kibana+docker-compose容器日志監控

核心思路：

Elasticsearch+Kibana部署在單獨的服務器上，Fluentd運行在承載docker容器的服務器上，通過容器內置參數logging driver來獲取日志，

 

docker-elastickibana.yml

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.1.4
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
    volumes:
      - ./esdata:/usr/share/elasticsearch/data
    networks:
      - efknet
    ports:
      - "9200:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana-oss:6.1.4
    container_name: kibana
    networks:
      - efknet
    ports:
      - "5601:5601"


networks:
  efknet:

docker-fluentd.yml

version: '2'
services:
  fluentd:
    build: .
    volumes:
      - ./conf:/fluentd/etc
      - ./log:/fluentd/log
    container_name: fluentd
    networks:
      - fluent_net
    ports:
      - "24224:24224"
      - "24224:24224/udp"

networks:
  fluent_net:

fluent.conf

# fluentd/conf/fluent.conf
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match *.**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>
  <store>
    @type stdout
  </store>
</match>

測試容器http_server，核心是logging一欄

version: '3'
services:
  http_server:
    image: httpd
    container_name: http_server
    ports:
      - "80:80"
    networks: 
      - efknet
    logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
        tag: httpd_server
networks:
  efknet:

需要注意的點：

Elasticsearch宿主機/etc/sysctl.conf文件底部添加一行 vm.max_map_count=655360

[root@localhost efk]# sysctl -p
vm.max_map_count = 655360

修改系統ulimit大小

Elasticsearch默認是集群方式啟動，compose中要指明以單節點啟動，一般用於開發環境。

docker中fluentd無法連接到Elasticsearch可能是firewalld的原因，systemctl stop firewalld && systemctl restart docker

Elasticsearch在容器中默認是以id為1000的賬戶在運行，如果-v掛載日志目錄到本地或者文件，需要保證宿主機目錄或文件有寫權限。

Fluentd容器運行時區的問題，可以掛載/etc/localtime到容器中。

暫時沒想到，持續更新。。。