Fluentd 例子

0. 安裝、啟動

安裝

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

更新：最近貌似會安裝最新版本2.3.2安裝會有問題，可以手動下載td-agent-2.2.1-0.el6.x86_64.rpm，手動安裝

sudo rpm -ivh td-agent-2.2.1-0.el6.x86_64.rpm

 

啟動

/opt/td-agent/embedded/bin/ruby /usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --use-v1-config --group td-agent --daemon /var/run/td-agent/td-agent.pid

1. http輸入，stdout輸出

例子

<source>
    @type http
    port 8888
    bind 0.0.0.0
</source>

<match td3.**>
    type stdout
</match>

請求

curl http://10.218.139.216:8888/td3 -d 'json={"hi":"abc"}'

結果（/var/log/td-agent/td-agent.log）

2016-02-03 10:17:00 +0800 td3: {"hi":"abc"}

格式：

<match td2.**>
    @type mongo
    host 10.218.139.216
    port 27017
    database db_log
    collection db_col
    time_format %H-%M-%S:%s  #時-分-秒.毫秒
    localtime                                  #本地時間
    flush_interval 10s
</match>

 

2. http輸入，文件輸出

例子

## Source descriptions
# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
<source>
    @type http
    port 8888
    bind 0.0.0.0
</source>

## Output
# File output
# match tag=td.*.* and output to file
<match td.**>
    @type file
    path /home/jihite/monitor/test.log
    flush_interval 10s
</match>

## match tag=td2.*.* and output to file
<match td2.**>
    @type file
    path /home/jihite/monitor/test_2.log
    flush_interval 10s
</match>

http請求：

curl http://10.218.139.216:8888/td2 -d 'json={"hi":1}'

linux命令：

發POST請求工具：

結果查看：

2016-02-01T15:46:06+08:00       td      {"hi":1}

3. http輸入，mongoDB輸出

例子

## Source descriptions
# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
<source>
    @type http
    port 8888
    bind 0.0.0.0
</source>

## Output
# MongoDB output
# match tag=td2.*.* and output to file
<match td2.**>
    @type mongo
    host 10.218.139.216
    port 27017
    database db_log
    collection db_col
    time_key time
    flush_interval 10s
</match>

請求

curl 10.218.139.216:8888/td2 -d 'json={"hi":"123"}'

 

結果查詢：

> show dbs
db_log  0.078GB
local   0.078GB

> use db_log
switched to db db_log

> show collections
db_col
system.indexes

> db.db_col.find()
{ "_id" : ObjectId("56af19dbdfb99f0f50000001"), "hi" : 2, "time" : ISODate("2016-02-01T08:39:47Z") }

4. python寫日志

安裝插件

sudo pip install fluent-logger

參考：https://github.com/fluent/fluent-logger-python

日志記錄示例：

import logging
from fluent import handler

custom_format = {
  'host': '%(hostname)s',
  'where': '%(module)s.%(funcName)s',  #具體到文件、函數
  'type': '%(levelname)s',
  'stack_trace': '%(exc_text)s'
}

logging.basicConfig(level=logging.DEBUG)

l = logging.getLogger('fluent.test')

h = handler.FluentHandler('python.mongo', host='10.218.139.216', port=24224)
formatter = handler.FluentRecordFormatter(custom_format)
h.setFormatter(formatter)

l.addHandler(h)

def funcs():
    l.warning("hello")
    l.error("hello  error")

l.info('{"from": "1", "to": "2"}')

修改td-agent.conf

<source>
    @type forward
    port 24224
</source>

<match python.mongo.**>
    @type mongo
    host 10.218.139.216
    port  27017
    database db_python
    collection col_python
    time_key time
    flush_interval 10s
</match>