Flume配置
。
flume生成的數據結構
<span style="font-size:18px;">"_index" : "logstash-2013.01.07",
"_type" : "tms_jboss_syslog",
"_id" : "a_M9X_0YSpmE7A_bEzIFiw",
"_score" : 1.0, "_source" : {"@source":"file://localhost.localdomain/tmp/logstash_test.log","@tags":[],"@fields":{},"@timestamp":"2013-01-07T10:53:50.941Z","@source_host":"localhost.localdomain","@source_path":"/tmp/logstash_test.log","@message":"[2013-01-05 11:02:19,969] packBoxNumber eq 00004000000044043412 createdOffice eq VIP_BJ:;null","@type":"tms_jboss_syslog"}</span>
flume配置文件
agent.sources = tail agent.channels = memoryChannel agent.channels.memoryChannel.type = memory agent.sources.tail.channels = memoryChannel agent.sources.tail.type = exec agent.sources.tail.command = tail -F /home/hadoop/flume/conf/es_log/es_log.log agent.sources.tail.interceptors=i1 i2 i3 agent.sources.tail.interceptors.i1.type=regex_extractor agent.sources.tail.interceptors.i1.regex = (\\w.*):(\\w.*):(\\w.*)\\s agent.sources.tail.interceptors.i1.serializers = s1 s2 s3 agent.sources.tail.interceptors.i1.serializers.s1.name = source agent.sources.tail.interceptors.i1.serializers.s2.name = type agent.sources.tail.interceptors.i1.serializers.s3.name = src_path agent.sources.tail.interceptors.i2.type=org.apache.flume.interceptor.TimestampInterceptor$Builder agent.sources.tail.interceptors.i3.type=org.apache.flume.interceptor.HostInterceptor$Builder agent.sources.tail.interceptors.i3.hostHeader = host agent.sinks = elasticsearch agent.sinks.elasticsearch.channel = memoryChannel agent.sinks.elasticsearch.type=org.apache.flume.sink.elasticsearch.ElasticSearchSink agent.sinks.elasticsearch.batchSize=100 agent.sinks.elasticsearch.hostNames=127.0.0.1:9300 agent.sinks.k1.indexType = bar_type agent.sinks.elasticsearch.indexName=logstash agent.sinks.elasticsearch.clusterName=elasticsearch agent.sinks.elasticsearch.serializer=org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer啟動：
../bin/flume-ng agent -c . -f es_log.conf -n agent -Dflume.root.logger=INFO,console
測試數據
website:weblog:login_page weblog data1 website:weblog:profile_page weblog data2 website:weblog:transaction_page weblog data3 website:weblog:docs_page weblog data4 syslog:syslog:sysloggroup syslog data1 syslog:syslog:sysloggroup syslog data2 syslog:syslog:sysloggroup syslog data3 syslog:syslog:sysloggroup syslog data4 syslog:syslog:sysloggroup syslog data5 syslog:syslog:sysloggroup syslog data6
之后就能夠在es集群上看到通過flume導入的數據了
這時候編輯log文件時候會被flume讀入es集群中並實時生成索引 例如以下圖所看到的：
這樣就能夠通過es對日志的實時檢索了