flume通過avro對接（匯總數據）

使用場景：

把多台服務器(flume generator)上面的日志匯總到一台或者幾台服務器上面（flume collector），然后對接到kafka或者HDFS上

 

Flume Collector服務端

vim flume-server.properties

# agent1 name
a1.channels = c1
a1.sources = r1
a1.sinks = k1

#set channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# other node, slave to master
a1.sources.r1.type = avro
a1.sources.r1.bind = master
a1.sources.r1.port = 52020

# set sink to logger
a1.sinks.k1.type = logger
      

a1.sources.r1.channels = c1
a1.sinks.k1.channel=c1

啟動：

## Master
/usr/local/flume/bin/flume-ng agent –f flume-server.properties –name a1

 

Flume Generator客戶端

vim flume-client.properties

# a1 name
a1.channels = c1
a1.sources = r1
a1.sinks = k1

#set channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.sources.r1.channels = c1
a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /root/test.log

# set sink1
#a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = master
a1.sinks.k1.port = 52020

a1.sources.r1.channels = c1
a1.sinks.k1.channel=c1

啟動：

分別在slave1和slave2服務器上面啟動

/usr/local/flume/bin/flume-ng agent –f flume-client.properties –name a1

 

啟動之后，在slave1和slave2服務器上面分別執行以下操作：

#slave1
echo "wangzai slave1" > /root/test.log

#slave2
echo "wangzai slave2" > /root/test.log

 

結果：

Master: