前言:
Flume-ng是數據收集/聚合/傳輸的組件, Flume-ng拋棄了Flume OG原本繁重的zookeeper和Master, Collector, 其整體的架構更加的簡潔和明了. 其基礎組件就Agent進程, 內部又可以細分為Source, Channel, Sink三個組件, Source是數據的輸入源, channel作為消息的管道, 而sink是作為數據流的輸出, Source可以配置多個channel, sink和channel一一對應.
*) 初體驗Flume-ng
以CDH5版本的Hadoop為基礎, 編輯一個簡單的agent用例
1). flume-ng的安裝目錄
which flume-ng
/usr/bin/flume-ng
2). flume-ng命令
[<username>@<hostname> study_flume]# flume-ng help
Usage: /usr/lib/flume-ng/bin/flume-ng <command> [options]... commands: help display this help text agent run a Flume agent avro-client run an avro Flume client version show Flume version info
3). 兩個簡單Demo
3.1). 簡單日志輸出Demo
以netcat型監聽服務作為source, 以日志輸出作為sink的Demo
#) 配置 flume-demo-logger.conf
mkdir -p conf
touch conf/flume-demo-logger.conf
conf/flume-demo-logger.conf的內容如下:
agent1.sources = source1 agent1.sinks = sink1 agent1.channels = channel1 agent1.sources.source1.type = netcat agent1.sources.source1.bind = 0.0.0.0 agent1.sources.source1.port = 9999 agent1.sinks.sink1.type = logger agent1.channels.channel1.type = memory agent1.channels.channel1.capacity = 1000 agent1.channels.channel1.transactionCapacity = 100 agent1.sources.source1.channels = channel1 agent1.sinks.sink1.channel = channel1
注: 從運維的角度, bind最好是0.0.0.0, port最好小於10000(監聽端口最好避免與臨時端口爭搶), 同時sources.<source_name>.channels是復數形式, sinks.<sink_name>.channel是單數形式, 請注意一對多, 一對一的對應關系.
#) 啟動flume-ng agent
flume-ng agent -c conf -f conf/flume-demo-logger.conf --name agent1 -Dflume.root.logger=INFO,Console
14/07/03 14:14:34 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:9999]
表示已經監聽在9999端口上了
#) 啟動telnet進行測試
telnet 127.0.0.1 9999
注: telnet 使用ctrl+']'轉換到命令模式, 然后鍵入quit, 退出
#) flume-ng agent輸出驗證
3.2). 日志提取到HDFS(可以假設web訪問日志, 自動進入HDFS)
#) 在hdfs上創建輸出目錄
sudo -u hdfs hdfs dfs -mkdir -p /flume/test/data
#) 配置 flume-demo-hdfs.conf
agent1.sources = source1 agent1.sinks = sink1 agent1.channels = channel1 agent1.sources.source1.type = exec agent1.sources.source1.command = tail -f /var/log/1.log agent1.sinks.sink1.type = hdfs agent1.sinks.sink1.hdfs.path = hdfs://<namenode_server_ip>:8020/flume/test/data agent1.sinks.sink1.hdfs.filePrefix = events- agent1.sinks.sink1.hdfs.fileType = DataStream agent1.sinks.sink1.hdfs.writeFormat = Text agent1.sinks.sink1.hdfs.roundUnit = minute agent1.channels.channel1.type = memory agent1.channels.channel1.capacity = 100 agent1.channels.channel1.transactionCapacity = 100 agent1.sources.source1.channels = channel1 agent1.sinks.sink1.channel = channel1
#) 啟動flume-ng agent
sudo -u hdfs flume-ng agent -c conf -f conf/flume-demo-hdfs.conf --name agent1 -Dflume.root.logger=INFO,Console
#) 追加日志內容
#) 數據驗證
詳細的source/channel/sink可參考官網文檔:
http://flume.apache.org/FlumeUserGuide.html
總結:
是不是很簡單? 讓我們深入研究源碼 Continue!
參考鏈接:
http://blog.chinaunix.net/uid-790245-id-3869211.html
http://www.kankanews.com/ICkengine/archives/130646.shtml
http://www.ibm.com/developerworks/cn/data/library/bd-1309biginsightsflume/index.html
http://blog.csdn.net/cnbird2008/article/details/18967449