配置監控
-
修改flume.env.sh
export JAVA_OPTS=”-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false” -
使用以下腳本
flume-ng agent \ --name a1 \ --conf $FLUME_HOME/conf \ --conf-file $FLUME_HOME/conf/exec-memory-hdfs-partition.conf \ -Dflume.root.logger=INFO,console \ -Dflume.monitoring.type=http \ -Dflume.monitoring.port=1234
指標項整理
OS監控項
| objectName | 指標項 | 說明 |
|---|---|---|
| java.lang:type=OperatingSystem | FreePhysicalMemorySize | 空閑物理內存 |
| java.lang:type=OperatingSystem | SystemCpuLoad | 系統CPU利用率 |
| java.lang:type=OperatingSystem | ProcessCpuLoad | 進程CPU利用率 |
| java.lang:type=GarbageCollector,name=PS Scavenge | CollectionCount | GC次數 |
json數據格式
{
"SOURCE.src-1":{
"OpenConnectionCount":"0",
"Type":"SOURCE",
"AppendBatchAcceptedCount":"1355",
"AppendBatchReceivedCount":"1355",
"EventAcceptedCount":"28286",
"AppendReceivedCount":"0",
"StopTime":"0",
"StartTime":"1442566410435",
"EventReceivedCount":"28286",
"AppendAcceptedCount":"0"
},
"CHANNEL.ch-1":{
"EventPutSuccessCount":"28286",
"ChannelFillPercentage":"0.0",
"Type":"CHANNEL",
"StopTime":"0",
"EventPutAttemptCount":"28286",
"ChannelSize":"0",
"StartTime":"1442566410326",
"EventTakeSuccessCount":"28286",
"ChannelCapacity":"1000000",
"EventTakeAttemptCount":"313734329512"
},
"SINK.sink-1":{
"Type":"SINK",
"ConnectionClosedCount":"0",
"EventDrainSuccessCount":"28286",
"KafkaEventSendTimer":"482493",
"BatchCompleteCount":"0",
"ConnectionFailedCount":"0",
"EventDrainAttemptCount":"0",
"ConnectionCreatedCount":"0",
"BatchEmptyCount":"0",
"StopTime":"0",
"RollbackCount":"9",
"StartTime":"1442566411897",
"BatchUnderflowCount":"0"
}
}
指標項說明
source監控項
| objectName (會隨實際情況而變化) |
指標項 | 說明 |
|---|---|---|
| org.apache.flume.source:type=r1 | OpenConnectionCount | 目前與客戶端或sink保持連接的總數量 |
| org.apache.flume.source:type=r1 | AppendBatchAcceptedCount | 成功提交到channel的批次的總數量 |
| org.apache.flume.source:type=r1 | AppendBatchReceivedCount | 接收到事件批次的總數量 |
| org.apache.flume.source:type=r1 | AppendAcceptedCount | 逐條錄入的次數 |
| org.apache.flume.source:type=r1 | AppendReceivedCount | 每批只有一個事件的事件總數量 |
| org.apache.flume.source:type=r1 | EventAcceptedCount | 成功寫出到channel的事件總數量 |
| org.apache.flume.source:type=r1 | EventReceivedCount | 目前為止source已經接收到的事件總數量 |
| org.apache.flume.source:type=r1 | StartTime | source啟動時的毫秒值時間 |
| org.apache.flume.source:type=r1 | StopTime | source停止時的毫秒值時間，為0表示一直在運行 |
channel監控項
| objectName (會隨實際情況而變化) |
指標項 | 說明 |
|---|---|---|
| org.apache.flume.channel:type=c1 | EventPutAttemptCount | Source嘗試寫入Channe的事件總次數 |
| org.apache.flume.channel:type=c1 | EventPutSuccessCount | 成功寫入channel且提交的事件總次數 |
| org.apache.flume.channel:type=c1 | EventTakeAttemptCount | sink嘗試從channel拉取事件的總次數。 |
| org.apache.flume.channel:type=c1 | EventTakeSuccessCount | sink成功從channel讀取事件的總數量 |
| org.apache.flume.channel:type=c1 | ChannelSize | 目前channel中事件的總數量 |
| org.apache.flume.channel:type=c1 | ChannelCapacity | channel的容量 |
| org.apache.flume.channel:type=c1 | ChannelFillPercentage | channel已填入的百分比 |
| org.apache.flume.channel:type=c1 | StartTime | channel啟動時的毫秒值時間 |
| org.apache.flume.channel:type=c1 | StopTime | channel停止時的毫秒值時間，為0表示一直在運行 |
sink監控項
| objectName (會隨實際情況而變化) |
指標項 | 說明 |
|---|---|---|
| org.apache.flume.sink:type=k1 | ConnectionCreatedCount | 創建的連接數量 |
| org.apache.flume.sink:type=k1 | ConnectionClosedCount | 關閉的連接數量 |
| org.apache.flume.sink:type=k1 | ConnectionFailedCount | 由於錯誤關閉的連接數量 |
| org.apache.flume.sink:type=k1 | BatchEmptyCount | 批量處理event的個數為0的數量-表示source寫入數據的速度比sink處理數據的速度慢 |
| org.apache.flume.sink:type=k1 | BatchUnderflowCount | 批量處理event的個數小於批處理大小的數量 |
| org.apache.flume.sink:type=k1 | BatchCompleteCount | 批量處理event的個數等於批處理大小的數量 |
| org.apache.flume.sink:type=k1 | EventDrainAttemptCount | sink嘗試寫出到存儲的事件總數量 |
| org.apache.flume.sink:type=k1 | EventDrainSuccessCount | sink成功寫出到存儲的事件總數量 |
| org.apache.flume.sink:type=k1 | StartTime | channel啟動時的毫秒值時間 |
| org.apache.flume.sink:type=k1 | StopTime | channel停止時的毫秒值時間，為0表示一直在運行 |