flume 使用遇到問題及解決

1. ../flume/fchannel/spool/data/ 目錄下發生緩存文件積壓

  可能原因：同一時間同一客戶端下向兩個監控目錄mv文件；或同一時間多個客戶端向服務端上傳文件

2.清空../flume/fchannel/spool/data/ 目錄下文件重啟后，監控目錄下的文件發生積壓，沒有上傳

  flume.log里面重復一個異常：

java.lang.IllegalStateException: Channel closed [channel=c1]. Due to java.lang.NullPointerException: null
        at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:352)
        at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:181)
        at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:188)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
        at org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:1028)
        at org.apache.flume.channel.file.Log.replay(Log.java:472)
        at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        ... 9 more

  原因是：清空../flume/fchannel/spool/data/ 目錄下文件的同時也需要清空../flume/fchannel/spool/checkpoint/ 目錄下的文件

另，有幾個關於flume的問題：

1. flume支持監控目錄有子目錄的情況，但是同時向兩個或多個子目錄mv文件的時候，對文件大小有沒有限制？

2. flume支持同時多少客戶端向同一監控目錄下mv文件呢？

3. flume客戶端的/opt/soft/flume/fchannel/spool/data目錄下的文件的作用是？

4. 怎么有效驗證flume客戶端上傳文件是否成功呢？