Flume啟動時報錯Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.解決辦法（圖文詳解）

 

 

　　前期博客

Flume自定義攔截器（Interceptors）或自帶攔截器時的一些經驗技巧總結（圖文詳解）

 

 

 

　　問題詳情

2017-07-29 10:30:12,784 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: SOURCE, name: fileSource. src.events.accepted == 21158
2017-07-29 10:30:12,786 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: SOURCE, name: fileSource. src.events.received == 21158
2017-07-29 10:30:12,786 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: SOURCE, name: fileSource. src.open-connection.count == 0
2017-07-29 10:30:12,959 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:447)] process failed
java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.
    at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(BucketWriter.java:649)
    at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:409)
    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:430)
    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
    at java.lang.Thread.run(Thread.java:745)
2017-07-29 10:30:13,152 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.
    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.     at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(BucketWriter.java:649)
    at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:409)
    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:430)
    ... 3 more

 

 　　這句話，說的很明顯。

　　你的hdfs.calltimeout可能設置得太低或HDFS需要花費太長的時間。

 

 

 　　我的是

#---------hdfsSink 相關配置------------------
agent1.sinks.hdfsSink.type = hdfs
# 注意, 我們輸出到下面一個子文件夾datax中
agent1.sinks.hdfsSink.hdfs.path = hdfs://master:9000/data/types/%Y%m%d/%{log_type}
agent1.sinks.hdfsSink.hdfs.writeFormat = Text
agent1.sinks.hdfsSink.hdfs.fileType = DataStream
agent1.sinks.hdfsSink.hdfs.callTimeout = 3600000
agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true

#當文件大小為52428800字節時，將臨時文件滾動成一個目標文件
agent1.sinks.hdfsSink.hdfs.rollSize = 52428800
#events數據達到該數量的時候，將臨時文件滾動成目標文件
agent1.sinks.hdfsSink.hdfs.rollCount = 0
#每隔N s將臨時文件滾動成一個目標文件
agent1.sinks.hdfsSink.hdfs.rollInterval = 1200

#配置前綴和后綴
agent1.sinks.hdfsSink.hdfs.filePrefix=run
                                                                       

 

 

 　　這個大家，可以還改大些。

agent1.sinks.hdfsSink.hdfs.callTimeout = 7200000

 

 

 hdfs.sinks.fs.hdfs.rollInterval=21600
hdfs.sinks.fs.hdfs.rollSize=8589934592
為何這兩個設置這么大。超時出現問題正常的設置小一點，根據情況自己調試下。
建議初始值：
hdfs.sinks.fs.hdfs.rollInterval=10
hdfs.sinks.fs.hdfs.rollSize=0
依次：
20
40
80
200
等值。