flume:spooldir采集日志,kafka輸出的配置問題


flume配置:

#DBFile
DBFile.sources = sources1  
DBFile.sinks = sinks1  
DBFile.channels = channels1  
  
# DBFile-DB-Source 
DBFile.sources.sources1.type = spooldir
DBFile.sources.sources1.spoolDir =/var/log/apache/flumeSpool//db
DBFile.sources.sources1.inputCharset=utf-8
  
# DBFile-Sink  
DBFile.sinks.sinks1.type = org.apache.flume.sink.kafka.KafkaSink  
DBFile.sinks.sinks1.topic = DBFile
DBFile.sinks.sinks1.brokerList = hdp01:6667,hdp02:6667,hdp07:6667
DBFile.sinks.sinks1.requiredAcks = 1  
DBFile.sinks.sinks1.batchSize = 2000  
  
# DBFile-Channel
DBFile.channels.channels1.type = memory
DBFile.channels.channels1.capacity = 10000
DBFile.channels.channels1.transactionCapacity = 1000

# DBFile-Source And Sink to the channel
DBFile.sources.sources1.channels = channels1
DBFile.sinks.sinks1.channel = channels1

 

       故障現象:第一次上傳文件時,flume能很快處理文件,后面上傳還是顯示文件未處理。如果重啟flume服務,又能立刻處理。

經測試,問題的原因在這個配置上:DBFile.sinks.sinks1.requiredAcks = -1

      requiredAcks 的官方解釋:How many replicas must acknowledge a message before its considered successfully written. Accepted values are 0 (Never wait for acknowledgement),

1 (wait for leader only), -1 (wait for all replicas) Set this to -1 to avoid data loss in some cases of leader failure.

    把這個值改為1就好了。


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM