namenode由於jn寫入超時異常


namenode異常信息:

2020-06-03 04:44:42,313 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [1xx:8485, xxx:8485, xxx:8485], stream=QuorumOutputStream starting at txid 47431233))
java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond.
        at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:107)
        at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113)
        at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107)
        at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:533)
        at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
        at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:57)
        at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:529)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:707)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:188)
        at java.lang.Thread.run(Thread.java:748)

 

解決辦法:

在hdfs-site.xml添加如下配置:

<property>
<!-- EditLog會被切割為很多段,每一段稱為一個segment,Namenode發起新寫入editlog的RPC調用,會使用startLogSegment方法,該參數表示發起新segment的超時時間 --> <name>dfs.qjournal.start-segment.timeout.ms</name> <value>90000</value> </property> <property> <name>dfs.qjournal.select-input-streams.timeout.ms</name> <value>90000</value> </property> <property>
<!--寫入超時時間 --> <name>dfs.qjournal.write-txns.timeout.ms</name> <value>90000</value> </property>

在core-site.xml

ipc.client.connect.timeout = 90000

2、關閉zk優先同步日志功能

forceSync=no

 

借鑒:

https://www.cppentry.com/bencandy.php?fid=118&id=195295

https://blog.csdn.net/xjping0794/article/details/77836543


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM