【原創】大叔問題定位分享(25)ambari metrics collector內置standalone hbase啟動失敗


ambari metrics collector內置hbase目錄位於

/usr/lib/ams-hbase

配置位於

/etc/ams-hbase/conf

通過ruby啟動

/usr/lib/ams-hbase/bin/hirb.rb

實際的啟動命令為

/usr/lib/ams-hbase/bin/hbase-daemon.sh --config /etc/ams-hbase/conf foreground_start master

但是啟動一段時間報錯:

java.lang.RuntimeException: Master not initialized after 200000ms
at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229)
at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:197)
at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3100)

查看日志發現第一個錯:

2019-01-16 16:20:29,295 WARN [StoreOpener-1588230740-1] util.CommonFSUtils: FileSystem doesn't support setStoragePolicy; HDFS-6584, HDFS-9345 not available. This is normal and expected on earlier Hadoop versions.
java.lang.NoSuchMethodException: org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy(org.apache.hadoop.fs.Path, java.lang.String)
at java.lang.Class.getDeclaredMethod(Class.java:2130)
at org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:533)
at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:514)
at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:482)
at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:193)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:258)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5571)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1025)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1022)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

這個錯可以忽略,然后是第二個錯:

2019-01-17 15:49:08,730 ERROR [Thread-24] master.HMaster: Failed to become active master
java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1082)
at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:423)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:714)
at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1398)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:857)
at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2225)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568)
at java.lang.Thread.run(Thread.java:748)

這個可以通過設置hbase.unsafe.stream.capability.enforce=false解決,詳見 https://www.cnblogs.com/barneywill/p/10283076.html

最后發現很多warn:

 

2019-01-17 18:22:12,657 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:13,660 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:15,660 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:19,661 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:27,661 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
2019-01-17 18:22:43,662 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.

清空hbase.rootdir,hbase.tmp.dir,hbase.zookeeper.property.dataDir這3個目錄,同時設置好用戶權限,然后重啟hbase即可;

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM