现象:表写入异常,通过批量方式写入,数据高峰期,几乎每次都提交失败。
原因:hbase抛出异常的根本原因待探索
处理方式:从业务角度规避了此问题,将出问题的表数据量(较大大?)备份,重建该表。
相关脚本:
disable 't1'
snapshot 't1', 't1_bak'
clone_snapshot 't1_bak', 't1_bak_new'
delete_snapshot 't1_bak'
drop 't1'
异常信息:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1883 actions: IOException: 1882 times, InterruptedIOException: 1 time,
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:247)
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:227)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1758)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:146)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1031)
at com.igen.parse.service.HbaseService.insertHbaseForNew(HbaseService.java:2095)
at com.igen.parse.service.HbaseService$22$1.run(HbaseService.java:1425)
at java.lang.Thread.run(Thread.java:745)
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2261 actions: IOException: 2261 times,
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:247)
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:227)
at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1758)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
at org.apache.hadoop.hbase.client.HTable.close(HTable.java:1466)
at com.igen.parse.service.HbaseService.insertHbaseForNew(HbaseService.java:2146)
at com.igen.parse.service.HbaseService$22$1.run(HbaseService.java:1425)
at java.lang.Thread.run(Thread.java:745)