解決Hue/hiveserver2報錯：java.io.IOException: Job status not available

sql是：select count(distinct col) from db.table;

排查過程中遇到過幾個不同的報錯：

1. beeline -u jdbc:hive2://0.0.0.0:10000 -e "select count(distinct col) from db.table;"

INFO  : Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1494385775332_0822
ERROR : Ended Job = job_1494385775332_0822 with exception 'java.io.IOException(Job status not available )'
java.io.IOException: Job status not available 
        at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:333)
        at org.apache.hadoop.mapreduce.Job.getJobState(Job.java:358)
        at org.apache.hadoop.mapred.JobClient$NetworkedJob.getJobState(JobClient.java:295)
        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:244)
        at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:437)
        at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
....
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)

Diagnostics:	Staging dir does not exist /mr-history/anonymous/.staging

2. beeline -u jdbc:hive2://0.0.0.0:10000 -n userxxx -e "select count(distinct col) from db.table;"

ERROR : Ended Job = job_1494385775332_0823 with exception 'java.io.IOException(java.io.IOException: Unknown Job job_1494385775332_0823
        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:218)
        at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:281)
        at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)
        at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
)'
java.io.IOException: java.io.IOException: Unknown Job job_1494385775332_0823

Diagnostics:	Job init failed : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://xxx/mr-history/userxxx/.staging/job_1494385775332_0823/job.splitmetainfo

使用不同的用戶執行還是報一樣的錯誤。

3. hive -e "select count(distinct col) from db.table;" 卻能執行成功

 

解決：

根據第二步Diagnostics的報錯信息搜索到一篇文章：https://community.hortonworks.com/questions/17489/job-init-fail-job-splitmetainfo-file-does-not-exis.html

對比了hive和hs2中此屬性的值，果然不一樣.hive上的值和配置文件中的一樣，hs2不一樣，在hs2中執行hql前設置為正確值后問題解決。詢問了運維同事，這個屬性近期剛改，改完后hiveserver2沒有重啟，導致使用beeline客戶端或Hue時報此錯。

解決方法：

1. 臨時性：hql中顯示糾正此屬性值：set yarn.app.mapreduce.am.staging-dir=xxx;

2. 重啟HS2