【原創】大叔經驗分享（46）用戶提交任務到yarn報錯

用戶提交任務到yarn時有可能遇到下面的錯誤：

1）

Requested user anything is not whitelisted and has id 980,which is below the minimum allowed 1000

這是因為yarn中配置min.user.id=1000，yarn認為id小於1000的是超級用戶，yarn禁止超級用戶提交任務；

Each account must have a user ID that is greater than or equal to 1000. In the /etc/hadoop/conf/taskcontroller.cfg file, the default setting for the banned.users property is mapred, hdfs, and bin to prevent jobs from being submitted via those user accounts. The default setting for the min.user.id property is 1000 to prevent jobs from being submitted with a user ID less than 1000, which are conventionally Unix super users.

改為更小即可

然后重啟yarn

 

2）

Requested user hdfs is banned

這是因為yarn中配置banned.users=hdfs,yarn,mapred,bin，會阻止這幾個用戶提交任務

the default setting for the banned.users property is hdfs, yarn, mapred, and bin to prevent jobs from being submitted via those user accounts.

 

3）其他組件比如oozie無法獲得提交到yarn上的application的狀態

2019-04-01 11:22:23,528 WARN org.apache.oozie.command.wf.ActionCheckXCommand: SERVER[cdp-test-server-04.bj] USER[impala] GROUP[-] TOKEN[] APP[test_sh_wf] JOB[0000000-190330171215585-oozie-oozi-W] ACTION[0000000-190330171215585-oozie-oozi-W@test_sh_action] Exception while executing check(). Error Code [JA017], Message[JA017: Could not lookup launched hadoop Job ID [job_1553937093273_0007] which was associated with  action [0000000-190330171215585-oozie-oozi-W@test_sh_action].  Failing this action!]
org.apache.oozie.action.ActionExecutorException: JA017: Could not lookup launched hadoop Job ID [job_1553937093273_0007] which was associated with  action [0000000-190330171215585-oozie-oozi-W@test_sh_action].  Failing this action!
        at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1507)
        at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:182)
        at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
        at org.apache.oozie.command.XCommand.call(XCommand.java:286)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

此時請檢查jobhistory server，如果jobhistory server頁面中沒有application，也可以通過curl查看：

# curl http:/localhost:19888/jobhistory -v
...
            <script type="text/javascript">
              var jobsTableData=[
]
            </script>
...

正常來說這個數組應該存放很多application，這個問題需要檢查hdfs上/user/history及子目錄是否存在以及權限

$ hdfs dfs -mkdir /user/history
$ hdfs dfs -chmod 777 /user/history

正常情況下jobhistory會從hdfs以下目錄加載應用信息

/user/history/done/$year/$month/$day/
/user/history/done_intermediate/$user/

 

4）從jobhistory server查看application時報錯

2019-04-01 16:04:16,115 WARN org.apache.oozie.command.wf.ActionCheckXCommand: SERVER[cdp-test-server-04.bj] USER[impala] GROUP[-] TOKEN[] APP[test_sh_wf] JOB[0000008-190330171215585-oozie-oozi-W] ACTION[0000008-190330171215585-oozie-oozi-W@test_sh_action] Exception while executing check(). Error Code [JA009], Message[JA009: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not load history file hdfs://hdfs:8020/user/history/done_intermediate/impala/job_1553937093273_0018-1554105820398-impala-oozie%3Alauncher%3AT%3Dshell%3AW%3Dtest_sh_wf%3AA%3Dtest_sh_acti-1554105833651-1-0-SUCCEEDED-root.users.impala-1554105825680.jhist
...
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=READ, inode="/user/history/done_intermediate/impala/job_1553937093273_0018-1554105820398-impala-oozie%3Alauncher%3AT%3Dshell%3AW%3Dtest_sh_wf%3AA%3Dtest_sh_acti-1554105833651-1-0-SUCCEEDED-root.users.impala-1554105825680.jhist":impala:supergroup:-rwxrwx---

hdfs中supergroup配置如下：

dfs.permissions.supergroup=supergroup
dfs.permissions.superusergroup=supergroup

解決方法：

# groupadd supergroup
# usermod -a -G supergroup mapred
# usermod -a -G supergroup hdfs
# usermod -a -G supergroup yarn

 

參考：https://community.hortonworks.com/questions/23914/requested-user-hive-is-not-whitelisted-and-has-id.html