hadoop執行hbase插入表操作,出錯:Stack trace: ExitCodeException exitCode=1:(xjl456852原創)

在執行hbase和mapreduce融合時,將hdfs上的文本文件插入到hbase中,我沒有使用"胖包"(胖包就是將項目依賴的jar包放入項目打包后的lib目錄中),而是直接將hbase的lib加入到了hadoop的classpath中.

我是將HBase的jar包加到/opt/modules/hadoop/etc/hadoop/hadoop-env.sh中。配置好這個文件,分發到各個節點,改這個配置不用重啟集群.

TEMP=`ls /opt/modules/hbase/lib/*.jar`
HBASE_JARS=`echo $TEMP | sed 's/ /:/g'`
HADOOP_CLASSPATH=$HBASE_JARS

可以通過命令查看hadoop的classpath現在包含哪些jar包:

[hadoop@master ~]$ hdfs classpath

　　

運行hadoop的mapreduce程序

hadoop jar xx.jar input '表名'

 

執行后出錯:

16/08/11 08:52:05 INFO mapreduce.Job: Running job: job_1470930593079_0001
16/08/11 08:52:24 INFO mapreduce.Job: Job job_1470930593079_0001 running in uber mode : false
16/08/11 08:52:24 INFO mapreduce.Job:  map 0% reduce 0%
16/08/11 08:52:24 INFO mapreduce.Job: Job job_1470930593079_0001 failed with state FAILED due to: Application application_1470930593079_0001 failed 2 times due to AM Container for appattempt_1470930593079_0001_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://master.xjl456852.com:8088/cluster/app/application_1470930593079_0001Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1470930593079_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
	at org.apache.hadoop.util.Shell.run(Shell.java:456)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
16/08/11 08:52:24 INFO mapreduce.Job: Counters: 0

　　

 

網上查詢到需要在/opt/modules/hadoop/etc/hadoop/yarn-site.xml中加入mapreduce運行時需要的類庫,需要設置 yarn.application.classpath :

所以我在yarn-site.xml中加入了如下配置,並加入了hbase的lib目錄,配置好這個文件,分發到各個節點, 這個配置需要重啟集群

<property>
  <name>yarn.application.classpath</name>
      <value>
           /opt/modules/hadoop/etc/*,
           /opt/modules/hadoop/etc/hadoop/*,
           /opt/modules/hadoop/lib/*,
           /opt/modules/hadoop/share/hadoop/common/*,
           /opt/modules/hadoop/share/hadoop/common/lib/*,
           /opt/modules/hadoop/share/hadoop/mapreduce/*,
           /opt/modules/hadoop/share/hadoop/mapreduce/lib/*,
           /opt/modules/hadoop/share/hadoop/hdfs/*,
           /opt/modules/hadoop/share/hadoop/hdfs/lib/*,
           /opt/modules/hadoop/share/hadoop/yarn/*,
           /opt/modules/hadoop/share/hadoop/yarn/lib/*,
		/opt/modules/hbase/lib/*
      </value>
</property>

 

參考: http://blog.chinaunix.net/uid-20776139-id-5586691.html