【Hive】Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown

前言

  將Hive的默認引擎Map-Reduce改為Tez后，經常出現TezSession has already shutdown錯誤；

錯誤內容

Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1589331609996_0001 failed 2 times due to AM Container for appattempt_1589331609996_0001_000002 exited with  exitCode: -103

For more detailed output, check application tracking page:http://hadoop112:8088/cluster/app/application_1589331609996_0001Then, click on links to logs of each attempt.
Diagnostics: Container [pid=3940,containerID=container_1589331609996_0001_02_000001] is running beyond virtual memory limits. Current usage: 155.1 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1589331609996_0001_02_000001 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 3940 3939 3940 3940 (bash) 0 0 115838976 359 /bin/bash -c /opt/module/jdk1.8.0_144/bin/java  -Xmx819m -Djava.io.tmpdir=/opt/module/hadoop-2.7.2/data/tmp/nm-local-dir/usercache/ssrs/appcache/application_1589331609996_0001/container_1589331609996_0001_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-2.7.2/logs/userlogs/application_1589331609996_0001/container_1589331609996_0001_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/opt/module/hadoop-2.7.2/logs/userlogs/application_1589331609996_0001/container_1589331609996_0001_02_000001/stdout 2>/opt/module/hadoop-2.7.2/logs/userlogs/application_1589331609996_0001/container_1589331609996_0001_02_000001/stderr  
	|- 3995 3940 3940 3940 (java) 168 124 2657517568 39346 /opt/module/jdk1.8.0_144/bin/java -Xmx819m -Djava.io.tmpdir=/opt/module/hadoop-2.7.2/data/tmp/nm-local-dir/usercache/ssrs/appcache/application_1589331609996_0001/container_1589331609996_0001_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-2.7.2/logs/userlogs/application_1589331609996_0001/container_1589331609996_0001_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel= org.apache.tez.dag.app.DAGAppMaster --session 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1589331609996_0001 failed 2 times due to AM Container for appattempt_1589331609996_0001_000002 exited with  exitCode: -103

問題原因

​ 該錯誤是YARN的虛擬內存計算方式導致，上例中用戶程序申請的內存為1Gb，YARN根據此值乘以一個比例（默認為2.1）得出申請的虛擬內存的值，當YARN計算的用戶程序所需虛擬內存值大於計算出來的值時，就會殺死該進程，然后報出以上錯誤。

解決方式

第一種方式：

1.關閉yarn虛擬內存檢查，需要在hadoop目錄下的yarn-site.xml添加如下配置。

[ssrs@hadoop111 hadoop]$ pwd
/opt/module/hadoop-2.7.2/etc/hadoop
[ssrs@hadoop111 hadoop]$ vi yarn-site.xml

<!-- 關閉虛擬內存檢查 -->
<property>
	<name>yarn.nodemanager.vmem-check-enabled</name>
	<value>false</value>
</property>

2.將該配置文件發送到集群中的其他服務器，重啟集群；

第二種方式：

1.增加yarn虛擬內存的上限；需要在hadoop目錄下的yarn-site.xml添加如下配置。

[ssrs@hadoop111 hadoop]$ pwd
/opt/module/hadoop-2.7.2/etc/hadoop
[ssrs@hadoop111 hadoop]$ vi yarn-site.xml

<!-- 增加最小內存上限 -->
<property>
	<name>yarn.scheduler.minimum-allocation-mb</name>
    <!-- 默認是1024 可以根據自己的服務器內存配置調整合適的數字 -->
	<value>2048</value>
</property>

<!-- 增大虛擬內存的比率 -->
<property>
	<name>yarn.nodemanager.vmem-pmem-ratio</name>
    <!-- 默認是2.1 適當增大該值 -->
	<value>4</value>
</property>

2.將該配置文件發送到集群中的其他服務器，重啟集群；

注意事項：

第一種方式是一勞永逸的方式，方便快捷；

第二種方式可能根據運行任務的不同，進行多次調整，最后調整到集群合適的一個數值；

無論是方案一還是方案二，都一定記得將yarn-site.xml文件同步到集群中所有服務器！