問題描述:
在hadoop中運行應用,出現了running beyond virtual memory錯誤。提示如下:
Container [pid=28920,containerID=container_1389136889967_0001_01_000121] is running beyond virtual memory limits. Current usage: 1.2 GB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
原因:從機上運行的Container試圖使用過多的內存,而被
NodeManager kill掉了。
[摘錄] The NodeManager is killing your container. It sounds like you are trying to use hadoop streaming which is running as a child process of the map-reduce task. The NodeManager monitors the entire process tree of the task and if it eats up more memory than the maximum set in mapreduce.map.memory.mb or mapreduce.reduce.memory.mb respectively, we would expect the Nodemanager to kill the task, otherwise your task is stealing memory belonging to other containers, which you don't want.
解決方法:
mapred-site.xml中設置map和reduce任務的內存配置如下:(value中實際配置的內存需要根據自己機器內存大小及應用情況進行修改)
<property>
<name>mapreduce.map.memory.mb</name>
<value>1536</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024M</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2560M</value>
</property>
附錄:
Container is running beyond memory limits
http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits