這兩天執行個wordcount或pi總是卡住或報錯failed,查找了多種原因,明確了跑個job時一些參數的設置;
Job Finished in 16.602 seconds
java.io.FileNotFoundException: File does not exist: hdfs://dell:8020/user/root/QuasiMonteCarlo_1482373118226_505074045/out/reduce-out
然后我在nodemanager日志中看到:
container [pid=26086,containerID=container_1482373104195_0001_02_000001] is running beyond virtual memory limits. Current usage: 161.4 MB of 200 MB physical memory used; 879.8 MB of 420.0 MB virtual memory used. Killing container.
發現虛擬內存溢出了,所以查閱了相關資料;
一、
yarn.nodemanager.vmem-pmem-ratio: 物理內存 與 虛擬內存的比率,每用1M物理內存,默認使用2.1M虛擬內存,(建議調大);
或是將 yarn.nodemanager.vmem-check-enabled 虛擬內存的檢查false掉,這一點很重要!