碼上快樂

相關內容    簡體    繁體

spark跑YARN模式或Client模式提交任務不成功（application state: ACCEPTED）

本文轉載自   查看原文   2017-04-09 09:58   11770    Spark集群搭建（1、3和5節點）（含CentOS系統和Ubuntu系統）

 

 

 　　不多說，直接上干貨！

 

 

　　問題詳情

　 電腦8G，目前搭建3節點的spark集群，采用YARN模式。

    master分配2G，slave1分配1G,slave2分配1G。（在安裝虛擬機時）

    export SPARK_WORKER_MERMORY=1g  （在spark-env.sh）

export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60         （必須寫）
export SCALA_HOME=/usr/local/scala/scala-2.10.5    （必須寫）
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0    （必須寫）
export HADOOP_CONF_DIR=/usr/local/hadoop/hadoop-2.6.0/etc/hadoop   （必須寫）
export SPARK_MASTER_IP=192.168.80.10　　
export SPARK_WORKER_MERMORY=1G     （官網上說，至少1g）

 

 

　其實這個問題解決辦法很簡單，就是3個節點的是，你的內存最好還是大點。如master分配4G，slave1分配2G,slave2分配2G！！！！（盡可能的大）

 當然，很多博友們，跟我的情況是一樣的，在學校階段，電腦只有8g已經是最大電腦內存限制了。

 

 

 

 一般是由於有多個用戶同時向集群提交任務或一個用戶向集群同時提交了多個任務導致Yarn資源的分配錯誤。解決這個問題，只需要更改Hadoop的配置文件：/etc/hadoop/conf/capacity-scheduler.xml，把選項：yarn.scheduler.capacity.maximum-am-resource-percent從0.1改成0.5。顧名思義，這個選項是增加Yarn可調度的資源量，當然也可以視具體情況增加更多。也可見，默認情況下，Yarn沒有將很多資源分配給任務的能力。

 

 

 

 

 

 

具體，見

Spark on YARN模式的安裝（spark-1.6.1-bin-hadoop2.6.tgz + hadoop-2.6.0.tar.gz）（master、slave1和slave2）（博主推薦）

 

 

[spark@master logs]$  $SPARK_HOME/bin/spark-submit  \
> --class org.apache.spark.examples.JavaSparkPi \
> --master yarn-cluster \
> --num-executors 1 \
> --driver-memory 512m \
> --executor-memory 512m \
> --executor-cores 1 \
>  /usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar





注意：
driver-memory不指定也可以，默認使用512M
executor-memory不指定的化， 默認是1G
 
  


17/04/09 17:03:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/09 17:03:55 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.80.10:8032
17/04/09 17:03:56 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers
17/04/09 17:03:56 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/04/09 17:03:56 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/04/09 17:03:56 INFO yarn.Client: Setting up container launch context for our AM
17/04/09 17:03:56 INFO yarn.Client: Setting up the launch environment for our AM container
17/04/09 17:03:56 INFO yarn.Client: Preparing resources for our AM container
17/04/09 17:03:59 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491728358337_0001/spark-assembly-1.6.1-hadoop2.6.0.jar
17/04/09 17:04:19 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491728358337_0001/spark-examples-1.6.1-hadoop2.6.0.jar
17/04/09 17:04:49 INFO yarn.Client: Uploading resource file:/tmp/spark-d152ed1b-09ca-47c8-8457-58f7e52ff419/__spark_conf__6499474209714260387.zip -> hdfs://master:9000/user/spark/.sparkStaging/application_1491728358337_0001/__spark_conf__6499474209714260387.zip
17/04/09 17:04:50 INFO spark.SecurityManager: Changing view acls to: spark
17/04/09 17:04:50 INFO spark.SecurityManager: Changing modify acls to: spark
17/04/09 17:04:50 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
17/04/09 17:04:50 INFO yarn.Client: Submitting application 1 to ResourceManager
17/04/09 17:04:51 INFO impl.YarnClientImpl: Submitted application application_1491728358337_0001
17/04/09 17:04:52 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:04:52 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1491728691236
     final status: UNDEFINED
     tracking URL: http://master:8088/proxy/application_1491728358337_0001/
     user: spark
17/04/09 17:04:53 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:04:54 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:04:55 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:04:56 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:04:57 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:04:58 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:04:59 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:05:00 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:05:01 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:05:02 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:05:03 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:05:04 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:05:05 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:05:06 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
17/04/09 17:05:07 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)
^C17/04/09 17:05:08 INFO util.ShutdownHookManager: Shutdown hook called
17/04/09 17:05:08 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-d152ed1b-09ca-47c8-8457-58f7e52ff419

.........
然后最后失敗

 

 

 

 

 　問題分析

　　　　其實，這一情況，是我剛開始啟動了太多的Application導致累積，使得內存超出了導致！

 

 

 

　解決方法

　　　　按照辦法1，辦法2，辦法3...的順序來

 

 

 

 

 辦法1:（一般能解決）

　　先，

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/stop-all.sh

 

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/stop-all.sh

 

 

 

　　再，

 

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/start-all.sh

 

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/start-all.sh

 

 

 

 　　在spark-env.sh，至少得

export SPARK_WORKER_MERMORY=1G

 

 

　　改成

 

 

 

[spark@master spark-1.6.1-bin-hadoop2.6]$ $SPARK_HOME/bin/spark-submit \
--class org.apache.spark.examples.JavaSparkPi \
--master yarn-cluster \
--num-executors 1 \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1 \
$SPARK_HOME/lib/spark-examples-1.6.1-hadoop2.6.0.jar



[spark@master spark-1.6.1-bin-hadoop2.6]$ $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi --master yarn-cluster --num-executors 1 --driver-memory 1g --executor-memory 1g --executor-cores 1 $SPARK_HOME/lib/spark-examples-1.6.1-hadoop2.6.0.jar
17/04/09 20:01:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/09 20:01:27 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.80.10:8032
17/04/09 20:01:31 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers
17/04/09 20:01:32 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/04/09 20:01:32 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
17/04/09 20:01:32 INFO yarn.Client: Setting up container launch context for our AM
17/04/09 20:01:32 INFO yarn.Client: Setting up the launch environment for our AM container
17/04/09 20:01:32 INFO yarn.Client: Preparing resources for our AM container
17/04/09 20:01:37 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491735225663_0006/spark-assembly-1.6.1-hadoop2.6.0.jar
17/04/09 20:01:57 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491735225663_0006/spark-examples-1.6.1-hadoop2.6.0.jar
17/04/09 20:02:18 INFO yarn.Client: Uploading resource file:/tmp/spark-8d8662b1-6410-4b72-9751-57bb0e2785d0/__spark_conf__5191259430263541003.zip -> hdfs://master:9000/user/spark/.sparkStaging/application_1491735225663_0006/__spark_conf__5191259430263541003.zip
17/04/09 20:02:18 INFO spark.SecurityManager: Changing view acls to: spark
17/04/09 20:02:19 INFO spark.SecurityManager: Changing modify acls to: spark
17/04/09 20:02:19 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
17/04/09 20:02:22 INFO yarn.Client: Submitting application 6 to ResourceManager
17/04/09 20:02:23 INFO impl.YarnClientImpl: Submitted application application_1491735225663_0006
17/04/09 20:02:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:25 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1491739342315
     final status: UNDEFINED
     tracking URL: http://master:8088/proxy/application_1491735225663_0006/
     user: spark
17/04/09 20:02:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:35 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:37 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:42 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:48 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:49 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:50 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:51 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:53 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:54 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:55 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:56 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)

17/04/09 20:03:05 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:06 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:07 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:08 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:09 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:10 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:11 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:12 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:13 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:14 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:15 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:16 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:17 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:18 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:19 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:20 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:21 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:23 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:24 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:27 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:28 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:29 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:30 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:31 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:33 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:34 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:40 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:41 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:42 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:48 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:50 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:51 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:52 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:53 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:54 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:55 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:56 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:03 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:05 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:06 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:07 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:08 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:09 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:10 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:11 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:12 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:13 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:15 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:16 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:17 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:18 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:19 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:20 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:21 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:22 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:23 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:24 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:27 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:28 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:29 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:30 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:31 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:33 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:34 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:35 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:37 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:40 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:41 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:48 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:49 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:50 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:51 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:52 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:53 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING) 只要最后出現這個了，就說明成功了！
17/04/09 20:04:53 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.80.11
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1491739342315
     final status: UNDEFINED
     tracking URL: http://master:8088/proxy/application_1491735225663_0006/
     user: spark
17/04/09 20:04:54 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:55 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:56 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:05 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:06 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:07 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:08 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:09 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:10 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:11 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:12 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:13 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:14 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:15 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:16 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:17 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:19 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:20 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:21 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:22 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:23 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:27 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:28 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:29 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:30 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:31 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:33 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:34 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:35 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:37 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:40 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:41 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:42 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
....
17/04/09 20:06:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:06:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:06:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:03 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: FINISHED)
17/04/09 20:07:04 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.80.11
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1491739342315
     final status: SUCCEEDED
     tracking URL: http://master:8088/proxy/application_1491735225663_0006/A
     user: spark
17/04/09 20:07:04 INFO yarn.Client: Deleting staging directory .sparkStaging/application_1491735225663_0006
17/04/09 20:07:07 INFO util.ShutdownHookManager: Shutdown hook called
17/04/09 20:07:07 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-8d8662b1-6410-4b72-9751-57bb0e2785d0
[spark@master spark-1.6.1-bin-hadoop2.6]$

 

 

 

 　　如果之后，再次去執行別的jar包，出現又是無法運行了，則

spark on yarn模式里需要有時手工釋放linux內存。

 

 

 

 

 

 

 解決辦法

方法2： 首先想到是集群中內存資源不足，可以檢查下每台機器是否有足夠剩余內存( free -g)；也可能是其他已經提交的Spark應用占了大部分資源；

[spark@master spark-1.6.1-bin-hadoop2.6]$ free -g
             total       used       free     shared    buffers     cached
Mem:             1          1          0          0          0          0
-/+ buffers/cache:          0          0
Swap:            1          0          1
[spark@master spark-1.6.1-bin-hadoop2.6]$

 　　那么，就要去殺死之前那些還正在運行（多余沒用）的Application。

　　怎么做？

　　進入你的$HADOOP_HOME/bin下

 

 

 　　執行命令

[spark@master bin]$ yarn application -list

　　然后可以看到應用的信息

 

　　如需殺死當前某個作業，使用kill application-id的命令如下

 

yarn application -kill application_1437456051228_1725

 

 

 

 

 

 

 

 

 

方法3： 如果1>正常，我們可以看看YARN集群是否啟動成功。注意“坑”可能就在這里： 即使Slave上的nodemanager進程存在，要注意檢查resource manager日志，看看各個node manager是否啟動成功，有些人的問題就出現在這里：進程在，但是日志顯示node manager狀態為UNHEALTHY，所以YARN集群能識別到的總內存資源為0。檢查了UNHEALTHY的原因，是因為/tmp下一個目錄被識別為bad, 因為是臨時目錄，把每個node manager的對應目錄刪掉，然后重啟YARN集群，最終問題解決。

 

  ResourceManager日志存放位置是${HADOOP_HOME}/logs/yarn-*-resourcemanager-*.log       

 

 

 

 

NodeManager日志存放位置是各個NodeManager節點上${HADOOP_HOME}/logs/yarn-*-nodemanager-*.log

 

 

 

 

 

　　先，

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/stop-all.sh

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/stop-all.sh

 

　　再，

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/start-all.sh

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/start-all.sh

 

 

 

方法4：如果還是有問題的話，只能去日志文件中查看出錯的異常，查看更多一些的報錯信息。

 　　　　　　但是，方法1和方法2是足夠可以解決該問題的!

 

 

方法5：

　　　　除非你的內存足夠，可以擴大內存。我的只是自己電腦最大才8G

 

 

 

方法6：

　　在yarn-site.xml下

<property>
          <name>yarn.nodemanager.pmem-check-enabled</name>
          <value>false</value>
</property>
<property>
          <name>yarn.nodemanager.vmem-check-enabled</name>
          <value>false</value>
</property>

 　　具體見

http://dongxicheng.org/mapreduce-nextgen/hadoop-yarn-memory-cpu-scheduling/

【YARN中內存資源的調度和隔離】

【YARN中CPU資源的調度和隔離】

 

 

 

 

 

 

 

 

方法7：

運行hadoop的時候提示物理內存或虛擬內存溢出的解決方案running beyond physical memory或者beyond vitual memory limits

 

 

 

 

 

 

方法8

　　　　　　最壞的打算，就是之前有快照，回到初始位置吧！重新安裝，哎

 

 

 

 

 

 

 

 

 

 

歡迎大家，加入我的微信公眾號：大數據躺過的坑        人工智能躺過的坑

 

 

 

 

 

 

 

同時，大家可以關注我的個人博客：

   http://www.cnblogs.com/zlslch/   和     http://www.cnblogs.com/lchzls/      http://www.cnblogs.com/sunnyDream/   

   詳情請見：http://www.cnblogs.com/zlslch/p/7473861.html

 

　　人生苦短，我願分享。本公眾號將秉持活到老學到老學習無休止的交流分享開源精神，匯聚於互聯網和個人學習工作的精華干貨知識，一切來於互聯網，反饋回互聯網。
　　目前研究領域：大數據、機器學習、深度學習、人工智能、數據挖掘、數據分析。 語言涉及：Java、Scala、Python、Shell、Linux等 。同時還涉及平常所使用的手機、電腦和互聯網上的使用技巧、問題和實用軟件。 只要你一直關注和呆在群里，每天必須有收獲

 

      對應本平台的討論和答疑QQ群：大數據和人工智能躺過的坑（總群）（161156071） 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 Spark通過YARN提交任務不成功（包含YARN cluster和YARN client) 以yarn-client方式提交spark任務，任務一直卡在ACCEPTED狀態 Spark2.x（五十九）：yarn-cluster模式提交Spark任務，如何關閉client進程? spark利用yarn提交任務報:YARN application has exited unexpectedly with state UNDEFINED spark yarn cluster模式下任務提交和計算流程分析 Spark- Spark Yarn模式下跑yarn-client無法初始化SparkConext,Over usage of virtual memory spark on yarn :state: ACCEPTED一直 出現 理解Spark運行模式（一）(Yarn Client) Spark 在yarn上運行模式詳解：cluster模式和client模式 Spark 集群 任務提交模式
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM