spark - tasks is bigger than spark.driver.maxResultSize

本文轉載自查看原文 2016-12-13 12:02 11489 spark/ 大數據

Error

ERROR TaskSetManager: Total size of serialized results of 8113 tasks (1131.0 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
ERROR TaskSetManager: Total size of serialized results of 8114 tasks (1131.1 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
ERROR TaskSetManager: Total size of serialized results of 8115 tasks (1131.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
ERROR TaskSetManager: Total size of serialized results of 8116 tasks (1131.3 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)

Cause

caused by actions like RDD’s collect() that send big chunk of data to the driver（不一定是因為RDD的問題哦~）

Solution

set by SparkConf: conf.set("spark.driver.maxResultSize", "3g")
set by spark-defaults.conf: spark.driver.maxResultSize 3g
set when calling spark-submit: --conf spark.driver.maxResultSize=3g

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 Spark的Driver和Executor 【Spark】SparkStreaming-Tasks-數量如何設置？ Spark的Driver節點和Executor節點 Spark Application、Driver、Job、stage、task Spark源碼分析之 Driver和Excutor是怎么跑起來的?(2.2.0版本) 記一次--------spark.driver.host參數報錯問題 java.lang.IllegalStateException(Connection to remote Spark driver was lost) 記一次--------spark 讀 mysql 報錯no suitable driver spark 2.3 導致driver OOM的一個SparkPlanGraphWrapper源碼的bug 【Spark】榨干Spark性能-driver、exector內存突破256M