spark - tasks is bigger than spark.driver.maxResultSize

本文转载自查看原文 2016-12-13 12:02 11489 spark/ 大数据

Error

ERROR TaskSetManager: Total size of serialized results of 8113 tasks (1131.0 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
ERROR TaskSetManager: Total size of serialized results of 8114 tasks (1131.1 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
ERROR TaskSetManager: Total size of serialized results of 8115 tasks (1131.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
ERROR TaskSetManager: Total size of serialized results of 8116 tasks (1131.3 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)

Cause

caused by actions like RDD’s collect() that send big chunk of data to the driver（不一定是因为RDD的问题哦~）

Solution

set by SparkConf: conf.set("spark.driver.maxResultSize", "3g")
set by spark-defaults.conf: spark.driver.maxResultSize 3g
set when calling spark-submit: --conf spark.driver.maxResultSize=3g

免责声明！

本站转载的文章为个人学习借鉴使用，本站对版权不负任何法律责任。如果侵犯了您的隐私权益，请联系本站邮箱yoyou2525@163.com删除。

猜您在找 Spark的Driver和Executor 【Spark】SparkStreaming-Tasks-数量如何设置？ Spark的Driver节点和Executor节点 Spark Application、Driver、Job、stage、task Spark源码分析之 Driver和Excutor是怎么跑起来的?(2.2.0版本) 记一次--------spark.driver.host参数报错问题 java.lang.IllegalStateException(Connection to remote Spark driver was lost) 记一次--------spark 读 mysql 报错no suitable driver spark 2.3 导致driver OOM的一个SparkPlanGraphWrapper源码的bug 【Spark】榨干Spark性能-driver、exector内存突破256M