spark - tasks is bigger than spark.driver.maxResultSize


Error

ERROR TaskSetManager: Total size of serialized results of 8113 tasks (1131.0 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
ERROR TaskSetManager: Total size of serialized results of 8114 tasks (1131.1 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
ERROR TaskSetManager: Total size of serialized results of 8115 tasks (1131.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
ERROR TaskSetManager: Total size of serialized results of 8116 tasks (1131.3 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)

Cause

  • caused by actions like RDD’s collect() that send big chunk of data to the driver(不一定是因為RDD的問題哦~)

Solution

  • set by SparkConf: conf.set("spark.driver.maxResultSize", "3g")
  • set by spark-defaults.confspark.driver.maxResultSize 3g
  • set when calling spark-submit--conf spark.driver.maxResultSize=3g


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM