Spark報錯處理
1、問題：org.apache.spark.SparkException: Exception thrown in awaitResult
分析：出現這個情況的原因是spark啟動的時候設置的是hostname啟動的，導致訪問的時候DNS不能解析主機名導致。
問題解決：
第一種方法：確保URL是spark://服務器ip:7077，而不是spark://hostname:7077；啟動的時候指定-h ip地址
第二種方法：修改主機的host文件添加主機的解析記錄（推薦這種方式）
Ip 主機名
第三種方法：hive.metastore.try.direct.sql: false (in hive-site.xml)
2、spark2.x版本使用hive，即copy一份hive-site.xml文件到spark2.x的conf目錄下。
使用spark的bin目錄下的spark-sql進入終端時總提示一個warning：
Thu Jun 15 12:56:05 CST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
解決方法：
修改hive-site.xml文件下的mysql連接的url，設置useSSL=false。由於hive-site.xml文件采用的是xml格式，所以不支持直接使用&連接，需要使用&進行連接。
<value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true&useSSL=false</value>
重啟spark即可，
#../sbin/stop-all.sh
#../sbin/start-all.sh
3、 問題：
Spark運行了一段時間，數據量上來以后，出現了一個這樣的報錯：
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/10/26 20:29:00 ERROR Executor: Exception in task 39.1 in stage 8.0 (TID 1122)
java.io.FileNotFoundException: /tmp/spark-2de5fa03-a7cb-47a2-9540-403de85d0371/executor-eebecccb-4cdb-4b85-80a3-73c4baa4c7bd/blockmgr-fc644c14-23e8-401c-aee8-00bc108bf607/2b/temp_shuffle_75eb7338-be41-41b4-bed4-5dcb0c1d0fdf (No space left on device)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:102)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
從日志報錯來看說是沒有空間了，spark默認是把臨時文件存放到/tmp目錄下。需要修改啊！！！放到一個大存儲的地方：
解決方法：
修改spark-env.sh
export SPARK_DRIVER_MEMORY=5g
export SPARK_LOCAL_DIRS=/data/sparktmp
不要添加到spark-defaault.conf里面去，因為spark從1.0版本已經放棄了spark.local.dir參數。
源碼分析：
（1） DiskBlockManager類中的下面的方法
通過日志我們最終定位這塊出現的錯誤
/**
* Create local directories for storing block data. These directories are
* located inside configured local directories and won't
* be deleted on JVM exit when using the external shuffle service.
*/
private def createLocalDirs(conf: SparkConf): Array[File] = {
Utils.getConfiguredLocalDirs(conf).flatMap { rootDir =>
try {
val localDir = Utils.createDirectory(rootDir, "blockmgr")
logInfo(s"Created local directory at $localDir")
Some(localDir)
} catch {
case e: IOException =>
logError(s"Failed to create local dir in $rootDir. Ignoring this directory.", e)
None
}
}
}
（2） SparkConf.scala 類中的方法
這個方法告訴我們在spark-defaults.conf 中配置spark.local.dir參數在spark1.0 版本后已經過時。
/** Checks for illegal or deprecated config settings. Throws an exception for the former. Not
* idempotent - may mutate this conf object to convert deprecated settings to supported ones. */
private[spark] def validateSettings() {
if (contains("spark.local.dir")) {
val msg = "In Spark 1.0 and later spark.local.dir will be overridden by the value set by " +
"the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN)."
logWarning(msg)
}
val executorOptsKey = "spark.executor.extraJavaOptions"
val executorClasspathKey = "spark.executor.extr
。。。。
}
（3）Utils.scala 類中的方法
通過分析下面的代碼，我們發現不在spark-env.sh 下配置SPARK_LOCAL_DIRS的情況下，
通過該conf.get("spark.local.dir", System.getProperty("java.io.tmpdir")).split(",")設置spark.local.dir，然后或根據路徑創建，導致上述錯誤。
故我們直接在spark-env.sh 中設置SPARK_LOCAL_DIRS 即可解決。
然后我們直接在spark-env.sh 中配置：
export SPARK_LOCAL_DIRS=/home/hadoop/data/sparktmp
/**
* Return the configured local directories where Spark can write files. This
* method does not create any directories on its own, it only encapsulates the
* logic of locating the local directories according to deployment mode.
*/
def getConfiguredLocalDirs(conf: SparkConf): Array[String] = {
val shuffleServiceEnabled = conf.getBoolean("spark.shuffle.service.enabled", false)
if (isRunningInYarnContainer(conf)) {
// If we are in yarn mode, systems can have different disk layouts so we must set it
// to what Yarn on this system said was available. Note this assumes that Yarn has
// created the directories already, and that they are secured so that only the
// user has access to them.
getYarnLocalDirs(conf).split(",")
} else if (conf.getenv("SPARK_EXECUTOR_DIRS") != null) {
conf.getenv("SPARK_EXECUTOR_DIRS").split(File.pathSeparator)
} else if (conf.getenv("SPARK_LOCAL_DIRS") != null) {
conf.getenv("SPARK_LOCAL_DIRS").split(",")
} else if (conf.getenv("MESOS_DIRECTORY") != null && !shuffleServiceEnabled) {
// Mesos already creates a directory per Mesos task. Spark should use that directory
// instead so all temporary files are automatically cleaned up when the Mesos task ends.
// Note that we don't want this if the shuffle service is enabled because we want to
// continue to serve shuffle files after the executors that wrote them have already exited.
Array(conf.getenv("MESOS_DIRECTORY"))
} else {
if (conf.getenv("MESOS_DIRECTORY") != null && shuffleServiceEnabled) {
logInfo("MESOS_DIRECTORY available but not using provided Mesos sandbox because " +
"spark.shuffle.service.enabled is enabled.")
}
// In non-Yarn mode (or for the driver in yarn-client mode), we cannot trust the user
// configuration to point to a secure directory. So create a subdirectory with restricted
// permissions under each listed directory.
conf.get("spark.local.dir", System.getProperty("java.io.tmpdir")).split(",")
}
}
3、Join condition is missing or trivial.Use the CROSS JOIN syntax to allow cartesian products between these relations.;
解決方法：
spark.sql.crossjoin.enabled: true
4、Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "eval(Lorg/apache/spark/sql/catalyst/InternalRow;)Z" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate" grows beyond 64 KB
解決方法：
spark.sql.codegen.wholeStage : false
5、java.lang.OutOfMemoryError: Java heap space
解決方法：
spark.driver.memory : 10g <to a higher-value>
spark.sql.ui.retainedExecutions: 5 <to some lower-value>