error:org.apache.spark.shuffle.FetchFailedException: Failed to connect to xxxx/xx.xx.xx.xx:xxxx
定位來定位去與防火牆等無關。反復查看日志：
2019-09-30 11:00:46,521 | WARN | [dispatcher-event-loop-50] | Lost task 5.0 in stage 1.2 (TID 24441, dggsafe0321-cm, executor 7): ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 4.6 GB of 4.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66)
2019-09-30 11:00:46,521 | INFO | [dag-scheduler-event-loop] | Resubmitted ShuffleMapTask(6, 25830), so marking it as still running | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
2019-09-30 11:00:46,522 | WARN | [dispatcher-event-loop-50] | Lost task 4.0 in stage 1.2 (TID 24440, dggsafe0321-cm, executor 7): ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 4.6 GB of 4.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66)
2019-09-30 11:00:46,522 | INFO | [dag-scheduler-event-loop] | Resubmitted ShuffleMapTask(6, 15603), so marking it as still running | org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
發現節點內存溢出,導致節假死，導致節點無法訪問，擴展相應執行內存重啟就行。
--driver-memory 4g --executor-memory 6g