spark、spark調優、spark streaming常見問題總結
1.server.TransportChannelHandler: Exception in connection from xxxxxx。java.io.IOException: Connection reset by peer
增加內存后解決了問題
2.集群模式下spark應用ID應該是local開頭，如果是application開頭，則要看下代碼里setMaster方法是否寫死了local模式
3.參數優化最終參數
spark-submit \
--master yarn \
--num-executors 50 \
--executor-memory 4G \
--executor-cores 2 \
--driver-memory 8G \
--conf spark.default.parallelism=200 \
--conf spark.yarn.executor.memoryOverhead=2048 \
--conf spark.storage.memoryFraction=0.5 \
--conf spark.shuffle.memoryFraction=0.3 \
--class com.xxx.rc.enginetask.service.article.ArticleHBaseALS
4.Error:(18, 31) type arguments [String,String,io.netty.handler.codec.string.StringDecoder,io.netty.handler.codec.string.StringDecoder] conform to the bounds of none of the overloaded alternatives of
value createDirectStream: [K, V, KD <: kafka.serializer.Decoder[K], VD <: kafka.serializer.Decoder[V]](jssc: org.apache.spark.streaming.api.java.JavaStreamingContext, keyClass: Class[K], valueClass: Class[V], keyDecoderClass: Class[KD], valueDecoderClass: Class[VD], kafkaParams: java.util.Map[String,String], topics: java.util.Set[String])org.apache.spark.streaming.api.java.JavaPairInputDStream[K,V] <and> [K, V, KD <: kafka.serializer.Decoder[K], VD <: kafka.serializer.Decoder[V]](ssc: org.apache.spark.streaming.StreamingContext, kafkaParams: Map[String,String], topics: Set[String])(implicit evidence$19: scala.reflect.ClassTag[K], implicit evidence$20: scala.reflect.ClassTag[V], implicit evidence$21: scala.reflect.ClassTag[KD], implicit evidence$22: scala.reflect.ClassTag[VD])org.apache.spark.streaming.dstream.InputDStream[(K, V)]
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicSet)
錯誤包：import io.netty.handler.codec.string.StringDecoder
正確包：import kafka.serializer.StringDecoder
5.查詢某個任務的日志
yarn logs -applicationId xxxxxxxxxxxxxxxx