使用idea編寫spark程序,想要查看源代碼,點進去是compiled code
private[sql] def this(sc : org.apache.spark.SparkContext) = { /* compiled code */ }
def version : scala.Predef.String = { /* compiled code */ }
網上很多方法都是下載java 反編譯工具,如java bytecode compiler。
按照方法試了,倒是java類的class點進去能夠看到反編譯的代碼,scala的class點進去依然沒有反編譯。可能是idea版本問題。
試了幾次遂放棄。
因為即使是反編譯成功了,依然不是我想要的結果。反統的代碼畢竟不是源碼,也沒有注釋。
所以為何不利用maven下載源碼與注釋呢。
可在全局配置里勾選source和doc兩項,maven在引入依賴時自動下載源碼和注釋。
當然更好的方案是,不勾選此兩項,僅在需要查看源碼的時候,去下載。
好了,點擊查看spark源碼,這是我想要的效果:
/**
* Activate the stream and then wrap a callout to runActivatedStream, handling start and stop.
*
* Note that this method ensures that [[QueryStartedEvent]] and [[QueryTerminatedEvent]] are
* posted such that listeners are guaranteed to get a start event before a termination.
* Furthermore, this method also ensures that [[QueryStartedEvent]] event is posted before the
* `start()` method returns.
*/
private def runStream(): Unit = {
try {
sparkSession.sparkContext.setJobGroup(runId.toString, getBatchDescriptionString,
interruptOnCancel = true)
sparkSession.sparkContext.setLocalProperty(StreamExecution.QUERY_ID_KEY, id.toString)
if (sparkSession.sessionState.conf.streamingMetricsEnabled) {
sparkSession.sparkContext.env.metricsSystem.registerSource(streamMetrics)
}
// `postEvent` does not throw non fatal exception.
val startTimestamp = triggerClock.getTimeMillis()
postEvent(new QueryStartedEvent(id, runId, name, formatTimestamp(startTimestamp)))