spark 3.0.0源碼編譯以及IDEA本地調試運行

1.下載源碼

2. 修改pom.xml

• 把maven.version修改為3.6.3（同本地maven版本一致）

• 把scala.version修改為2.12.11（同本地scala版本一致）

• 把maven源修改為阿里雲加快文件下載速度

    <repositories>
      <repository>
        <id>gcs-maven-central-mirror</id>
        <!--
          Google Mirror of Maven Central, placed first so that it's used instead of flaky Maven Central.
          See https://storage-download.googleapis.com/maven-central/index.html
        -->
        <name>GCS Maven Central mirror</name>
        <url>https://maven-central.storage-download.googleapis.com/maven2/</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
      <repository>
        <!--
          This is used as a fallback when the first try fails.
        -->
        <id>central</id>
        <name>Maven Repository</name>
        <url>https://repo.maven.apache.org/maven2</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
    </repositories>
  
  替換為
    <repositories>
      <repository>
        <id>alimaven</id>
        <name>aliyun maven</name>
        <url>https://maven.aliyun.com/repository/central</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
    </repositories>
  
  
    <pluginRepositories>
      <pluginRepository>
        <id>gcs-maven-central-mirror</id>
        <!--
          Google Mirror of Maven Central, placed first so that it's used instead of flaky Maven Central.
          See https://storage-download.googleapis.com/maven-central/index.html
        -->
        <name>GCS Maven Central mirror</name>
        <url>https://maven-central.storage-download.googleapis.com/maven2/</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </pluginRepository>
      <pluginRepository>
        <id>central</id>
        <url>https://repo.maven.apache.org/maven2</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </pluginRepository>
    </pluginRepositories>
  替換為
    <pluginRepositories>
      <pluginRepository>
        <id>alimaven</id>
        <name>aliyun maven</name>
        <url>https://maven.aliyun.com/repository/central</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </pluginRepository>
    </pluginRepositories>
  
  

3.編譯

至此，編譯結束

4.IDEA打開

用IDEA打開pom.xml，選擇maven project。

以sparkPi為例添加運行config。

將assembly/target/scala/jars添加到Project Structure -- Moudles中。

5.遇到的問題

• 找不到spark-version-info.properties

  在git bash 中執行

  build/spark-build-info /core/target/extra-resources 3.0.0
  

  然后將spark-version-info.properties復制到examples/target/scala-2.12/classes 目錄下（這里具體的位置可以通過debug來找到）

• java.lang.NoClassDefFoundError 例如：org/eclipse/jetty/server/connecter

  在pom.xml文件中找到相關配置，將scope的provided修改為compile

• 注意在Run Configurations中，需要將Before launch 中的build減掉