spark 3.0.0源码编译以及IDEA本地调试运行

1.下载源码

2. 修改pom.xml

• 把maven.version修改为3.6.3（同本地maven版本一致）

• 把scala.version修改为2.12.11（同本地scala版本一致）

• 把maven源修改为阿里云加快文件下载速度

    <repositories>
      <repository>
        <id>gcs-maven-central-mirror</id>
        <!--
          Google Mirror of Maven Central, placed first so that it's used instead of flaky Maven Central.
          See https://storage-download.googleapis.com/maven-central/index.html
        -->
        <name>GCS Maven Central mirror</name>
        <url>https://maven-central.storage-download.googleapis.com/maven2/</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
      <repository>
        <!--
          This is used as a fallback when the first try fails.
        -->
        <id>central</id>
        <name>Maven Repository</name>
        <url>https://repo.maven.apache.org/maven2</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
    </repositories>
  
  替换为
    <repositories>
      <repository>
        <id>alimaven</id>
        <name>aliyun maven</name>
        <url>https://maven.aliyun.com/repository/central</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
    </repositories>
  
  
    <pluginRepositories>
      <pluginRepository>
        <id>gcs-maven-central-mirror</id>
        <!--
          Google Mirror of Maven Central, placed first so that it's used instead of flaky Maven Central.
          See https://storage-download.googleapis.com/maven-central/index.html
        -->
        <name>GCS Maven Central mirror</name>
        <url>https://maven-central.storage-download.googleapis.com/maven2/</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </pluginRepository>
      <pluginRepository>
        <id>central</id>
        <url>https://repo.maven.apache.org/maven2</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </pluginRepository>
    </pluginRepositories>
  替换为
    <pluginRepositories>
      <pluginRepository>
        <id>alimaven</id>
        <name>aliyun maven</name>
        <url>https://maven.aliyun.com/repository/central</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </pluginRepository>
    </pluginRepositories>
  
  

3.编译

至此，编译结束

4.IDEA打开

用IDEA打开pom.xml，选择maven project。

以sparkPi为例添加运行config。

将assembly/target/scala/jars添加到Project Structure -- Moudles中。

5.遇到的问题

• 找不到spark-version-info.properties

  在git bash 中执行

  build/spark-build-info /core/target/extra-resources 3.0.0
  

  然后将spark-version-info.properties复制到examples/target/scala-2.12/classes 目录下（这里具体的位置可以通过debug来找到）

• java.lang.NoClassDefFoundError 例如：org/eclipse/jetty/server/connecter

  在pom.xml文件中找到相关配置，将scope的provided修改为compile

• 注意在Run Configurations中，需要将Before launch 中的build减掉