Flink1.10.1集成Hadoop3.0.0源碼編譯實戰


詳細的死磕到底系列,可關注微信公眾號:大數據從業者

https://mp.weixin.qq.com/s/saCIS5XCfTZisXlEeSHsuA

直接從github拉取flink-1.10.1版本代碼
git clone -b release-1.10.1 https://github.com/apache/flink.git

flink-1.10.1修復了1.10中158處bug,並進行了優化。
官方也是強烈建議所有用戶升級到1.10.1。

廢話不多說,直接進入正題,開始flink-1.10.1版本的源碼編譯。

編譯環境需要提前准備好maven和java。
最簡單的編譯命令如下:
mvn clean install -DskipTests

如果想加快源碼編譯進度,可以在編譯過程中跳過tests、QA插件、文檔等
編譯命令如下:
mvn clean install -DskipTests -Dfast

這兩種方式說的都是不考慮flink與Hadoop整合的使用場景。
如果需要與Hadoop整合使用,需要提供hadoop classes依賴。
這里有兩種方式:
方式1:將hadoop classpath添加到flink。這種方式工作量比較小,簡單方便,但是hadoop class依賴眾多,可能與flink class有jar沖突的情況
方式2:直接將Hadoop classes依賴打包放置到flink/lib文件夾下
官方有打好的幾個hadoop版本對應的包
Pre-bundled Hadoop 2.4.1:https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.4.1-10.0/flink-shaded-hadoop-2-uber-2.4.1-10.0.jar
Pre-bundled Hadoop 2.6.5:https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.6.5-10.0/flink-shaded-hadoop-2-uber-2.6.5-10.0.jar
Pre-bundled Hadoop 2.7.5:https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-10.0/flink-shaded-hadoop-2-uber-2.7.5-10.0.jar
Pre-bundled Hadoop 2.8.3:https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
當然,如果這里沒有你需要的hadoop版本,你就得自己打包了。
打包需要使用另外一個項目:flink-shaded
git clone -b release-9.0 https://github.com/apache/flink-shaded.git
注意:這里一定要使用9.0,因為flink-1.10.1項目中依賴的為9.0
flink-release-1.10.1\flink-release-1.10.1\pom.xml中可以查看或者更改版本
<flink.shaded.version>9.0</flink.shaded.version>
編譯打包命令:
mvn clean install -Dhadoop.version=3.0.0
注意:如果需要指定hadoop的發行商,需要使用-Pvendor-repos。當然前提是maven倉庫增加了對應發行商的倉庫地址,詳見https://maven.apache.org/guides/mini/guide-multiple-repositories.html
如:mvn clean install -Pvendor-repos -Dhadoop.version=3.0.0-cdh6.2.0

執行上述步驟,將打好的包flink-shaded-hadoop-2-uber-3.0.0-9.0.jar拷貝到flink/lib文件夾下

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:56 min
[INFO] Finished at: 2020-06-01T18:28:42+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-hadoop-fs: Could not resolve dependencies for project org.apache.flink:flink-hadoop-fs:jar:1.10.1: Could not find artifact org.apache.flink:flink-shaded-hadoop-2:jar:3.0.0-9.0 in nexus (http://maven.aliyun.com/nexus/content/groups/public/) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :flink-hadoop-fs


[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ flink-shaded-hadoop-2-uber ---
[INFO] Installing /root/Desktop/sourceCodes/flink-shaded-release-9.0/flink-shaded-hadoop-2-uber/target/flink-shaded-hadoop-2-uber-3.0.0-9.0.jar to /root/.m2/repository/org/apache/flink/flink-shaded-hadoop-2-uber/3.0.0-9.0/flink-shaded-hadoop-2-uber-3.0.0-9.0.jar
[INFO] Installing /root/Desktop/sourceCodes/flink-shaded-release-9.0/flink-shaded-hadoop-2-uber/target/dependency-reduced-pom.xml to /root/.m2/repository/org/apache/flink/flink-shaded-hadoop-2-uber/3.0.0-9.0/flink-shaded-hadoop-2-uber-3.0.0-9.0.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] flink-shaded 9.0 ................................... SUCCESS [ 3.026 s]
[INFO] flink-shaded-force-shading ......................... SUCCESS [ 0.903 s]
[INFO] flink-shaded-asm-7 7.1-9.0 ......................... SUCCESS [ 1.076 s]
[INFO] flink-shaded-guava-18 18.0-9.0 ..................... SUCCESS [ 1.928 s]
[INFO] flink-shaded-netty-4 4.1.39.Final-9.0 .............. SUCCESS [ 4.613 s]
[INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-9.0 SUCCESS [ 0.915 s]
[INFO] flink-shaded-jackson-parent 2.10.1-9.0 ............. SUCCESS [ 0.108 s]
[INFO] flink-shaded-jackson-2 2.10.1-9.0 .................. SUCCESS [ 1.929 s]
[INFO] flink-shaded-jackson-module-jsonSchema-2 2.10.1-9.0 SUCCESS [ 1.529 s]
[INFO] flink-shaded-hadoop-2 3.0.0-9.0 .................... SUCCESS [ 28.785 s]
[INFO] flink-shaded-hadoop-2-uber 3.0.0-9.0 ............... SUCCESS [ 57.217 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:42 min
[INFO] Finished at: 2020-06-01T18:37:42+08:00
[INFO] ------------------------------------------------------------------------


mvn clean install -DskipTests -Dfast -Dhadoop.version=3.0.0

[INFO] another-dummy-fs ................................... SUCCESS [ 0.336 s]
[INFO] flink-tpch-test .................................... SUCCESS [ 4.647 s]
[INFO] flink-streaming-kinesis-test ....................... SUCCESS [ 20.212 s]
[INFO] flink-elasticsearch7-test .......................... SUCCESS [ 6.283 s]
[INFO] flink-end-to-end-tests-common-kafka ................ SUCCESS [ 3.538 s]
[INFO] flink-tpcds-test ................................... SUCCESS [ 0.910 s]
[INFO] flink-statebackend-heap-spillable .................. SUCCESS [ 0.594 s]
[INFO] flink-contrib ...................................... SUCCESS [ 0.063 s]
[INFO] flink-connector-wikiedits .......................... SUCCESS [ 18.538 s]
[INFO] flink-yarn-tests ................................... SUCCESS [01:11 min]
[INFO] flink-fs-tests ..................................... SUCCESS [ 1.067 s]
[INFO] flink-docs ......................................... SUCCESS [ 5.033 s]
[INFO] flink-ml-parent .................................... SUCCESS [ 0.056 s]
[INFO] flink-ml-api ....................................... SUCCESS [ 0.412 s]
[INFO] flink-ml-lib ....................................... SUCCESS [01:05 min]
[INFO] flink-walkthroughs ................................. SUCCESS [ 0.066 s]
[INFO] flink-walkthrough-common ........................... SUCCESS [ 1.145 s]
[INFO] flink-walkthrough-table-java ....................... SUCCESS [ 0.205 s]
[INFO] flink-walkthrough-table-scala ...................... SUCCESS [ 0.254 s]
[INFO] flink-walkthrough-datastream-java .................. SUCCESS [ 0.273 s]
[INFO] flink-walkthrough-datastream-scala 1.10.1 .......... SUCCESS [ 0.165 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 55:02 min
[INFO] Finished at: 2020-06-02T16:13:57+08:00
[INFO] ------------------------------------------------------------------------


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM