Flink升級到1.4版本遇到的坑


  Flink 1.4沒出來以前,一直使用Flink 1.3.2,感覺還算穩定,最近將運行環境升級到1.4,遇到了一些坑:

 

1.需要將可運行程序,基於1.4.0重新編譯一次

2.對比了一下flink-conf.yml中的配置,發現一處不同點:

# The port under which the web-based runtime monitor listens.
# A value of -1 deactivates the web server.

1.3.2:jobmanager.web.port: 8081

1.4.0:web.port: 8081

1.4.0 少了前面的“jobmanager”,但是查看官網1.4.0的文檔,配置項卻是:jobmanager.web.port(https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/config.html)

查看源代碼,原來是兩個都行,不過后面應該建議都用web開頭的,改了還不少:

3.在Configuration的Common Options部分,1.4多了如下的東西:

classloader.resolve-order: Whether Flink should use a child-first ClassLoader when loading 
user-code classes or a parent-first ClassLoader. Can be one of parent-first or child-first. (default: child-first) classloader.parent-first-patterns: A (semicolon-separated) list of patterns that specifies
which classes should always be resolved through the parent ClassLoader first.
A pattern is a simple prefix that is checked against the fully qualified class name.
By default, this is set to java.;org.apache.flink.;javax.annotation;org.slf4j;org.apache.log4j;org.apache.logging.log4j;ch.qos.logback.
If you want to change this setting you have to make sure to also include the default patterns in your list of patterns if you want to keep that default behaviour.

Flink程序,原來在1.3.2上跑着沒問題,換到1.4后,總是報:

java.lang.LinkageError: loader constraint violation:
loader (instance of org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$ChildFirstClassLoader)
previously initiated loading for a different type with name "scala/collection/Iterable"

 

查看發布說明,才知道默認類加載順序變了:

https://flink.apache.org/news/2017/12/12/release-1.4.0.html

 

在flink-conf.yml中配置了

classloader.resolve-order: parent-first

后問題解決

所以每個版本的發布說明,得仔細看清楚。

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM