Flink開發部署踩坑記錄


~~~~踩坑一:springboot+flink項目的日志問題~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

現象:開發環境IDE里調試運行都ok,部署到flink服務器上運行報如下錯誤: 

The main method caused an error: LoggerFactory is not a Logback LoggerContext but Logback is on the classpath. Either remove Logback or the competing implementation (class org.apache.logging.slf4j.Log4jLoggerFactory loaded from file:/usr/flink/lib/log4j-slf4j-impl-2.12.1.jar). If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml: org.apache.logging.slf4j.Log4jLoggerFactory

原因:java常用的日志框架(或者說api)有Commons Logging(對應的具體實現是log4j)與slf4j(對應的具體實現是logback)兩種。springboot與flink默認使用的應該都是slf4j(即logback,詳見官網:Apache Flink 1.7 Documentation: Best Practices),因此,除開我們的項目pom.xml文件里要排除掉log4j的引用之外,還需要在flink服務器上部署logback的jar包,否則就會出現上述錯誤。在我部署的flink服務器目錄下(位置:/usr/flink/lib/),沒有logback的包,只有log4j的包,所以就出錯了。可以參考這篇帖子:(1條消息) Flink使用logback須知_cloudera首席用戶的博客-CSDN博客_flink logback

 相關jar包下載位置:

——logback:Central Repository: ch/qos/logback (maven.org)

——slf4j:Central Repository: org/slf4j (maven.org)

 

~~~~踩坑二:springboot+flink項目本地IDE調試OK,發布到服務器執行失敗~~~~~~~~~~~~~~~~~

現象1:開發環境IDE里調試運行都ok,采用maven打包,部署到flink服務器上運行報如下錯誤: 

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate outputs in order.
    at org.apache.flink.streaming.api.graph.StreamConfig.getOutEdgesInOrder(StreamConfig.java:485)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriters(StreamTask.java:1341)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.createRecordWriterDelegate(StreamTask.java:1325)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:327)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:308)
    at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:76)
    at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.<init>(SourceStreamTask.java:72)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.flink.runtime.taskmanager.Task.loadAndInstantiateInvokable(Task.java:1525)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:730)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.keySelector of type org.apache.flink.api.java.functions.KeySelector in instance of org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner
    at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
    at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
    at java.util.ArrayList.readObject(ArrayList.java:799)
    at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2295)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
    at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
    at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
    at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
    at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
    at org.apache.flink.streaming.api.graph.StreamConfig.getOutEdgesInOrder(StreamConfig.java:482)
 
現象2:開發環境IDE里調試運行都ok,采用idea打包,部署到flink服務器上運行報如下錯誤:
org.apache.flink.util.FlinkRuntimeException: Could not execute application. at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:88) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ... 7 more Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No auto configuration classes found in META-INF/spring.factories. If you are using a custom packaging, make sure that file is correct. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ... 10 more Caused by: java.lang.IllegalArgumentException: No auto configuration classes found in META-INF/spring.factories. If you are using a custom packaging, make sure that file is correct. at org.springframework.util.Assert.notEmpty(Assert.java:470) at org.springframework.boot.autoconfigure.AutoConfigurationImportSelector.getCandidateConfigurations(AutoConfigurationImportSelector.java:180) at org.springframework.boot.autoconfigure.AutoConfigurationImportSelector.getAutoConfigurationEntry(AutoConfigurationImportSelector.java:123) at org.springframework.boot.autoconfigure.AutoConfigurationImportSelector$AutoConfigurationGroup.process(AutoConfigurationImportSelector.java:434) at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGrouping.getImports(ConfigurationClassParser.java:879) at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGroupingHandler.processGroupImports(ConfigurationClassParser.java:809) at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorHandler.process(ConfigurationClassParser.java:780) at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:193) at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:331) at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:247) at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:311) at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:112) at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:746) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) at catsti.innovation.flink.common.SpringBootApplicationUtil.run(SpringBootApplicationUtil.java:38) at catsti.innovation.flink.CarCountApplication.main(CarCountApplication.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)  
 
原因:還未找到。這坑如果爬不出來估計就得放棄springboot了。
 
 

~~~~踩坑三:編譯的jar包發上傳到flink服務器,顯示找不到主類~~~~~~~~~~~~~~~~~

原因:設置打包的時候,路徑沒設置對,不能按照默認的來,src后面的要全刪掉。嚴格來說這不是flink的問題,是對idea打包設置不熟悉造成的。

 可以參考這篇帖子:

 
 
 
~~~~踩坑四:上傳到flink服務器的jar包,當flink重啟后丟失的問題~~~~~~~~~~~~~~~~~

在flink-conf.yaml中添加配置,告訴flink把上傳的包存在哪里,以及重啟后去哪里找(具體位置可以根據自己需要設置):

jobmanager.web.tmpdir: /tmp/flink-web
jobmanager.web.upload.dir: /tmp/flink-web


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM