Flink独立部署通过start-cluster.sh启动集群,在JobManager启动时候,调用StandaloneSessionClusterEntrypoint。
将之前通过start-cluster.sh脚本启动得到的配置信息:
/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/bin/java -Xms1024m -Xmx1024m -Dlog.file=/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/log/flink-xxxx-standalonesession-0-ZBMAC-C02VR1RWH.local.log -Dlog4j.configuration=file:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/conf/log4j.properties -Dlogback.configurationFile=file:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/conf/logback.xml -classpath /usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/flink-table-blink_2.12-1.10.0.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/flink-table_2.12-1.10.0.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/log4j-1.2.17.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/slf4j-log4j12-1.7.15.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/flink-dist_2.12-1.10.0.jar::: org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint --configDir /usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/conf --executionMode cluster > /usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/log/flink-xxxx-standalonesession-0-ZBMAC-C02VR1RWH.local.out 200<&- 2>&1 < /dev/null &
配置到启动参数VM options和Program arguments中。
这里面需要注意的是VM options,内容如下:
-Xms1024m -Xmx1024m -Dlog.file=/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/log/flink-${username}-standalonesession-0-ZBMAC-C02VR1RWH.local.log -Dlog4j.configuration=file:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/conf/log4j.properties -Dlogback.configurationFile=file:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/conf/logback.xml
Program arguments,内容如下:
--configDir /usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/conf --executionMode cluster
一开始的时候对这两个配置不正确,通过start-cluster.sh 脚本启动时候,日志中包含代码logEnvironmentInfo中输出的参数信息,内容如下
里面有对参数进行分类,哪些属于JVM参数,哪些数据程序参数。
在启动过程中发现,程序能够启动,logEnvironmentInfo方法内容无法输出,生成的LOG对象isInfoEnabled为false,调整log4j的配置参数,把日志级别设置成INFO也不生效,控制台输出以下内容:
/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:63889,suspend=y,server=n -Xms1024m -Xmx1024m -javaagent:/Users/fanjiaxing/Library/Caches/IntelliJIdea2018.1/captureAgent/debugger-agent.jar=file:/private/var/folders/lp/f11z9p_j2hl6q7pct857rx9wfsr47x/T/capture.props -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/lib/tools.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-runtime/target/classes:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-core/target/classes:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-annotations/target/classes:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-metrics/flink-metrics-core/target/classes:/Users/fanjiaxing/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar:/Users/fanjiaxing/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:/Users/fanjiaxing/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/fanjiaxing/.m2/repository/org/apache/commons/commons-compress/1.18/commons-compress-1.18.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-java/target/classes:/Users/fanjiaxing/.m2/repository/org/apache/commons/commons-math3/3.5/commons-math3-3.5.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-queryable-state/flink-queryable-state-client-java/target/classes:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-filesystems/flink-hadoop-fs/target/classes:/Users/fanjiaxing/.m2/repository/org/apache/flink/flink-shaded-hadoop-2/2.4.1-9.0/flink-shaded-hadoop-2-2.4.1-9.0.jar:/Users/fanjiaxing/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2.jar:/Users/fanjiaxing/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/fanjiaxing/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/fanjiaxing/.m2/repository/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar:/Users/fanjiaxing/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar:/Users/fanjiaxing/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/fanjiaxing/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar:/Users/fanjiaxing/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/Users/fanjiaxing/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/Users/fanjiaxing/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/fanjiaxing/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/fanjiaxing/.m2/repository/com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbuilder-0.4.jar:/Users/fanjiaxing/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/fanjiaxing/.m2/repository/commons-configuration/commons-configuration/1.7/commons-configuration-1.7.jar:/Users/fanjiaxing/.m2/repository/commons-digester/commons-digester/1.8.1/commons-digester-1.8.1.jar:/Users/fanjiaxing/.m2/repository/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar:/Users/fanjiaxing/.m2/repository/commons-beanutils/commons-beanutils/1.9.3/commons-beanutils-1.9.3.jar:/Users/fanjiaxing/.m2/repository/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar:/Users/fanjiaxing/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/Users/fanjiaxing/.m2/repository/javax/xml/bind/jaxb-api/2.3.1/jaxb-api-2.3.1.jar:/Users/fanjiaxing/.m2/repository/javax/activation/javax.activation-api/1.2.0/javax.activation-api-1.2.0.jar:/Users/fanjiaxing/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/fanjiaxing/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/fanjiaxing/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/fanjiaxing/.m2/repository/org/apache/flink/flink-shaded-netty/4.1.39.Final-9.0/flink-shaded-netty-4.1.39.Final-9.0.jar:/Users/fanjiaxing/.m2/repository/org/apache/flink/flink-shaded-guava/18.0-9.0/flink-shaded-guava-18.0-9.0.jar:/Users/fanjiaxing/.m2/repository/org/apache/flink/flink-shaded-asm-7/7.1-9.0/flink-shaded-asm-7-7.1-9.0.jar:/Users/fanjiaxing/.m2/repository/org/apache/flink/flink-shaded-jackson/2.10.1-9.0/flink-shaded-jackson-2.10.1-9.0.jar:/Users/fanjiaxing/.m2/repository/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.jar:/Users/fanjiaxing/.m2/repository/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1.jar:/Users/fanjiaxing/.m2/repository/org/javassist/javassist/3.24.0-GA/javassist-3.24.0-GA.jar:/Users/fanjiaxing/.m2/repository/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.jar:/Users/fanjiaxing/.m2/repository/com/typesafe/akka/akka-actor_2.11/2.5.21/akka-actor_2.11-2.5.21.jar:/Users/fanjiaxing/.m2/repository/com/typesafe/config/1.3.0/config-1.3.0.jar:/Users/fanjiaxing/.m2/repository/org/scala-lang/modules/scala-java8-compat_2.11/0.7.0/scala-java8-compat_2.11-0.7.0.jar:/Users/fanjiaxing/.m2/repository/com/typesafe/akka/akka-remote_2.11/2.5.21/akka-remote_2.11-2.5.21.jar:/Users/fanjiaxing/.m2/repository/io/netty/netty/3.10.6.Final/netty-3.10.6.Final.jar:/Users/fanjiaxing/.m2/repository/com/typesafe/akka/akka-stream_2.11/2.5.21/akka-stream_2.11-2.5.21.jar:/Users/fanjiaxing/.m2/repository/org/reactivestreams/reactive-streams/1.0.2/reactive-streams-1.0.2.jar:/Users/fanjiaxing/.m2/repository/com/typesafe/ssl-config-core_2.11/0.3.7/ssl-config-core_2.11-0.3.7.jar:/Users/fanjiaxing/.m2/repository/com/typesafe/akka/akka-protobuf_2.11/2.5.21/akka-protobuf_2.11-2.5.21.jar:/Users/fanjiaxing/.m2/repository/com/typesafe/akka/akka-slf4j_2.11/2.5.21/akka-slf4j_2.11-2.5.21.jar:/Users/fanjiaxing/.m2/repository/org/clapper/grizzled-slf4j_2.11/1.3.2/grizzled-slf4j_2.11-1.3.2.jar:/Users/fanjiaxing/.m2/repository/com/github/scopt/scopt_2.11/3.5.0/scopt_2.11-3.5.0.jar:/Users/fanjiaxing/.m2/repository/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar:/Users/fanjiaxing/.m2/repository/com/twitter/chill_2.11/0.7.6/chill_2.11-0.7.6.jar:/Users/fanjiaxing/.m2/repository/com/twitter/chill-java/0.7.6/chill-java-0.7.6.jar:/Users/fanjiaxing/.m2/repository/org/apache/zookeeper/zookeeper/3.4.10/zookeeper-3.4.10.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-shaded-curator/target/classes:/Users/fanjiaxing/.m2/repository/org/apache/curator/curator-recipes/2.12.0/curator-recipes-2.12.0.jar:/Users/fanjiaxing/.m2/repository/org/apache/curator/curator-framework/2.12.0/curator-framework-2.12.0.jar:/Users/fanjiaxing/.m2/repository/org/apache/curator/curator-client/2.12.0/curator-client-2.12.0.jar:/Users/fanjiaxing/.m2/repository/org/lz4/lz4-java/1.5.0/lz4-java-1.5.0.jar:/Users/fanjiaxing/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/Users/fanjiaxing/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.1.1/scala-parser-combinators_2.11-1.1.1.jar:/Users/fanjiaxing/.m2/repository/com/github/oshi/oshi-core/3.4.0/oshi-core-3.4.0.jar:/Users/fanjiaxing/.m2/repository/net/java/dev/jna/jna-platform/4.2.2/jna-platform-4.2.2.jar:/Users/fanjiaxing/.m2/repository/net/java/dev/jna/jna/4.2.2/jna-4.2.2.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/tools/force-shading/target/classes:/Users/fanjiaxing/.m2/repository/org/slf4j/slf4j-api/1.7.15/slf4j-api-1.7.15.jar:/Users/fanjiaxing/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/fanjiaxing/.m2/repository/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint -Dlog.file=/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/log/flink-fanjiaxing-standalonesession-0-ZBMAC-C02VR1RWH.local.log -classpath /usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/flink-table-blink_2.12-1.10.0.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/flink-table_2.12-1.10.0.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/log4j-1.2.17.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/slf4j-log4j12-1.7.15.jar:/usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/lib/flink-dist_2.12-1.10.0.jar --configDir /usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/conf --executionMode cluster > /usr/local/OpenSourceProjectCode/flink-release-1.10.0/flink-dist/target/flink-1.10.0-bin/flink-1.10.0/log/flink-fanjiaxing-standalonesession-0-ZBMAC-C02VR1RWH.local.out Connected to the target VM, address: '127.0.0.1:63889', transport: 'socket' SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
通过调试发现,在绑定LogFactory的时候,由于slf4j的版本不匹配,造成使用默认的NOPLoggerFactory工厂生成logger对象。
该工厂isInfoEnabled总是返回false:
/** * Always returns false. * @return always false */ final public boolean isInfoEnabled() { // NOP return false; }
定位问题以后,添加flink-runtime中的slf4g的配置:
<!-- slf4j --> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>${slf4j.version}</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.1.11</version> </dependency> <!-- slf4j end -->
重新debug,出现Log4j的异常信息,
java.lang.NoClassDefFoundError: org/apache/log4j/Level
添加Log4j的依赖,
<dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.17</version> </dependency>
再次调试成功。
能看到StandaloneSessionClusterEntrypoint进程。