Ubuntu16.04安裝flink-1.10.0

本來想cdh集成flink，但是我的cdh版本為5.16.2，參考了下面的issue可能cdh版本太低，至少要cdh6

https://github.com/pkeropen/flink-parcel/issues

進行獨立安裝

wget https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.10.0/flink-1.10.0-bin-scala_2.11.tgz

安裝路徑

/home/lintong/software/apache/flink-1.10.0

/etc/profile添加，並source /etc/profile

#flink
export FLINK_HOME=/home/lintong/software/apache/flink-1.10.0
export PATH=${FLINK_HOME}/bin:$PATH

下載flink-shaded-hadoop-2-uber-2.7.5-7.0.jar包，放到flink的lib目錄下

wget https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-7.0/flink-shaded-hadoop-2-uber-2.7.5-7.0.jar

不然flink on yarn的時候會報

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more

啟動yarn-session

yarn-session.sh -n 3 -s 5 -jm 1024 -tm 4096 -d

yarn-seesion參數

  -n ： 指定TaskManager的數量；
  -d: 以分離模式運行；
  -id：指定yarn的任務ID；
  -j:Flink jar文件的路徑;
  -jm：JobManager容器的內存（默認值：MB）;
  -nl：為YARN應用程序指定YARN節點標簽;
  -nm:在YARN上為應用程序設置自定義名稱;
  -q:顯示可用的YARN資源（內存，內核）;
  -qu:指定YARN隊列;
  -s:指定TaskManager中slot的數量;
  -st:以流模式啟動Flink;
  -tm:每個TaskManager容器的內存（默認值：MB）;
  -z:命名空間，用於為高可用性模式創建Zookeeper子路徑;

去CDH上查看，第一個是正在運行，第二個是結束

去appliance id進到yarn的app頁面

 

再點擊ApplicationMaster進到Flink Dashboard頁面

再提交Flink任務到yarn上

lintong@master:~/software/apache/flink-1.10.0$ bin/flink run examples/batch/WordCount.jar

再查看Flink Dashboard

 

輸出

 

 

如果頁面中的Available task slot一直為0的話，運行任務報，那說明yarn的資源不夠

Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources.
	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:452)
	... 45 more
Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	... 25 more
Caused by: java.util.concurrent.TimeoutException
	... 23 more

去cdh上調整yarn.nodemanager.resource.memory-mb，容器內存，我這邊從2G調整成4G就可以運行flink任務