Flink安裝排坑指南


Flink安裝排坑指南

Flink依賴與java環境,請先安裝jdk,最好1.8以上,配置好JAVA_HOME環境變量

首先我是在window上安裝flink的,下載flink

下載之后解壓到任意目錄,進入bin目錄運行start-cluster.bat腳本啟動flink服務,但是Task Slots卻為0,查看log發現是TaskManager沒有初始化成功

image-20200314182933049

2020-03-14 18:28:37,888 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - TaskManager initialization failed.
org.apache.flink.configuration.IllegalConfigurationException: Failed to create TaskExecutorResourceSpec
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:72)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.startTaskManager(TaskManagerRunner.java:356)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:152)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:308)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerSecurely$2(TaskManagerRunner.java:322)
	at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerSecurely(TaskManagerRunner.java:321)
	at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:287)
Caused by: org.apache.flink.configuration.IllegalConfigurationException: The required configuration option Key: 'taskmanager.cpu.cores' , default: null (fallback keys: []) is not set
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkConfigOptionIsSet(TaskExecutorResourceUtils.java:90)
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.lambda$checkTaskExecutorResourceConfigSet$0(TaskExecutorResourceUtils.java:84)
	at java.util.Arrays$ArrayList.forEach(Unknown Source)
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkTaskExecutorResourceConfigSet(TaskExecutorResourceUtils.java:84)
	at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:70)
	... 7 more
2020-03-14 18:28:37,892 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
2020-03-14 18:28:37,892 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache

以為是沒有權限創建文件夾,用管理員權限試了也報一樣的錯

搞了半天也沒個結果,最后果斷放棄了window環境。

在vmware上弄了個centos7,在centos7上搭建了jdk和flink環境,推薦centos7最少給2G內存,不然后面啟動flink時會被內存不夠的問題折騰死。

flink最少需要192M的內存才能啟動,jdk1.8之后堆內存采用的是元空間,元空間是根據內存大小自動分配的。如果元空間給的太小,flink將會啟動不起來。

The derived from fraction jvm overhead memory (19.200mb (20132659 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead

在啟動flink時可以通過jvm參數指定元空間大小

  • -XX:MetaspaceSize,初始空間大小,達到該值就會觸發垃圾收集進行類型卸載,同時GC會對該值進行調整:如果釋放了大量的空間,就適當降低該值;如果釋放了很少的空間,那么在不超過MaxMetaspaceSize時,適當提高該值。

flink的cofig/flink-conf.yaml配置的jobmanager.heap.sizetaskmanager.memory.process.size也不能超過元空間大小。它們默認大小分別是:1024m和1568m

配置方法:env.java.opts: -XX:MetaspaceSize=512m,指定元空間為512m

如果電腦內存夠大的話,推薦還是直接給centos弄個2G吧,能省很多事。

運行./bin/start-cluster.sh,在linux上順利啟動,在win10上通過瀏覽器訪問

image-20200314203247067

win10上啟動不了不知道是不是因為權限太嚴格導致的,希望有路過的大佬能在評論區指點一二


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM