2020雙十 阿里雲服務器ECS團購特惠鏈接
前提條件：需先安裝zookeeper與hadoop
zookeeper安裝
hadoop安裝
一、系統配置
3台虛擬機
hadoop1: 4g內存 2核 80g硬盤
hadoop2 1g內存 1核 8g硬盤
hadoop3: 1g內存 1核 8g硬盤
二、基礎概念
1、Flink Client
2、JobManager
3、TaskManager
三、flink集群搭建(on yarn)
集群搭建的主要內容為JobManager高可用配置
1、下載與解壓
(1)下載安裝包flink-1.10.0-bin-scala_2.12.tgz至hadoop1節點
curl -O http://apache.mirrors.hoobly.com/flink/flink-1.10.0/flink-1.10.0-bin-scala_2.12.tgz
(2)遠程傳輸到hadoop2、hadoop3節點
scp flink-1.10.0-bin-scala_2.12.tgz root@hadoop2:`pwd`
scp flink-1.10.0-bin-scala_2.12.tgz root@hadoop3:`pwd`
(3)解壓安裝包
tar zxvf flink-1.10.0-bin-scala_2.12.tgz
安裝結束
2、JobManager高可用配置
(1)yarn-site.xml
文件目錄：/usr/local/softwareinstall/hadoop-3.1.3/etc/hadoop/yarn-site.xml(hadoop安裝目錄下的yarn集群配置文件）
添加如下配置
<property>
<name>yarn.resourcemanager.am.max-attempts</name>
<value>4</value>
<description>
The maximum number of application master execution attempts.
</description>
</property>
(2)flink配置文件flink-conf.yaml
文件目錄：(/usr/local/softwareinstall/flink-1.10.0/conf/flink-conf.yaml)
高可用設置
high-availability: zookeeper
high-availability.zookeeper.quorum: hadoop1:2181,hadoop2:2181,hadoop3:2181
high-availability.storageDir: hdfs:///flink/recovery
high-availability.zookeeper.path.root: /flink
yarn.application-attempts: 10
JobManager與TaskManager配置
(3)添加flink環境變量/etc/profile
vim /etc/profile
#flink環境變量配置
export FLINK_HOME=/usr/local/softwareinstall/flink-1.10.0
export PATH=$PATH:$FLINK_HOME/bin
#flink on yarn配置
HADOOP_CONF_DIR=$HADOOP_HOME
export HADOOP_CLASSPATH=`hadoop classpath`
source /etc/profile #使環境變量生效
3、向yarn集群申請資源並提交任務
3.1 申請資源
(1)申請資源
bin/yarn-session.sh -nm wordCount -n 2
(2)查看申請的資源
使用命令查看
yarn application --list
yarn集群ui界面查看
yarn.webapp.ui2界面查看
3.2 提交wordCount計算作業
(1)提交作業到application id為application_1589204773282_0003的yarn session中
cd /usr/local/softwareinstall/flink-1.10.0
./bin/flink run -yid application_1589204773282_0003 ./examples/batch/WordCount.jar
(2)flink job ui界面觀察作業執行信息
參考：
(1)YARN Setup
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html
(2)JobManager High Availability (HA)
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/jobmanager_high_availability.html