大數據安全系列的其它文章
https://www.cnblogs.com/bainianminguo/p/12548076.html-----------安裝kerberos
https://www.cnblogs.com/bainianminguo/p/12548334.html-----------hadoop的kerberos認證
https://www.cnblogs.com/bainianminguo/p/12548175.html-----------zookeeper的kerberos認證
https://www.cnblogs.com/bainianminguo/p/12584732.html-----------hive的kerberos認證
https://www.cnblogs.com/bainianminguo/p/12584880.html-----------es的search-guard認證
https://www.cnblogs.com/bainianminguo/p/12639821.html-----------flink的kerberos認證
https://www.cnblogs.com/bainianminguo/p/12639887.html-----------spark的kerberos認證
今天的博客介紹大數據安全系列之flink的kerberos配置
一、flink安裝
1、解壓安裝包
tar -zxvf flink-1.8.0-bin-scala_2.11.tgz -C /usr/local/
2、重命名安裝目錄
[root@cluster2-host1 local]# mv flink-1.8.0/ flink
3、修改環境變量文件
export FLINK_HOME=/usr/local/flink
export PATH=${PATH}:${FLINK_HOME}/bin
[root@cluster2-host1 data]# source /etc/profile
[root@cluster2-host1 data]# echo $FLINK_HOME
/usr/local/flink
4、修改flink的配置文件
[root@cluster2-host1 conf]# vim flink-conf.yaml [root@cluster2-host1 conf]# pwd /usr/local/flink/conf
jobmanager.rpc.address: cluster2-host1
修改slaver文件
[root@cluster2-host1 conf]# vim slaves [root@cluster2-host1 conf]# pwd /usr/local/flink/conf
cluster2-host2 cluster2-host3
修改master文件
[root@cluster2-host1 bin]# cat /usr/local/flink/conf/masters cluster2-host1
修改yarn-site.xml文件
<property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>5</value> </property>
5、創建flink用戶
[root@cluster2-host3 hadoop]# useradd flink -g flink [root@cluster2-host3 hadoop]# passwd flink Changing password for user flink. New password: BAD PASSWORD: The password fails the dictionary check - it is based on a dictionary word Retype new password:
6、修改flink安裝目錄的屬主和屬組
[root@cluster2-host3 hadoop]# chown -R flink:flink /usr/local/flink/
7、啟動flink驗證安裝步驟
[root@cluster2-host1 bin]# ./start-cluster.sh Starting cluster. [INFO] 1 instance(s) of standalonesession are already running on cluster2-host1. Starting standalonesession daemon on host cluster2-host1. Starting taskexecutor daemon on host cluster2-host2. Starting taskexecutor daemon on host cluster2-host3.
檢查進程
[root@cluster2-host1 bin]# jps 10400 Secur 30817 StandaloneSessionClusterEntrypoint 12661 ResourceManager 12805 NodeManager 4998 QuorumPeerMain 30935 Jps 2631 NameNode
登陸頁面
http://10.87.18.34:8081/#/overview
關閉flink，上面的standalone的啟動方法，下面啟動flink-session模式
拷貝hadoop的依賴包到flink的lib目錄
scp flink-shaded-hadoop2-uber-2.7.5-1.8.0.jar /usr/local/flink/lib/
啟動flink-session模式
./yarn-session.sh -n 2 -s 6 -jm 1024 -tm 1024 -nm test -d
檢查yanr的頁面
二、配置flink的kerberos的配置
1、創建flink的kerberos認證主體文件
kadmin.local: addprinc flink/cluster2-host1 kadmin.local: addprinc flink/cluster2-host2 kadmin.local: addprinc flink/cluster2-host3
kadmin.local: ktadd -norandkey -k /etc/security/keytab/flink.keytab flink/cluster2-host1 kadmin.local: ktadd -norandkey -k /etc/security/keytab/flink.keytab flink/cluster2-host2 kadmin.local: ktadd -norandkey -k /etc/security/keytab/flink.keytab flink/cluster2-host3
2、拷貝keytab文件到其它節點
[root@cluster2-host1 bin]# scp /etc/security/keytab/flink.keytab root@cluster2-host2:/usr/local/flink/ flink.keytab 100% 1580 1.5KB/s 00:00 [root@cluster2-host1 bin]# scp /etc/security/keytab/flink.keytab root@cluster2-host3:/usr/local/flink/ flink.keytab
3、修改flink的配置文件
security.kerberos.login.use-ticket-cache: true security.kerberos.login.keytab: /usr/local/flink/flink.keytab security.kerberos.login.principal: flink/cluster2-host3 yarn.log-aggregation-enable: true
4、啟動yarn-session，看到如下操作，則配置完成
flink@cluster2-host1 bin]$ ./yarn-session.sh -n 2 -s 6 -jm 1024 -tm 1024 -nm flink5 -d 2020-03-05 02:42:23,706 INFO org.apache.hadoop.security.UserGroupInformation - Login successful for user flink/cluster2-host1 using keytab file /usr/local/flink/flink.keytab
查看頁面
檢查進程
[root@cluster2-host1 sbin]# jps 6118 ResourceManager 15975 NameNode 22472 -- process information unavailable 6779 NodeManager 23483 YarnSessionClusterEntrypoint 24717 Master 9790 QuorumPeerMain 25534 Jps 20239 Secur
5、flink的kerberos的配置完成