1 環境描述
三台hadoop集群,分別是master、slave1和slave2。下面是這三台機器的軟件分布:
- master:NameNode、ZK、HiveMetaSotre、HiveServer2、SentryServer
- slave1:DataNode、ZK
- slave2:DataNode、ZK
2 軟件需求
- MySql
- mysql-jdbc.jar:mysql-connector-java-5.1.38-bin.jar
- Hadoop
- Hive1.1.0:apache-hive-1.1.0-bin.tar.gz
- Sentry1.6.0:apache-sentry-1.6.0-incubating-bin.tar.gz
注意:這里的Sentry1.6.0版本對應的是Hive1.1.0版本,目前還不支持Hive1.2及以上版本。
3 軟件安裝
3.1 MySql安裝
- 在線安裝:
sudo apt-get install mysql-server
。 - 修改文件:
vim /etc/mysql/my.cnf
,內容如下:
#bind-address = 127.0.0.1 bind-address = 0.0.0.0
- 重啟mysql:
service mysql restart
3.2 Hadoop安裝
此處省略
3.2 Hive安裝
此處省略
創建hive的元數據mysql數據庫 create database hive; CREATE USER hive IDENTIFIED BY 'hive'; GRANT all ON hive.* TO hive@'%' IDENTIFIED BY'hive'; flush privileges;
3.3 Sentry安裝
1. 解壓安裝
tar -zxvf apache-sentry-1.6.0-incubating-bin.tar.gz -C /opt
2. 修改sentry的配置文件(sentry-site.xml)
mv apache-sentry-1.6.0-incubating-bin apache-sentry-1.6.0 cp mysql-connector-java-5.1.38-bin.jar /opt/apache-sentry-1.6.0/lib cd /opt/apache-sentry-1.6.0/conf vim sentry-site.xml
3. 修改sentry-sitem.xml文件
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>sentry.service.admin.group</name>
<value>admin</value>
</property>
<property>
<name>sentry.service.allow.connect</name>
<value>hive,admin</value>
</property>
<property>
<name>sentry.service.reporting</name>
<value>JMX</value>
</property>
<property>
<name>sentry.service.server.rpc-address</name>
<value>192.168.70.110</value>
</property>
<property>
<name>sentry.service.server.rpc-port</name>
<value>8038</value>
</property>
<property>
<name>sentry.store.group.mapping</name>
<value>org.apache.sentry.provider.common.HadoopGroupMappingService</value>
</property>
<property>
<name>sentry.hive.server</name>
<value>server1</value>
</property>
<!-- 配置Webserver -->
<property>
<name>sentry.service.web.enable</name>
<value>true</value>
</property>
<property>
<name>sentry.service.web.port</name>
<value>51000</value>
</property>
<property>
<name>sentry.service.web.authentication.type</name>
<value>NONE</value>
</property>
<property>
<name>sentry.service.web.authentication.kerberos.principal</name>
<value> </value>
</property>
<property>
<name>sentry.service.web.authentication.kerberos.keytab</name>
<value> </value>
</property>
<!--配置認證-->
<property>
<name>sentry.service.security.mode</name>
<value>none</value>
</property>
<property>
<name>sentry.service.server.principal</name>
<value> </value>
</property>
<property>
<name>sentry.service.server.keytab</name>
<value> </value>
</property>
<!-- 配置Jdbc -->
<property>
<name>sentry.store.jdbc.url</name>
<value>jdbc:mysql://192.168.70.110:3306/sentry</value>
</property>
<property>
<name>sentry.store.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>sentry.store.jdbc.user</name>
<value>sentry</value>
</property>
<property>
<name>sentry.store.jdbc.password</name>
<value>sentry</value>
</property>
</configuration>
4. 配置環境變量
編輯vim /etc/profile
export SENTRY_HOME=/opt/apache-sentry-1.6.0 export PATH=${SENTRY_HOME}/bin:$PATH
使生效source /etc/profile
5. 在mysql中創建sentry數據庫
create database sentry; CREATE USER sentry IDENTIFIED BY 'sentry'; GRANT all ON sentry.* TO sentry@'%' IDENTIFIED BY 'sentry'; flush privileges;
6. 將mysql-jdbc的jar包拷貝到sentry/lib目錄下
cp mysql-connector-java-5.1.38-bin.jar ${SENTRY_HOME}/lib
7. 替換hadoop中jline的jar包
rm ${HADOOP_HOME}/share/hadoop/yarn/lib/jline-0.9.94.jar cp ${SENTRY_HOME}/lib/jline-2.12.jar ${HADOOP_HOME}/share/hadoop/yarn/lib
8. 初始化sentry數據庫
sentry --command schema-tool --conffile ${SENTRY_HOME}/conf/sentry-site.xml --dbType mysql --initSchema
9. 啟動sentry服務
sentry --command service --conffile ${SENTRY_HOME}/conf/sentry-site.xml
4 Hive集成Sentry
1. 修改hive-site.xml配置文件
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.sentry.conf.url</name>
<value>file:///opt/apache-hive-1.1.0/conf/sentry-site.xml</value>
</property>
<!-- 配置開關控制列訪問權限 -->
<property>
<name>hive.stats.collect.scancols</name>
<value>true</value>
</property>
<!-- Hive Metastore集成Sentry -->
<property>
<name>hive.metastore.pre.event.listeners</name>
<value>org.apache.sentry.binding.metastore.MetastoreAuthzBinding</value>
</property>
<property>
<name>hive.metastore.event.listeners</name>
<value>org.apache.sentry.binding.metastore.SentryMetastorePostEventListener</value>
</property>
<!-- Hive Server2集成Sentry -->
<property>
<name>hive.server2.session.hook</name>
<value>org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook</value>
</property>
<property>
<name>hive.security.authorization.task.factory</name>
<value>org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl</value>
</property>
<!-- 設置mysql-jdbc -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.70.110:3306/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
</property>
</configuration>
2. 修改sentry-site.xml配置文件
注意:該文件是在hive/conf目錄下的,不要和sentry/conf目錄下的sentry-site.xml文件混淆!
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>sentry.service.client.server.rpc-address</name>
<value>192.168.70.110</value>
</property>
<property>
<name>sentry.service.client.server.rpc-port</name>
<value>8038</value>
</property>
<property>
<name>sentry.service.client.server.rpc-connection-timeout</name>
<value>200000</value>
</property>
<!--配置認證-->
<property>
<name>sentry.service.security.mode</name>
<value>none</value>
</property>
<property>
<name>sentry.service.server.principal</name>
<value> </value>
</property>
<property>
<name>sentry.service.server.keytab</name>
<value> </value>
</property>
<property>
<name>sentry.provider</name>
<value>org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider</value>
</property>
<property>
<name>sentry.hive.provider.backend</name>
<value>org.apache.sentry.provider.db.SimpleDBProviderBackend</value>
</property>
<property>
<name>sentry.metastore.service.users</name>
<value>hive</value>
<!--queries made by hive user (beeline) skip meta store check-->
</property>
<property>
<name>sentry.hive.server</name>
<value>server1</value>
</property>
<property>
<name>sentry.hive.testing.mode</name>
<value>true</value>
</property>
</configuration>
3. 拷貝sentry/lib下的jar包到hive/lib目錄下
cp ${SENTRY_HOME}/lib/sentry*.jar ${HIVE_HOME}/lib cp ${SENTRY_HOME}/lib/shiro-*.jar ${HIVE_HOME}/lib
注意:shiro-*.jar一定也要拷貝過去,否則會報錯。
4. 修改Hive在HDFS上的文件權限
hadoop fs -chown -R hive:hive /user/hive/warehouse hadoop fs -chmod -R 770 /user/hive/warehouse
由於HDFS上文件是hive:hive權限,所以linux用戶上能
5. 啟動Hive Metastore服務
hive --service metastore -hiveconf hive.root.logger=DEBUG,console
6. 啟動Hive Server2服務
hive --service hiveserver2 -hiveconf hive.root.logger=DEBUG,console
5 Hive權限驗證
5.1 准備測試數據
- 創建測試數據文件:
vim /tmp/events.csv
,內容如下:
10.1.2.3,US,android,createNote 10.200.88.99,FR,windows,updateNote 10.1.2.3,US,android,updateNote 10.200.88.77,FR,ios,createNote 10.1.4.5,US,windows,updateTag
- 然后,在hive中運行下面sql語句(注意:以hive用戶啟動hive命令):
create database sensitive;
create table sensitive.events (ip STRING, country STRING, client STRING, action STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
load data local inpath '/tmp/events.csv' overwrite into table sensitive.events;
create database filtered;
create view filtered.events as select country, client, action from sensitive.events;
create view filtered.events_usonly as select * from filtered.events where country = 'US';
- 在master上通過 beeline 連接 hiveserver2,運行下面命令創建角色和組:
beeline -u "jdbc:hive2://master:10000/" -n admin -p admin -d org.apache.hive.jdbc.HiveDriver
- 在beelin中執行下面的 sql 語句創建 role、group等:
create role admin_role;
GRANT ALL ON SERVER server1 TO ROLE admin_role;
GRANT ROLE admin_role TO GROUP admin;
create role test_role;
GRANT ALL ON DATABASE filtered TO ROLE test_role;
use sensitive;
GRANT SELECT(ip) on TABLE sensitive.events TO ROLE test_role;
GRANT ROLE test_role TO GROUP test;
上面創建了兩個角色:
- admin_role,具有管理員權限,可以讀寫所有數據庫,並授權給 admin 和 hive 組(對應操作系統上的組)
- test_role,只能讀寫 filtered 數據庫,和只能讀取sensitive庫中events 表中的ip字段,並授權給 test 組。
因為系統上沒有test用戶和組,所以需要手動創建:useradd test
5.2 測試
5.2.1 測試admin_role角色
使用admin用戶訪問beelin:
beeline -u "jdbc:hive2://master:10000/" -n admin -p admin -d org.apache.hive.jdbc.HiveDriver
admin用戶屬於admin_role組,具有管理員權限,可以查看所有角色:
0: jdbc:hive2://master:10000/> show roles; +-------------+--+ | role | +-------------+--+ | admin_role | | test_role | +-------------+--+ 2 rows selected (0.251 seconds)
查看所有權限:
0: jdbc:hive2://master:10000/> show grant role admin_role; +-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+ | database | table | partition | column | principal_name | principal_type | privilege | grant_option | grant_time | grantor | +-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+ | * | | | | admin_role | ROLE | * | false | 1461507543582000 | -- | +-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+ 1 row selected (0.111 seconds) 0: jdbc:hive2://master:10000/> show grant role test_role; +------------+---------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+ | database | table | partition | column | principal_name | principal_type | privilege | grant_option | grant_time | grantor | +------------+---------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+ | sensitive | events | | ip | test_role | ROLE | select | false | 1461558337008000 | -- | | filtered | | | | test_role | ROLE | * | false | 1461557354579000 | -- | +------------+---------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+ 2 rows selected (0.111 seconds)
hive用戶可以查看所有數據庫、訪問所有表:
0: jdbc:hive2://master:10000/> show databases; +----------------+--+ | database_name | +----------------+--+ | default | | filtered | | sensitive | +----------------+--+ 3 rows selected (0.142 seconds) 0: jdbc:hive2://master:10000/> use filtered; No rows affected (0.144 seconds) 0: jdbc:hive2://master:10000/> select * from filtered.events; +-----------------+----------------+----------------+--+ | events.country | events.client | events.action | +-----------------+----------------+----------------+--+ | US | android | createNote | | FR | windows | updateNote | | US | android | updateNote | | FR | ios | createNote | | US | windows | updateTag | +-----------------+----------------+----------------+--+ 5 rows selected (0.458 seconds) 0: jdbc:hive2://master:10000/> select * from sensitive.events; +---------------+-----------------+----------------+----------------+--+ | events.ip | events.country | events.client | events.action | +---------------+-----------------+----------------+----------------+--+ | 10.1.2.3 | US | android | createNote | | 10.200.88.99 | FR | windows | updateNote | | 10.1.2.3 | US | android | updateNote | | 10.200.88.77 | FR | ios | createNote | | 10.1.4.5 | US | windows | updateTag | +---------------+-----------------+----------------+----------------+--+ 5 rows selected (0.731 seconds)
5.2.2 測試test_role角色
使用test用戶訪問beeline:
beeline -u "jdbc:hive2://master:10000/" -n test -p test -d org.apache.hive.jdbc.HiveDriver
test用戶不是管理員,是不能查看所有角色的:
0: jdbc:hive2://master:10000/> show roles; ERROR : Error processing Sentry command: Access denied to test.Please grant admin privilege to test. Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. SentryAccessDeniedException: Access denied to test (state=08S01,code=1)
test用戶可以列出所有數據庫:
0: jdbc:hive2://master:10000/> show databases; +----------------+--+ | database_name | +----------------+--+ | default | | filtered | | sensitive | +----------------+--+
test用戶可以filtered庫:
0: jdbc:hive2://master:10000/> use filtered; No rows affected (0.256 seconds) 0: jdbc:hive2://master:10000/> select * from events; +-----------------+----------------+----------------+--+ | events.country | events.client | events.action | +-----------------+----------------+----------------+--+ | US | android | createNote | | FR | windows | updateNote | | US | android | updateNote | | FR | ios | createNote | | US | windows | updateTag | +-----------------+----------------+----------------+--+ 5 rows selected (0.199 seconds)
test用戶只能查看sensitive庫中events表中的ip字段,其他的都都無法查看:
0: jdbc:hive2://master:10000/> use sensitive; No rows affected (0.112 seconds) 0: jdbc:hive2://master:10000/> select * from events; Error: Error while compiling statement: FAILED: SemanticException No valid privileges User test does not have privileges for QUERY (state=42000,code=40000) 0: jdbc:hive2://master:10000/> select ip from events; +---------------+--+ | ip | +---------------+--+ | 10.1.2.3 | | 10.200.88.99 | | 10.1.2.3 | | 10.200.88.77 | | 10.1.4.5 | +---------------+--+ 5 rows selected (0.176 seconds)