碼上快樂

相關內容    簡體    繁體

kerberos系列之hive認證配置

本文轉載自   查看原文   2020-03-27 22:15   5537    大數據安全系列

 大數據安全系列之hive的kerberos認證配置，其它系列鏈接如下

https://www.cnblogs.com/bainianminguo/p/12548076.html-----------安裝kerberos

https://www.cnblogs.com/bainianminguo/p/12548334.html-----------hadoop的kerberos認證

https://www.cnblogs.com/bainianminguo/p/12548175.html-----------zookeeper的kerberos認證

https://www.cnblogs.com/bainianminguo/p/12584732.html-----------hive的kerberos認證

https://www.cnblogs.com/bainianminguo/p/12584880.html-----------es的search-guard認證

https://www.cnblogs.com/bainianminguo/p/12639821.html-----------flink的kerberos認證

https://www.cnblogs.com/bainianminguo/p/12639887.html-----------spark的kerberos認證

一、安裝mysql

1、卸載mariadb

[root@cluster2-host1 yum.repos.d]# rpm -qa|grep mariadb
mariadb-libs-5.5.44-2.el7.centos.x86_64
[root@cluster2-host1 yum.repos.d]# rpm -e --nodeps mariadb-libs-5.5.44-2.el7.centos.x86_64
[root@cluster2-host1 yum.repos.d]# rpm -qa|grep mariadb

　　

2、創建mysql用戶

[root@cluster2-host1 yum.repos.d]# groupadd mysql
[root@cluster2-host1 yum.repos.d]# useradd mysql -g mysql

　

3、下載mysql安裝包

https://cdn.mysql.com//Downloads/MySQL-5.7/mysql-5.7.29-1.el7.x86_64.rpm-bundle.tar

　　

4、解壓安裝包

 

[root@cluster2-host1 data]# tar -xvf mysql-5.7.29-1.el7.x86_64.rpm-bundle.tar -C /usr/local/mysql/

[root@cluster2-host1 mysql]# ll
total 533048
-rw-r--r--. 1 7155 31415  27768112 Dec 19 03:12 mysql-community-client-5.7.29-1.el7.x86_64.rpm
-rw-r--r--. 1 7155 31415    318972 Dec 19 03:12 mysql-community-common-5.7.29-1.el7.x86_64.rpm
-rw-r--r--. 1 7155 31415   4085448 Dec 19 03:12 mysql-community-devel-5.7.29-1.el7.x86_64.rpm
-rw-r--r--. 1 7155 31415  47521016 Dec 19 03:12 mysql-community-embedded-5.7.29-1.el7.x86_64.rpm
-rw-r--r--. 1 7155 31415  23354680 Dec 19 03:12 mysql-community-embedded-compat-5.7.29-1.el7.x86_64.rpm
-rw-r--r--. 1 7155 31415 131015588 Dec 19 03:12 mysql-community-embedded-devel-5.7.29-1.el7.x86_64.rpm
-rw-r--r--. 1 7155 31415   2596180 Dec 19 03:12 mysql-community-libs-5.7.29-1.el7.x86_64.rpm
-rw-r--r--. 1 7155 31415   1353080 Dec 19 03:12 mysql-community-libs-compat-5.7.29-1.el7.x86_64.rpm
-rw-r--r--. 1 7155 31415 183618644 Dec 19 03:12 mysql-community-server-5.7.29-1.el7.x86_64.rpm
-rw-r--r--. 1 7155 31415 124193252 Dec 19 03:12 mysql-community-test-5.7.29-1.el7.x86_64.rpm

　　

5、rpm的方式安裝mysql

 

[root@cluster2-host1 mysql]# rpm -ivh mysql-community-common-5.7.29-1.el7.x86_64.rpm 
warning: mysql-community-common-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-common-5.7.29-1.e################################# [100%]
[root@cluster2-host1 mysql]# rpm -ivh mysql-community-libs-5.7.29-1.el7.x86_64.rpm 
warning: mysql-community-libs-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-libs-5.7.29-1.el7################################# [100%]
[root@cluster2-host1 mysql]# rpm -ivh mysql-community-client-5.7.29-1.el7.x86_64.rpm 
warning: mysql-community-client-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-client-5.7.29-1.e################################# [100%]
[root@cluster2-host1 mysql]# rpm -ivh mysql-community-server-5.7.29-1.el7.x86_64.rpm 
warning: mysql-community-server-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
error: Failed dependencies:
	net-tools is needed by mysql-community-server-5.7.29-1.el7.x86_64

 

　　

[root@cluster2-host1 mysql]# yum install net-tools -y


[root@cluster2-host1 mysql]# rpm -ivh mysql-community-server-5.7.29-1.el7.x86_64.rpm 
warning: mysql-community-server-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-server-5.7.29-1.e################################# [100%]

　　

6、啟動mysql

[root@cluster2-host1 mysql]# service mysqld start

　　

7、查看mysql的臨時密碼

[root@cluster2-host1 mysql]# grep "A temporary password" /var/log/mysqld.log
2020-03-02T07:59:38.098144Z 1 [Note] A temporary password is generated for root@localhost: ln/Ot4j-j#hQ
[root@cluster2-host1 mysql]#

　　

8、修改mysql的臨時密碼

set global validate_password_policy=0;
set global validate_password.length=1;
alter user user() identified by "123456";

　　

9、設置mysql支持遠程訪問

[root@cluster2-host1 conf]# mysql -u root -p
update user set host = '%' where user = 'root';

　　

10、在其它節點確認可以遠程訪問mysql即可

mysql -h 10.87.18.34 -p3306 -uroot -p

　　

二、安裝hive

1、解壓和重命名hive安裝路徑

 

 

 538  tar -zxvf apache-hive-1.2.0-bin.tar.gz -C /usr/local/
  539  cd /usr/local/
  540  ll
  541  mv apache-hive-1.2.0-bin/ hive

 

　　

2、修改hive的env文件

 

[root@cluster2-host1 conf]# pwd
/usr/local/hive/conf
[root@cluster2-host1 conf]# cp hive-env.sh.template hive-env.sh

 

　　

3、修改hive-env文件

export HIVE_HOME=/usr/local/hive
export HADOOP_HOME=/usr/local/hadoop
# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/usr/local/hive/conf
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export PATH=${HIVE_HOME}/bin:$PATH:$HOME/bin:

　　

4、修改hive的配置文件

 

vim hive-default.xml

    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>
   <property>
        <name>javax.jdo.option.ConnectionURL</name>mysql
        <value>jdbc:mysql://10.87.18.34:3306/hive?</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>

　　

5、安裝hive的mysql驅動

 

拷貝驅動到hive/lib目錄

 

  474  cd mysql-connector-java-5.1.48
  475  ll
  476  cp mysql-connector-java-5.1.48-bin.jar /usr/local/hive/lib/

 

　　

6、在mysql中創建hive的database

 

mysql> create database hive;
Query OK, 1 row affected (0.00 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| hive               |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

 

　　

7、拷貝mysql連接驅動到其他未按照hive的節點

 

[root@cluster2-host1 lib]# scp mysql-connector-java-5.1.48-bin.jar root@cluster2-host2:/usr/local/hive/lib/
mysql-connector-java-5.1.48-bin.jar                                                                                                                                                                                                         100%  983KB 983.4KB/s   00:00    
[root@cluster2-host1 lib]# scp mysql-connector-java-5.1.48-bin.jar root@cluster2-host3:/usr/local/hive/lib/
mysql-connector-java-5.1.48-bin.jar

 

　　

三、配置hive的kerberos配置

1、創建主體文件

kadmin.local:  addprinc hive/cluster2-host1

kadmin.local:  ktadd -norandkey -k /etc/security/keytab/hive.keytab hive/cluster2-host1

　　

2、拷貝秘鑰文件到hive的目錄

 

 scp /etc/security/keytab/hive.keytab /usr/local/hive/conf/

 

　　

3、修改hive的配置文件

 

 <property>
        <name>hive.server2.enable.doAs</name>
        <value>true</value>
  </property>
  <property>
        <name>hive.server2.authentication</name>
        <value>KERBEROS</value>
  </property>
  <property>
        <name>hive.server2.authentication.kerberos.principal</name>
        <value>hive/cluster2-host1@HADOOP.COM</value>
  </property>
  <property>
        <name>hive.server2.authentication.kerberos.keytab</name>
        <value>/usr/local/hive/conf/hive.keytab</value>
  </property>
  <property>
        <name>hive.server2.authentication.spnego.keytab</name>
        <value>/usr/local/hive/conf/hive.keytab</value>
  </property>
  <property>
        <name>hive.server2.authentication.spnego.principal</name>
        <value>hive/cluster2-host1@HADOOP.COM</value>
  </property>
  <property>
        <name>hive.metastore.sasl.enabled</name>
        <value>true</value>
  </property>
  <property>
        <name>hive.metastore.kerberos.keytab.file</name>
        <value>/usr/local/hive/conf/hive.keytab</value>
  </property>
  <property>
        <name>hive.metastore.kerberos.principal</name>
        <value>hive/cluster2-host1@HADOOP.COM</value>
  </property>

 

　　

4、修改hadoop的core-site.xml配置文件

 

      <property>
        <name>hadoop.proxyuser.hive.users</name>
        <value>*</value>
      </property>
      <property>
        <name>hadoop.proxyuser.hive.hosts</name>
        <value>*</value>
      </property>

 

　　

5、啟動hive

[root@cluster2-host1 hive]# nohup ./bin/hive --service metastore > metastore.log 2>&1 &
[1] 5637
[root@cluster2-host1 hive]# nohup ./bin/hiveserver2 > hive.log 2>&1 &
[2] 7361

　　

6、通過beeline的方式連接hive

 

[root@cluster2-host1 hive]#   ./bin/beeline -u "jdbc:hive2://cluster2-host1:10000/default;principal=hive/cluster2-host1@HADOOP.COM"
ls: cannot access /usr/local/spark/lib/spark-assembly-*.jar: No such file or directory
Connecting to jdbc:hive2://cluster2-host1:10000/default;principal=hive/cluster2-host1@HADOOP.COM
Connected to: Apache Hive (version 1.2.0)
Driver: Hive JDBC (version 1.2.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.0 by Apache Hive
0: jdbc:hive2://cluster2-host1:10000/default>

 

　　

7、登陸進去創建hive表

create database myhive;
CREATE TABLE student(id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' LINES TERMINATED BY '\n' STORED AS TEXTFILE;

　　

查看創建的表

0: jdbc:hive2://cluster2-host1:10000/default> show tables;
+------------+--+
|  tab_name  |
+------------+--+
| student    |
| test1      |
| test2      |
| test3      |
| test4      |
| test_user  |
+------------+--+

　　

8、檢查mysql有hive的元數據信息

 

 

 

 

9、准備本地數據

2014001 小王1
2014002 小李2
2014003 小明3
2014004 阿狗4
2014005 姚明5

　　

10、上傳數據

load data local inpath '/data/hive.txt' into table test1;

　　

11、查看hdfs上的 數據

[root@cluster2-host1 data]# hdfs dfs -ls /user/hive/warehouse/test1
Found 1 items
-rwxr-xr-x   2 hdfs supergroup        112 2020-03-05 04:55 /user/hive/warehouse/test1/hive.txt

　　

 


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 kerberos系列之flink認證配置 kerberos系列之spark認證配置 kerberos系列之zookeeper的認證配置 kerberos系列之hdfs&yarn認證配置 SSIS 系列 - kerberos認證 使用KERBEROS認證訪問Spark JDBC和Hive HDFS配置Kerberos認證[轉] java 鏈接hive（kerberos認證）實例 以及 錯誤點講解 java 通過jdbc 連接hive2 使用kerberos 認證 springboot 1.5.6 + jdbc使用kerberos認證方式連接hive
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM