通過jdbc連接hive

目前使用hive的方式主要有2種：

a、僅把hive作為一個可以對文件使用sql查詢的工具來使用，即常見的在hive客戶端上，指定hive的metastore和hive的數據倉庫目錄（即hdfs目錄），然后指定hadoop的目錄以及相應執行的jar包即可，在命令行下調用：

mammut@classb-ds-bigdata16:~/apache-hive-1.2.1-bin$ bin/hive
Logging initialized using configuration in /home/mammut/apache-hive-0.13.1-bin/conf/hive-log4j.properties.self
Hive history file=/tmp/mammut/hive_job_log_21e28d9e-7756-402e-b0d2-242f4ceef095_1602622773.txt
hive> 

b、將hive作為一個可以提供SQL接口的數據庫來使用，這樣可以通過jdbc連接

這里就是對第二種方式做一下整理和記錄，在實現的過程中，遇到了一些問題，解決的過程中也順帶學習了一些概念和方法。

 

分解成各個子模塊后，主要是下面幾點：

1、搭建hiveserver2服務器

2（可選）、認證方式為kerberos認證，則需要配置相應的kerberos

3、代碼實現：

a、認證kerberos

b、建立jdbc連接

4、驗證結果

 

一、搭建hiveserver2服務器：

a、分別設置hiveclient中的hive-site.xml中：warehouse路徑、jdbc（metastore）為mysql以及相應連接方式、用戶名密碼    ---這幾個是hive-client搭建必須的配置

b、設置hiveserver2的配置，配置認證方式為kerberos，keytab為一個有代理權限的keytab和相應principal（這里選擇代理權限的keytab，1是因為hiveserver2是個代理服務器，2是為了方便多個不同用戶通過jdbc連接時，可以通過代理擁護代理執行，否則一個普通用戶訪問會有如下錯誤）

16/03/02 17:32:33 [main]: ERROR transport.TSaslTransport: SASL negotiation failure  
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]  
     at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)  
     at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)  
     at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)  
     at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)  
     at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)  
     at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)  
     at java.security.AccessController.doPrivileged(Native Method)  

c、設置hiveserver2的port和host(host設置成0.0.0.0，來接收未知來源的ip):

<property>
  <name>hive.server2.thrift.port</name>
  <value>10000</value>
  <description>Port number of HiveServer2 Thrift interface.
  Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT</description>
</property>

<property>
  <name>hive.server2.thrift.bind.host</name>
  <value>0.0.0.0</value>
  <description>Bind host on which to run the HiveServer2 Thrift interface.
  Can be overridden by setting $HIVE_SERVER2_THRIFT_BIND_HOST</description>
</property>

測試hiveserver2是否可用：

 

nohup ./bin/hive --service hiveserver2  &
./bin/beeline
~/apache-hive-1.2.1-bin$ bin/beeline 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/mammut/spark-1.5.2-SNAPSHOT-bin-2.4.0/lib/spark-assembly-1.5.2-SNAPSHOT-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/mammut/hadoop-binjiang/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/mammut/spark-1.5.2-SNAPSHOT-bin-2.4.0/lib/spark-assembly-1.5.2-SNAPSHOT-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/mammut/hadoop-binjiang/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Beeline version 1.2.1 by Apache Hive
beeline> !connect jdbc:hive2://10.165.139.121:10000/mammut_auto_test;principal=mammut/[e-mail];User=;Password=;
Connecting to jdbc:hive2://10.165.139.121:10000/mammut_auto_test;principal=mammut/[e-mail];User=;Password=;
Enter username for jdbc:hive2://10.165.139.121:10000/mammut_auto_test;principal=mammut/[e-mail];User=;Password=;: 
Enter password for jdbc:hive2://10.165.139.121:10000/mammut_auto_test;principal=mammut/[e-mail];User=;Password=;: 
Connected to: Apache Hive (version 1.2.1)
Driver: Hive JDBC (version 1.2.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://10.165.139.121:10000/mammut_a> 
0: jdbc:hive2://10.165.139.121:10000/mammut_a> 
0: jdbc:hive2://10.165.139.121:10000/mammut_a> show tables;
+-------------+--+
|  tab_name   |
+-------------+--+
| auto_test1  |
+-------------+--+
1 row selected (0.145 seconds)
0: jdbc:hive2://10.165.139.121:10000/mammut_a>

至此，hiveserver2搭建完成

 

遇到的問題：

遇到的最大的問題，是hiveserver2啟動時報錯，這個問題耽誤了一段時間，后來發現啥hive自身的一個bug，在hive_0.14版本就修復了，我用的是0.13，因此版本換到1.2.1后就可以正常啟動了（ https://issues.apache.org/jira/browse/HIVE-7620）

2016-03-02 17:02:22,602 ERROR [Thread-6]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(93)) - Error:   
java.lang.NoSuchFieldError: SASL_PROPS  
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S.getHadoopSaslProperties(HadoopThriftAuthBridge20S.java:126)  
        at org.apache.hive.service.auth.HiveAuthFactory.getSaslProperties(HiveAuthFactory.java:116)  
        at org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(HiveAuthFactory.java:133)  
        at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:43)  
        at java.lang.Thread.run(Thread.java:701)  
2016-03-02 17:02:22,605 INFO  [Thread[Thread-7,5,main]]: delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:updateCurrentKey(  
222)) - Updating the current master key for generating delegation tokens  
2016-03-02 17:02:22,612 INFO  [Thread-3]: server.HiveServer2 (HiveStringUtils.java:run(623)) - SHUTDOWN_MSG:   
/************************************************************  
SHUTDOWN_MSG: Shutting down HiveServer2 at   
************************************************************/ 

 

2、代碼實現kerberos認證：

在早期配置認證的時候，出現如下報錯：

java.io.IOException: Login failure for qatest/[e-mail] from keytab D:\00Tools\keytab\interface\qatest.keytab
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:695)
    at test.common.AuthKrb5.authKrb5(AuthKrb5.java:23)
    at test.common.BaseHive.<init>(BaseHive.java:21)
    at test.common.DataCheck.checkMysqlHive(DataCheck.java:23)
    at test.verify.CheckRdsHiveTest.checkMsqToHive(CheckRdsHiveTest.java:24)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:696)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:882)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1189)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
    at org.testng.TestRunner.privateRun(TestRunner.java:767)
    at org.testng.TestRunner.run(TestRunner.java:617)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
    at org.testng.SuiteRunner.run(SuiteRunner.java:254)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
    at org.testng.TestNG.run(TestNG.java:1057)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:125)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException:

 

看路徑等都是正確的，將配置放到linux上去跑ok，懷疑是krb5的配置沒有加載上，於是谷歌之，在IBM的知識社區中找到了可能的原因： JVM啟動時沒有加載krb5的配置

http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/rsec_SPNEGO_config_krb5.html?lang=zh

（kerberos的配置也可以參照這篇文章）

加上參數

System.setProperty("java.security.krb5.conf", Config.KRB5CONFIG_PATH);

就可以正常work了

 PS：由於后續還要寫操作hdfs的代碼，也是通過kerberos認證，故把kerberos認證這塊當做一個單獨的類去提煉出來，也便於后期維護，代碼如下：

public static void authKrb5(){
//設置jvm啟動時krb5的讀取路徑參數
System.setProperty("java.security.krb5.conf", Config.KRB5CONFIG_PATH);
//配置kerberos認證
Configuration conf = new Configuration();
conf.setBoolean("hadoop.security.authorization", true);
conf.set("hadoop.security.authentication", "kerberos");
//        System.out.println(System.getProperty("java.security.krb5.conf"));
UserGroupInformation.setConfiguration(conf);
        try {
            UserGroupInformation.loginUserFromKeytab("qatest/[e-mail]", "D:\\00Tools\\keytab\\interface\\qatest.keytab");
} catch (IOException e) {
            e.printStackTrace();
}
//        System.out.println("Succeeded in authenticating through Kerberos!");
}

3、代碼實現hive的jdbc連接

這個就不贅述了，由於經常調用，單獨使用一個類來維護，還有一些常用方法的封裝：

protected Statement stmt;
    protected Connection con;
    private static String driverName = "org.apache.hive.jdbc.HiveDriver";

//    public static void main(String[] args) throws SQLException{
public BaseHive() {
//認證kerberos
AuthKrb5 authentkrb5 = new AuthKrb5();
authentkrb5.authKrb5();

        try {
            Class.forName(driverName);
} catch (ClassNotFoundException e) {
            e.printStackTrace();
System.exit(1);
}

//連接hiveserver2
try{
con = DriverManager.getConnection("jdbc:hive2://"+Config.HIVE2_URL+"/mammut_auto_test;principal=mammut/[e-mail];User=;Password=;", "", "");
stmt = con.createStatement();
}catch (SQLException e){
            e.printStackTrace();
}
    }