hadoop系列 第二坑： hive hbase關聯表問題

關鍵詞：

hive創建表卡住了

創建hive和hbase關聯表卡住了

 

其實針對這一問題在info級別的日志下是看出哪里有問題的（為什么只能在debug下才能看見呢，不太理解開發者的想法）。

以調試模式進入hive客戶端：

hive --hiveconf hive.root.logger=DEBUG,console

執行創建關聯表的語句

create table IF NOT EXISTS event_content_temp
(id string,pack string,activity string,title string, sx string, dx string, num bigint) 
stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
with serdeproperties ("hbase.columns.mapping" = ":key,cf:pack,cf:activity,cf:title,cf:sx,cf:dx,cf:num") 
tblproperties ("hbase.table.name" = "event_content_temp");

顯示錯誤日志：

15/06/30 09:20:00 INFO util.RetryCounter: Sleeping 4000ms before retry #2...
15/06/30 09:20:00 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
15/06/30 09:20:00 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
15/06/30 09:20:00 DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:663)
        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:430)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:189)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1164)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1104)
15/06/30 09:20:00 DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:674)
        at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:438)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1164)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1104)
Interrupting... Be patient, this might take some time.
15/06/30 09:20:01 INFO CliDriver: Interrupting... Be patient, this might take some time.

很顯然是zookeeper的問題，可是在ClouderaManager中已經設置了呀，核對之后發現只有Hive Service Configuration Safety Valve for hive-site.xml里加了屬性

<property>
  <name>hive.aux.jars.path</name>  
  <value>
         file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.7.1.jar,
         file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hbase.jar,
         file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/zookeeper.jar
  </value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-xxx.cn-north-1.compute.internal:2181,ip-xxxx.cn-north-1.compute.internal:2181,ip-xxxx-112.cn-north-1.compute.internal:2181,ip-xxxx-108.cn-north-1.compute.internal:2181,ip-xxxx111.cn-north-1.compute.internal:2181</value>
</property>

Hive Client Configuration Safety Valve for hive-site.xml里沒有添加，這里很好理解，hive服務里配置了hbase的zk，但是hive客戶端沒有配置（hive-site.xml），建議二者配置一致！

配置完之后，對於關聯表的操作都可以順利完成。

 

總結：建議hive服務端和客戶端的hive-site.xml一樣

另外，我建了個QQ群：305994766，希望對大數據、算法研發、系統架構感興趣的朋友能夠加入進來，大家一起學習，共同進步（進群請說明自己的公司-職業-昵稱）