hive讀取hbase數據

簡單的例子看這個:https://blog.csdn.net/dominic_tiger/article/details/70237542 和 https://blog.csdn.net/xiewenbo/article/details/24627323

之前創建時報錯

FAILED: Error in metadata: java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hadoop.hive.hbase.HBaseSerDe: columns has 3 elements while hbase.columns.mapping has 4 elements (counting the key if implicit))

解決方法:

在創建hive/hbase相關聯的表時，hbase表結構默認會有一個字段key，如果沒有一個顯示的字段'key'那么在創建表的進修，會自己創建，這樣hive對應的表就會出現問題，所以在hive對應的表里一定要加上key這個字段，為了避免這個問題，在hbase表結構里可以顯示的添加'key'字段，這樣不容易出問題。

CREATE TABLE hperson1(key string,id string, name string,email string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf0:id,cf1:name,cf2:email") TBLPROPERTIES ("hbase.table.name" = "hbperson1");

或者：

CREATE TABLE hperson1(key string,id string, name string,email string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf0:id,cf1:name,cf2:email") TBLPROPERTIES ("hbase.table.name" = "hbperson1");

 

1.-------------------------------------
 hive> CREATE EXTERNAL TABLE tm05(key int,ROWKEY string ,BUS_NO int,CITY_NO int,DEPT_NO int,DOWN_PASSENGER int,FILA_NO int,GROUP_NO int,
  INS_TIME string, IS_UP_DOWN int ,LABEL_NO int ,LINE_NO  string ,MACH_NO int,SITE_TIME string ,UP_PASSENGER int)
    > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    > WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:ROWKEY,cf1:BUS_NO,cf1:CITY_NO,cf1:DEPT_NO,cf1:DOWN_PASSENGER,cf1:FILA_NO,cf1:GROUP_NO,
    cf1:INS_TIME,cf1:IS_UP_DOWN,cf1:LABEL_NO,cf1:LINE_NO,cf1:MACH_NO,cf1:SITE_TIME,cf1:UP_PASSENGER")
    > TBLPROPERTIES ("hbase.table.name" = "tm_temp05", "hbase.mapred.output.outputtable" = "tm_temp05");
2.--------------------------------------------------------------------------------------------------------
     CREATE EXTERNAL TABLE tm01(key int,ROWKEY string ,BUS_NO int,CITY_NO int,DEPT_NO int,DOWN_PASSENGER int,FILA_NO int,GROUP_NO int,
     INS_TIME string, IS_UP_DOWN int ,LABEL_NO int ,LINE_NO  string ,MACH_NO int,SITE_TIME string ,UP_PASSENGER int)
    > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    >  WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:ROWKEY,cf1:BUS_NO,cf1:CITY_NO,cf1:DEPT_NO,cf1:DOWN_PASSENGER,cf1:FILA_NO,cf1:GROUP_NO,
    cf1:INS_TIME,cf1:IS_UP_DOWN,cf1:LABEL_NO,cf1:LINE_NO,cf1:MACH_NO,cf1:SITE_TIME,cf1:UP_PASSENGER") 
    > TBLPROPERTIES ("hbase.table.name" = "tm_temp05", "hbase.mapred.output.outputtable" = "tm_temp05");
3.---------------------------------------------------------------------------------------------------------------------
     CREATE EXTERNAL TABLE tm02(key string,ROWKEY string ,BUS_NO int,CITY_NO int,DEPT_NO int,DOWN_PASSENGER int,FILA_NO int,GROUP_NO int,
     INS_TIME string, IS_UP_DOWN int ,LABEL_NO int ,LINE_NO  string ,MACH_NO int,SITE_TIME string ,UP_PASSENGER int)
    > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    >  WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:ROWKEY,cf1:BUS_NO,cf1:CITY_NO,cf1:DEPT_NO,cf1:DOWN_PASSENGER,cf1:FILA_NO,cf1:GROUP_NO,
    cf1:INS_TIME,cf1:IS_UP_DOWN,cf1:LABEL_NO,cf1:LINE_NO,cf1:MACH_NO,cf1:SITE_TIME,cf1:UP_PASSENGER") 
    > TBLPROPERTIES ("hbase.table.name" = "tm_temp05", "hbase.mapred.output.outputtable" = "tm_temp05");

方案1和方案2結果都一樣

方案3:

總結：在必須要使用一個無用列（不使用無用列key會報錯），建議使用方案1或方案2. 

上面創建的表在hive里是外部表，數據沒有存到hadoop的hdfs上。下面創建將外部表的數據放到hadoop的hdfs上

A.//---創建表
CREATE  TABLE mt01(ROWKEY string ,BUS_NO int,CITY_NO int,DEPT_NO int,DOWN_PASSENGER int,FILA_NO int,GROUP_NO int,
     INS_TIME string, IS_UP_DOWN int ,LABEL_NO int ,LINE_NO  string ,MACH_NO int,SITE_TIME string ,UP_PASSENGER int)
B.//---將查詢的數據放到上面創建表里
INSERT INTO  mt01 (ROWKEY,BUS_NO,CITY_NO,DEPT_NO,DOWN_PASSENGER,FILA_NO,GROUP_NO,
     INS_TIME, IS_UP_DOWN ,LABEL_NO,LINE_NO,MACH_NO,SITE_TIME,UP_PASSENGER)  ( select ROWKEY,BUS_NO,CITY_NO,DEPT_NO,DOWN_PASSENGER,FILA_NO,GROUP_NO,
     INS_TIME, IS_UP_DOWN ,LABEL_NO,LINE_NO,MACH_NO,SITE_TIME,UP_PASSENGER from tm01  )

 

     start-hbase.sh