hive讀取hbase數據


簡單的例子看這個:https://blog.csdn.net/dominic_tiger/article/details/70237542 和 https://blog.csdn.net/xiewenbo/article/details/24627323

之前創建時報錯

FAILED: Error in metadata: java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hadoop.hive.hbase.HBaseSerDe: columns has 3 elements while hbase.columns.mapping has 4 elements (counting the key if implicit))

解決方法:

在創建hive/hbase相關聯的表時,hbase表結構默認會有一個字段key,如果沒有一個顯示的字段'key'那么在創建表的進修,會自己創建,這樣hive對應的表就會出現問題,所以在hive對應的表里一定要加上key這個字段,為了避免這個問題,在hbase表結構里可以顯示的添加'key'字段,這樣不容易出問題。

CREATE TABLE hperson1(key string,id string, name string,email string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf0:id,cf1:name,cf2:email") TBLPROPERTIES ("hbase.table.name" = "hbperson1");

或者:

CREATE TABLE hperson1(key string,id string, name string,email string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf0:id,cf1:name,cf2:email") TBLPROPERTIES ("hbase.table.name" = "hbperson1");

 

1.-------------------------------------
 hive> CREATE EXTERNAL TABLE tm05(key int,ROWKEY string ,BUS_NO int,CITY_NO int,DEPT_NO int,DOWN_PASSENGER int,FILA_NO int,GROUP_NO int,
  INS_TIME string, IS_UP_DOWN int ,LABEL_NO int ,LINE_NO  string ,MACH_NO int,SITE_TIME string ,UP_PASSENGER int)
    > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    > WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:ROWKEY,cf1:BUS_NO,cf1:CITY_NO,cf1:DEPT_NO,cf1:DOWN_PASSENGER,cf1:FILA_NO,cf1:GROUP_NO,
    cf1:INS_TIME,cf1:IS_UP_DOWN,cf1:LABEL_NO,cf1:LINE_NO,cf1:MACH_NO,cf1:SITE_TIME,cf1:UP_PASSENGER
") > TBLPROPERTIES ("hbase.table.name" = "tm_temp05", "hbase.mapred.output.outputtable" = "tm_temp05"); 2.-------------------------------------------------------------------------------------------------------- CREATE EXTERNAL TABLE tm01(key int,ROWKEY string ,BUS_NO int,CITY_NO int,DEPT_NO int,DOWN_PASSENGER int,FILA_NO int,GROUP_NO int, INS_TIME string, IS_UP_DOWN int ,LABEL_NO int ,LINE_NO string ,MACH_NO int,SITE_TIME string ,UP_PASSENGER int) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:ROWKEY,cf1:BUS_NO,cf1:CITY_NO,cf1:DEPT_NO,cf1:DOWN_PASSENGER,cf1:FILA_NO,cf1:GROUP_NO, cf1:INS_TIME,cf1:IS_UP_DOWN,cf1:LABEL_NO,cf1:LINE_NO,cf1:MACH_NO,cf1:SITE_TIME,cf1:UP_PASSENGER") > TBLPROPERTIES ("hbase.table.name" = "tm_temp05", "hbase.mapred.output.outputtable" = "tm_temp05"); 3.--------------------------------------------------------------------------------------------------------------------- CREATE EXTERNAL TABLE tm02(key string,ROWKEY string ,BUS_NO int,CITY_NO int,DEPT_NO int,DOWN_PASSENGER int,FILA_NO int,GROUP_NO int, INS_TIME string, IS_UP_DOWN int ,LABEL_NO int ,LINE_NO string ,MACH_NO int,SITE_TIME string ,UP_PASSENGER int) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:ROWKEY,cf1:BUS_NO,cf1:CITY_NO,cf1:DEPT_NO,cf1:DOWN_PASSENGER,cf1:FILA_NO,cf1:GROUP_NO, cf1:INS_TIME,cf1:IS_UP_DOWN,cf1:LABEL_NO,cf1:LINE_NO,cf1:MACH_NO,cf1:SITE_TIME,cf1:UP_PASSENGER") > TBLPROPERTIES ("hbase.table.name" = "tm_temp05", "hbase.mapred.output.outputtable" = "tm_temp05");

方案1和方案2結果都一樣

方案3:

總結:在必須要使用一個無用列(不使用無用列key會報錯),建議使用方案1或方案2. 

上面創建的表在hive里是外部表,數據沒有存到hadoop的hdfs上。下面創建將外部表的數據放到hadoop的hdfs上

A.//---創建表
CREATE  TABLE mt01(ROWKEY string ,BUS_NO int,CITY_NO int,DEPT_NO int,DOWN_PASSENGER int,FILA_NO int,GROUP_NO int,
     INS_TIME string, IS_UP_DOWN int ,LABEL_NO int ,LINE_NO  string ,MACH_NO int,SITE_TIME string ,UP_PASSENGER int)
B.//---將查詢的數據放到上面創建表里
INSERT INTO  mt01 (ROWKEY,BUS_NO,CITY_NO,DEPT_NO,DOWN_PASSENGER,FILA_NO,GROUP_NO,
     INS_TIME, IS_UP_DOWN ,LABEL_NO,LINE_NO,MACH_NO,SITE_TIME,UP_PASSENGER)  ( select ROWKEY,BUS_NO,CITY_NO,DEPT_NO,DOWN_PASSENGER,FILA_NO,GROUP_NO,
     INS_TIME, IS_UP_DOWN ,LABEL_NO,LINE_NO,MACH_NO,SITE_TIME,UP_PASSENGER from tm01  )

 


     start-hbase.sh

 


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM