[轉]Hive/Beeline 使用筆記

FROM : http://www.7mdm.com/1407.html

Hive:

利用squirrel-sql 連接hive

add driver -> name&example url(jdbc:hive2://xxx:10000)->extra class path ->Add

{hive/lib/hive-common-*.jar

hive/lib/hive-contrib-*.jar

hive/lib/hive-jdbc-*.jar

hive/lib/libthrift-*.jar

hive/lib/hive-service-*.jar

hive/lib/httpclient-*.jar

hive/lib/httpcore-*.jar

hadoop/share/hadoop/common/hadoop-common--*.jar

hadoop/share/hadoop/common/lib/common-configuration-*.jar

hadoop/share/hadoop/common/lib/log4j-*.jar

hadoop/share/hadoop/common/lib/slf4j-api-*.jar

hadoop/share/hadoop/common/lib/slf4j-log4j-*.jar}

->List Drivers(wait ..then class name will auto set org.apache.hive.jdbc/HiveDriver)->OK->Add aliases ->chose the hive driver->done

 

Hive數據遷移

1.導出表

EXPORT TABLE <table_name> TO 'path/to/hdfs';

2.復制數據到另一個hdfs

hadoop distcp hdfs://:8020/path/to/hdfs hdfs:///path/to/hdfs

3.導入表

IMPORT TABLE <table_name> FROM 'path/to/another/hdfs';

Hive 輸出查詢結果到文件

輸出到本地文件：

insert overwrite local directory './test-04'
row format delimited
FIELDS TERMINATED BY '\t'
COLLECTION ITEMS TERMINATED BY ','
MAP KEYS TERMINATED BY ':'
select * from src;

輸出到hdfs：

輸出到hdfs好像不支持 row format，只能另辟蹊徑了

INSERT OVERWRITE DIRECTORY '/outputable.txt'
select concat(col1, ',', col2, ',', col3) from myoutputtable;

當然默認的分隔符是\001

若要直接對文件進行操作課直接用stdin的形式

eg. hadoop fs -cat ../000000_0 |python doSomeThing.py

#!/usr/bin/env python

import sys

for line in sys.stdin:

    (a,b,c)=line.strip().split('\001')

 

Hive 語法：

hive好像不支持select dicstinct col1 as col1 from table group by col1

需要用grouping sets

select col1 as col1 from table group by col1 grouping sets((col1))

Beeline:

文檔：https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients

利用jdbc連接hive：

hive2='JAVA_HOME=/opt/java7 HADOOP_HOME=/opt/hadoop /opt/hive/bin/beeline -u jdbc:hive2://n1.hd2.host.dxy:10000 -n hadoop -p fake -d org.apache.hive.jdbc.HiveDriver --color=true --silent=false --fastConnect=false --verbose=true'

beeline利用jdbc連接hive若需要執行多條命令使用

hive2 -e "xxx" -e "yyy"  -e...