類似Mysql的數據庫概念:
hive> CREATE DATABASE cui;
hive> USE cui;
創建表:
CREATE TABLE test(
first STRING,
second STRING
)
默認記錄和字段分隔符:
\n 每行一條記錄
^A 分隔列(八進制 \001)
^B 分隔ARRAY或者STRUCT中的元素,或者MAP中多個鍵值對之間分隔(八進制 \002)
^C 分隔MAP中鍵值對的“鍵”和“值”(八進制 \003)
自定義分隔符:
CREATE TABLE test(
……
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001'
COLLECTION ITEMS TERMINATED BY '\002'
MAP KEYS TERMINATED BY '\003'
LINES TERMINATED BY '\n'
查看信息:
DESCRIBE DATABASE cui;
DESCRIBE DATABASE EXTENDED cui;
分區表:
CREATE TABLE test(
……
)
PARTITIONED BY ( country STRING ); #分區鍵和字段不能重復
------------------------------------------------------------------
加載數據:
LOAD DATA LOCAL INPATH '/path/to/local/files'
OVERWRITE INTO TABLE test
PARTITION (country='CHINA')
有LOCAL表示從本地文件系統加載(文件會被拷貝到HDFS中)
無LOCAL表示從HDFS中加載數據(注意:文件直接被移動!!!而不是拷貝!!! 並且。。文件名都不帶改的。。)
OVERWRITE 表示是否覆蓋表中數據(或指定分區的數據)(沒有OVERWRITE 會直接APPEND,而不會濾重!)
關於加載數據的LOCAL關鍵字:
【使用前】:
[cui@node1 ~]$ hadoop fs -ls /workspace
Found 2 items
-rw-r--r-- 1 yjt-app1-web1 supergroup 1716110 2016-01-14 16:31 /workspace/IOS_OPERA_STATISTICS_20160113.01.DAT
-rw-r--r-- 1 yjt-app1-web1 supergroup 520964539 2016-01-14 21:20 /workspace/bpUserinfo_201511.log
【使用中】:
hive> LOAD DATA INPATH '/workspace/bpUserinfo_201511.log' INTO table testkv;
Loading data to table default.testkv
Table default.testkv stats: [numFiles=3, numRows=0, totalSize=520964575, rawDataSize=0]
OK
Time taken: 1.015 seconds
【使用后】:
[cui@node1 ~]$ hadoop fs -ls /workspace
Found 1 items
-rw-r--r-- 1 yjt-app1-web1 supergroup 1716110 2016-01-14 16:31 /workspace/IOS_OPERA_STATISTICS_20160113.01.DAT
[cui@node1 ~]$ hadoop fs -ls /user/hive/warehouse/testkv
Found 5 items
drwxrwxr-x - yjt-app1-web1 supergroup 0 2016-01-14 21:06 /user/hive/warehouse/testkv/.hive-staging_hive_2016-01-14_21-06-53_158_2726638877292950395-1
drwxrwxr-x - yjt-app1-web1 supergroup 0 2016-01-14 21:08 /user/hive/warehouse/testkv/.hive-staging_hive_2016-01-14_21-08-59_461_1557138301562621871-1
-rwxrwxr-x 1 yjt-app1-web1 supergroup 520964539 2016-01-14 21:20 /user/hive/warehouse/testkv/bpUserinfo_201511.log
-rwxrwxr-x 1 yjt-app1-web1 supergroup 24 2016-01-14 21:09 /user/hive/warehouse/testkv/part-00000
-rwxrwxr-x 1 yjt-app1-web1 supergroup 12 2016-01-14 21:09 /user/hive/warehouse/testkv/part-00001
如果加載同樣文件名的文件,會被自動重命名:
【使用前】:
[cui@node1 ~]$ cp ~/oltpUserSqlLogs/crm/data/bpUserinfo_201504.log .
[cui@node1 ~]$ mv bpUserinfo_201504.log bpUserinfo_201511.log
[cui@node1 ~]$ hadoop fs -put bpUserinfo_201511.log /workspace
[cui@node1 ~]$ hadoop fs -ls /workspace
Found 2 items
-rw-r--r-- 1 yjt-app1-web1 supergroup 1716110 2016-01-14 16:31 /workspace/IOS_OPERA_STATISTICS_20160113.01.DAT
-rw-r--r-- 1 yjt-app1-web1 supergroup 40393299 2016-01-15 13:20 /workspace/bpUserinfo_201511.log
【使用中】:
hive> LOAD DATA INPATH '/workspace/bpUserinfo_201511.log' INTO table testkv;
Loading data to table default.testkv
Table default.testkv stats: [numFiles=4, numRows=0, totalSize=561357874, rawDataSize=0]
OK
Time taken: 1.745 seconds
【使用后】:
[cui@node1 ~]$ hadoop fs -ls /workspace
Found 1 items
-rw-r--r-- 1 yjt-app1-web1 supergroup 1716110 2016-01-14 16:31 /workspace/IOS_OPERA_STATISTICS_20160113.01.DAT
[cui@node1 ~]$ hadoop fs -ls /user/hive/warehouse/testkv
Found 6 items
drwxrwxr-x - yjt-app1-web1 supergroup 0 2016-01-14 21:06 /user/hive/warehouse/testkv/.hive-staging_hive_2016-01-14_21-06-53_158_2726638877292950395-1
drwxrwxr-x - yjt-app1-web1 supergroup 0 2016-01-14 21:08 /user/hive/warehouse/testkv/.hive-staging_hive_2016-01-14_21-08-59_461_1557138301562621871-1
-rwxrwxr-x 1 yjt-app1-web1 supergroup 520964539 2016-01-14 21:20 /user/hive/warehouse/testkv/bpUserinfo_201511.log
-rwxrwxr-x 1 yjt-app1-web1 supergroup 40393299 2016-01-15 13:20 /user/hive/warehouse/testkv/bpUserinfo_201511_copy_1.log
-rwxrwxr-x 1 yjt-app1-web1 supergroup 24 2016-01-14 21:09 /user/hive/warehouse/testkv/part-00000
-rwxrwxr-x 1 yjt-app1-web1 supergroup 12 2016-01-14 21:09 /user/hive/warehouse/testkv/part-00001