hive lateral view 與 explode詳解

 

ref： https://blog.csdn.net/bitcarmanlee/article/details/51926530

1.explode

hive wiki對於expolde的解釋如下：

explode() takes in an array (or a map) as an input and outputs the elements of the array (map) as separate rows. UDTFs can be used in the SELECT expression list and as a part of LATERAL VIEW.

As an example of using explode() in the SELECT expression list, consider a table named myTable that has a single column (myCol) and two rows:

Then running the query:

SELECT explode(myCol) AS myNewCol FROM myTable;

 

will produce: 
 
The usage with Maps is similar:

SELECT explode(myMap) AS (myMapKey, myMapValue) FROM myMapTable;

 

總結起來一句話：explode就是將hive一行中復雜的array或者map結構拆分成多行。

使用實例： 
xxx表中有一個字段mvt為string類型，數據格式如下：

[{“eid”:”38”,”ex”:”affirm_time_Android”,”val”:”1”,”vid”:”31”,”vr”:”var1”},{“eid”:”42”,”ex”:”new_comment_Android”,”val”:”1”,”vid”:”34”,”vr”:”var1”},{“eid”:”40”,”ex”:”new_rpname_Android”,”val”:”1”,”vid”:”1”,”vr”:”var1”},{“eid”:”19”,”ex”:”hotellistlpage_Android”,”val”:”1”,”vid”:”1”,”vr”:”var01”},{“eid”:”29”,”ex”:”bookhotelpage_Android”,”val”:”0”,”vid”:”1”,”vr”:”var01”},{“eid”:”17”,”ex”:”trainMode_Android”,”val”:”1”,”vid”:”1”,”vr”:”mode_Android”},{“eid”:”44”,”ex”:”ihotelList_Android”,”val”:”1”,”vid”:”36”,”vr”:”var1”},{“eid”:”47”,”ex”:”ihotelDetail_Android”,”val”:”0”,”vid”:”38”,”vr”:”var1”}]

用explode小試牛刀一下：

select explode(split(regexp_replace(mvt,'\\[|\\]',''),'\\},\\{')) from ods_mvt_hourly where day=20160710 limit 10;

 

最后出來的結果如下： 
{“eid”:”38”,”ex”:”affirm_time_Android”,”val”:”1”,”vid”:”31”,”vr”:”var1” 
“eid”:”42”,”ex”:”new_comment_Android”,”val”:”1”,”vid”:”34”,”vr”:”var1” 
“eid”:”40”,”ex”:”new_rpname_Android”,”val”:”1”,”vid”:”1”,”vr”:”var1” 
“eid”:”19”,”ex”:”hotellistlpage_Android”,”val”:”1”,”vid”:”1”,”vr”:”var01” 
“eid”:”29”,”ex”:”bookhotelpage_Android”,”val”:”0”,”vid”:”1”,”vr”:”var01” 
“eid”:”17”,”ex”:”trainMode_Android”,”val”:”1”,”vid”:”1”,”vr”:”mode_Android” 
“eid”:”44”,”ex”:”ihotelList_Android”,”val”:”1”,”vid”:”36”,”vr”:”var1” 
“eid”:”47”,”ex”:”ihotelDetail_Android”,”val”:”0”,”vid”:”38”,”vr”:”var1”} 
{“eid”:”38”,”ex”:”affirm_time_Android”,”val”:”1”,”vid”:”31”,”vr”:”var1” 
“eid”:”42”,”ex”:”new_comment_Android”,”val”:”1”,”vid”:”34”,”vr”:”var1”

2.lateral view

hive wiki 上的解釋如下：

Lateral View Syntax

lateralView: LATERAL VIEW udtf(expression) tableAlias AS columnAlias (‘,’ columnAlias)* 
fromClause: FROM baseTable (lateralView)*

Description

Lateral view is used in conjunction with user-defined table generating functions such as explode(). As mentioned in Built-in Table-Generating Functions, a UDTF generates zero or more output rows for each input row. A lateral view first applies the UDTF to each row of base table and then joins resulting output rows to the input rows to form a virtual table having the supplied table alias.

Example

Consider the following base table named pageAds. It has two columns: pageid (name of the page) and adid_list (an array of ads appearing on the page) 

An example table with two rows: 

and the user would like to count the total number of times an ad appears across all pages. 
A lateral view with explode() can be used to convert adid_list into separate rows using the query:

SELECT pageid, adid FROM pageAds LATERAL VIEW explode(adid_list) adTable AS adid;

 

The resulting output will be 
 
Then in order to count the number of times a particular ad appears, count/group by can be used:

SELECT adid, count(1) FROM pageAds LATERAL VIEW explode(adid_list) adTable AS adid GROUP BY adid;

 

The resulting output will be 

由此可見，lateral view與explode等udtf就是天生好搭檔，explode將復雜結構一行拆成多行，然后再用lateral view做各種聚合。

3.實例

還是第一部分的例子，上面我們explode出來以后的數據，不是標准的json格式，我們通過lateral view與explode組合解析出標准的json格式數據:

SELECT ecrd, CASE WHEN instr(mvtstr,'{')=0 AND instr(mvtstr,'}')=0 THEN concat('{',mvtstr,'}') WHEN instr(mvtstr,'{')=0 AND instr(mvtstr,'}')>0 THEN concat('{',mvtstr) WHEN instr(mvtstr,'}')=0 AND instr(mvtstr,'{')>0 THEN concat(mvtstr,'}') ELSE mvtstr END AS mvt FROM ods.ods_mvt_hourly LATERAL VIEW explode(split(regexp_replace(mvt,'\\[|\\]',''),'\\},\\{')) addTable AS mvtstr WHERE DAY='20160710' and ecrd is not null limit 10

 

查詢出來的結果： 
xxx 
{“eid”:”38”,”ex”:”affirm_time_Android”,”val”:”1”,”vid”:”31”,”vr”:”var1”} 
xxx 
{“eid”:”42”,”ex”:”new_comment_Android”,”val”:”1”,”vid”:”34”,”vr”:”var1”} 
xxx 
{“eid”:”40”,”ex”:”new_rpname_Android”,”val”:”1”,”vid”:”1”,”vr”:”var1”} 
xxx 
{“eid”:”19”,”ex”:”hotellistlpage_Android”,”val”:”1”,”vid”:”1”,”vr”:”var01”} 
xxx 
{“eid”:”29”,”ex”:”bookhotelpage_Android”,”val”:”0”,”vid”:”1”,”vr”:”var01” 
xxx 
{“eid”:”17”,”ex”:”trainMode_Android”,”val”:”1”,”vid”:”1”,”vr”:”mode_Android”} 
xxx 
{“eid”:”44”,”ex”:”ihotelList_Android”,”val”:”1”,”vid”:”36”,”vr”:”var1”} 
xxx 
{“eid”:”47”,”ex”:”ihotelDetail_Android”,”val”:”1”,”vid”:”38”,”vr”:”var1”} 
xxx 
{“eid”:”38”,”ex”:”affirm_time_Android”,”val”:”1”,”vid”:”31”,”vr”:”var1”} 
xxx 
{“eid”:”42”,”ex”:”new_comment_Android”,”val”:”1”,”vid”:”34”,”vr”:”var1”}

4.Ending

Lateral View通常和UDTF一起出現，為了解決UDTF不允許在select字段的問題。 
Multiple Lateral View可以實現類似笛卡爾乘積。 
Outer關鍵字可以把不輸出的UDTF的空結果，輸出成NULL，防止丟失數據。

參考內容： 
1.http://blog.csdn.net/oopsoom/article/details/26001307 lateral view的用法實例 
2.https://my.oschina.net/leejun2005/blog/120463 復合函數的用法，比較詳細 
3.http://blog.csdn.net/zhaoli081223/article/details/46637517 udtf的介紹

 

Lateral View用法 與 Hive UDTF explode

 

Lateral View是Hive中提供給UDTF的conjunction，它可以解決UDTF不能添加額外的select列的問題。

1. Why we need Lateral View？

當我們想對hive表中某一列進行split之后，想對其轉換成1 to N的模式，即一行轉多列。

hive不允許我們在UDTF函數之外，再添加其它select語句。

如下，我們想將登錄某個游戲的用戶id放在一個字段user_ids里，對每一行數據用UDTF后輸出多行。

 

1.  
   select game_id, explode(split(user_ids, '\\[\\[\\[')) as user_id   from login_game_log  where dt='2014-05-15'
2.  
   FAILED: Error in semantic analysis: UDTF 's are not supported outside the SELECT clause, nor nested in expressions。

 

提示語法分析錯誤，UDTF不支持函數之外的select 語句，真無語。。。

如果我們想支持怎么辦呢？接下來就是Lateral View 登場的時候了。

2. Lateral View explain

2.1 單個Lateral View

Lateral view is used in conjunction with user-defined table generatingfunctions such as explode(). As mentioned in Built-in Table-Generating Functions, a UDTF generates zero or more output rows foreach input row. A lateral view first applies the UDTF to each row of base tableand then joins resulting output rows to the input rows to form a virtual tablehaving the supplied table alias.

解釋一下：

 

Lateral view 其實就是用來和像類似explode這種UDTF函數聯用的。lateral view 會將UDTF生成的結果放到一個虛擬表中，然后這個虛擬表會和輸入行即每個game_id進行join 來達到連接UDTF外的select字段的目的。

Lateral View Syntax

lateralView: LATERAL VIEW udtf(expression) tableAlias AS columnAlias ( ','  columnAlias)*

fromClause: FROM baseTable (lateralView)*

可以看出，可以在2個地方用Lateral view：

1. 在udtf前面用

2. 在from baseTable后面用

 

舉個例子：

1. 先創建一個文件，里面2列用\t分割，game_id和user_ids

 

1.  
   hive> create table test_lateral_view_shengli(game_id string,userl_ids string) row format delimited fields terminated by '\t' stored as textfile;
2.  
   OK
3.  
   Time taken: 2.451 seconds
4.  
   hive> load data local inpath '/home/hadoop/test_lateral' into table test_lateral_view_shengli;
5.  
   Copying data from file:/home/hadoop/test_lateral
6.  
   Copying file: file:/home/hadoop/test_lateral
7.  
   Loading data to table dw.test_lateral_view_shengli
8.  
   OK
9.  
   Time taken: 6.716 seconds
10.  
    hive> select * from test_lateral_view_shengli;
11.  
    OK
12.  
    game101 15358083654[[[ab33787873[[[zjy18052480603[[[shlg1881826[[[lxqab110
13.  
    game66 winning1ren[[[ 13810537508
14.  
    game101 hu330602003[[[hu330602004[[[hu330602005[[[ 15967506560

下面使用lateral_view

 

 

1.  
   hive> select game_id, user_id
2.  
   > from test_lateral_view_shengli lateral view explode(split(userl_ids,'\\[\\[\\[')) snTable as user_id
3.  
   > ;
4.  
   Total MapReduce jobs = 1
5.  
   Launching Job 1 out of 1
6.  
   Number of reduce tasks is set to 0 since there's no reduce operator
7.  
   Starting Job = job_201403301416_445839, Tracking URL = http://10.1.9.10:50030/jobdetails.jsp?jobid=job_201403301416_445839
8.  
   Kill Command = /app/home/hadoop/src/hadoop-0.20.2-cdh3u5/bin/../bin/hadoop job -Dmapred.job.tracker=10.1.9.10:9001 -kill job_201403301416_445839
9.  
   2014-05-16 17:39:19,108 Stage-1 map = 0%, reduce = 0%
10.  
    2014-05-16 17:39:28,157 Stage-1 map = 100%, reduce = 0%
11.  
    2014-05-16 17:39:38,830 Stage-1 map = 100%, reduce = 100%
12.  
    Ended Job = job_201403301416_445839
13.  
    OK
14.  
    game101 hu330602003
15.  
    game101 hu330602004
16.  
    game101 hu330602005
17.  
    game101 15967506560
18.  
    game101 15358083654
19.  
    game101 ab33787873
20.  
    game101 zjy18052480603
21.  
    game101 shlg1881826
22.  
    game101 lxqab110
23.  
    game66 winning1ren
24.  
    game66 13810537508

 

2.2 多個Lateral View

From語句后可以跟多個Lateral View。

A FROM clause can have multiple LATERAL VIEW clauses. Subsequent LATERAL VIEWS can reference columns from any of the tables appearing to the left of the LATERAL VIEW.

給定數據：

Array<int> col1	Array<string> col2
[1, 2]	[a", "b", "c"]
[3, 4]	[d", "e", "f"]

轉換目標：

想同時把第一列和第二列拆開，類似做笛卡爾乘積。

int myCol1	string myCol2
1	"a"
1	"b"
1	"c"
2	"a"
2	"b"
2	"c"
3	"d"
3	"e"
3	"f"
4	"d"
4	"e"
4	"f"

我們可以這樣寫：

1.  
   SELECT myCol1, myCol2 FROM baseTable
2.  
   LATERAL VIEW explode(col1) myTable1 AS myCol1
3.  
   LATERAL VIEW explode(col2) myTable2 AS myCol2;

3. Outer Lateral View

還有一種情況，如果UDTF轉換的Array是空的怎么辦呢？

在Hive0.12里面會支持outer關鍵字，如果UDTF的結果是空，默認會被忽略輸出。

如果加上outer關鍵字，則會像left outer join 一樣，還是會輸出select出的列，而UDTF的輸出結果是NULL。

hive> select * FROM test_lateral_view_shengli LATERAL VIEW explode(array()) C AS a ;

結果是什么都不輸出。

 

如果加上outer關鍵字：

SELECT * FROM src LATERAL VIEW OUTER explode(array()) C AS a limit 10;

1.  
   238 val_238 NULL
2.  
   86 val_86 NULL
3.  
   311 val_311 NULL
4.  
   27 val_27 NULL
5.  
   165 val_165 NULL
6.  
   409 val_409 NULL
7.  
   255 val_255 NULL
8.  
   278 val_278 NULL
9.  
   98 val_98 NULL
10.  
    ...

4.總結：

 

Lateral View通常和UDTF一起出現，為了解決UDTF不允許在select字段的問題。

Multiple Lateral View可以實現類似笛卡爾乘積。

Outer關鍵字可以把不輸出的UDTF的空結果，輸出成NULL，防止丟失數據。

 

原創文章，轉載請注明出自： http://blog.csdn.net/oopsoom/article/details/26001307