Hive分組取Top N

Hive在0.11.0版本開始加入了row_number、rank、dense_rank分析函數，可以查詢分組排序后的top值

 

說明：

row_number() over ([partition col1] [order by col2] )

rank() over ( [partition col1] [order by col2] )

dense_rank() over ( [partition col1] [order by col2] )

它們都是根據col1字段分組，然后對col2字段進行排序，對排序后的每行生成一個行號，這個行號從1開始遞增

col1、col2都可以是多個字段，用','分隔

 

區別：

1）row_number：不管col2字段的值是否相等，行號一直遞增， 比如：有兩條記錄的值相等，但一個是第一，一個是第二

2）rank：上下兩條記錄的col2相等時，記錄的行號是一樣的，但下一個col2值的行號遞增N（N是重復的次數），比如：有兩條並列第一，下一個是第三，沒有第二

3）dense_rank：上下兩條記錄的col2相等時，下一個col2值的行號遞增1， 比如：有兩條並列第一，下一個是第二

 

row_number可以實現分頁查詢

 

實例：

 

1. hive> create table t(name string, sub string, score int) row format delimited fields terminated by '\t';

 

數據在附件的a.txt里

1. a chinese 98
2. a english 90
3. d chinese 88
4. c english 82
5. c math 98
6. b math 89
7. b chinese 79
8. z english 90
9. z math 89
10. z chinese 80
11. e math 99
12. e english 87
13. d english 90

 

 

1、row_number

1. hive (test)> select *, row_number() over (partition by sub order by score) as od from t; 

 

2、rank

1. hive (test)> select *, rank() over (partition by sub order by score) as od from t; 

 

3、dense_ran

1. hive (test)> select *, dense_rank() over (partition by sub order by score desc) from t;

 

業務實例：

統計每個學科的前三名

1. select * from (select *, row_number() over (partition by sub order by score desc) as od from t ) t where od<=3;

語文成績是80分的排名是多少

1. hive (test)> select od from (select *, row_number() over (partition by sub order by score desc) as od from t ) t where sub='chinese' and score=80;

分頁查詢

1. hive (test)> select * from (select *, row_number() over () as rn from t) t1 where rn between 1 and 5;

 

 

 

 

 

 

來自為知筆記(Wiz)

附件列表