pandas實現hive的lag和lead函數 以及 first_value和last_value函數

lag和lead VS shift 

該函數的格式如下:

• 第一個參數為列名，
• 第二個參數為往上第n行（可選，默認為1），
• 第三個參數為默認值（當往上第n行為NULL時候，取默認值，如不指定，則為NULL）

lag

lag(字段名,N，默認值) over(partition by 分組字段 order by 排序字段 排序方式) 

lead

lead(字段名,N，默認值) over(partition by 分組字段 order by 排序字段 排序方式)

案例:

select 
  cookieid, 
  createtime, 
  url, 
  row_number() over (partition by cookieid order by createtime) as rn, 
  LAG(createtime,1,'1970-01-01 00:00:00') over (partition by cookieid order by createtime) as last_1_time, 
  LAG(createtime,2) over (partition by cookieid order by createtime) as last_2_time 
from cookie.cookie4

select 
  cookieid, 
  createtime, 
  url, 
  row_number() over (partition by cookieid order by createtime) as rn, 
  LEAD(createtime,1,'1970-01-01 00:00:00') over (partition by cookieid order by createtime) as next_1_time, 
  LEAD(createtime,2) over (partition by cookieid order by createtime) as next_2_time 
from cookie.cookie4;

窗口函數的pandas實現

pandas中使用shift函數來實現lag/lead函數

import pandas as pd
df=pd.read_csv('c:/Users/WQBin/Desktop/data.csv',engine='python', names=['cookieid','createtime','url'])
df['last_url'] = df.sort_values('createtime').groupby('cookieid')['url'].shift(1)
df['next_url'] = df.sort_values('createtime').groupby('cookieid')['url'].shift(-1)
df.sort_values(by=['cookieid','createtime'])

 

first_value和 last_value VS first()和last()

• FIRST_VALUE 返回組中數據窗口的第一個值
  • 　　FIRST_VALUE ( [scalar_expression )OVER ( [ partition_by_clause ] order_by_clause )
• LAST_VALUE 返回組中數據窗口的最后一個值
  • 　　 LAST_VALUE ( [scalar_expression )OVER ( [ partition_by_clause order_by_clause )

 

select 
  cookieid, 
  createtime, 
  url, 
  row_number() over (partition by cookieid order by createtime) as rn, 
  first_value(url) over (partition by cookieid order by createtime) as first1 
from cookie.cookie4;

 

 

 

select 
  cookieid, 
  createtime, 
  url, 
  row_number() over (partition by cookieid order by createtime) as rn, 
  last_value(url) over (partition by cookieid order by createtime) as last1 
from cookie.cookie4;

 

 

窗口函數的pandas實現

 

 

df.sort_values(['createtime'], ascending=[ 1]).groupby(['cookieid']).first()

 

 

df.sort_values(['createtime'], ascending=[ 1]).groupby(['cookieid']).last()