合並數據集
- pandas.merge 可根據一個或多個鍵將不同DataFrame中的行連接起來。
- pandas.concat 可以沿着一條軸將多個對象堆疊到一起。
- combine_first
merge
默認情況下，merge做的是'inner'連接;結果中的鍵是交集
和數據庫中的left、right以及outer連接這些外連全部是形成笛卡爾積
merge合並的數據如果是多對多，則是笛卡爾積的形式合並
import pandas as pd
import numpy as np
df1 = pd.DataFrame({'key1':['b','b','a','c','a','a','b'],
'data1':range(7)
})
df2 = pd.DataFrame({'key1':['a','b','d'],
'data2':range(3)
})
df1
key1 data1
0 b 0
1 b 1
2 a 2
3 c 3
4 a 4
5 a 5
6 b 6
df2
key1 data2
0 a 0
1 b 1
2 d 2
# merge默認會合並相同的列名，但是最好顯示指定一下，on用於連接左右都存在的列名，如果只有一側有，那不能使用on,使用left_on或者right_on
pd.merge(df1, df2, on='key1',how='left')
key1 data1 data2
0 b 0 1.0
1 b 1 1.0
2 a 2 0.0
3 c 3 NaN
4 a 4 0.0
5 a 5 0.0
6 b 6 1.0
# 如果兩個對象的列名不同，可以詳細顯示合並的各列情況
df3 = pd.DataFrame({'key1':['b','b','a','c','a','a','b'],
'data1':range(7)
})
df4 = pd.DataFrame({'key2':['a','b','d'],
'data2':range(3)
})
# left_on,right_on用於連接存在於一方的列
pd.merge(df3,df4,left_on='key1',right_on='key2')
key1 data1 key2 data2
0 b 0 b 1
1 b 1 b 1
2 b 6 b 1
3 a 2 a 0
4 a 4 a 0
5 a 5 a 0
# how參數里面填寫連接的類別，有left、right、outer分別對應左連接，右連接，全連接
pd.merge(df3, df4,left_on='key1',right_on='key2',how='outer')
key1 data1 key2 data2
0 b 0.0 b 1.0
1 b 1.0 b 1.0
2 b 6.0 b 1.0
3 a 2.0 a 0.0
4 a 4.0 a 0.0
5 a 5.0 a 0.0
6 c 3.0 NaN NaN
7 NaN NaN d 2.0
# 示例數據源
df5 = pd.DataFrame({'key':['b','b','a','c','a','b'],
'data1':range(6)
})
df6 = pd.DataFrame({'key':['a','b','a','b','d'],
'data2':range(5)
})
df5
key data1
0 b 0
1 b 1
2 a 2
3 c 3
4 a 4
5 b 5
df6
key data2
0 a 0
1 b 1
2 a 2
3 b 3
4 d 4
# 左外連接的笛卡爾積,左右3個b，右有2個b，2*3=6個b,又會形成並集
pd.merge(df5,df6,how='left')
key data1 data2
0 b 0 1.0
1 b 0 3.0
2 b 1 1.0
3 b 1 3.0
4 a 2 0.0
5 a 2 2.0
6 c 3 NaN
7 a 4 0.0
8 a 4 2.0
9 b 5 1.0
10 b 5 3.0
left = pd.DataFrame({'key1':['foo','foo','bar'],
'key2':['one','two','one'],
'lval':[1,2,3]})
right = pd.DataFrame({'key1':['foo','foo','bar','bar'],
'key2':['one','one','one','two'],
'rval':[4,5,6,7]})
left
key1 key2 lval
0 foo one 1
1 foo two 2
2 bar one 3
right
key1 key2 rval
0 foo one 4
1 foo one 5
2 bar one 6
3 bar two 7
# 合並多個列名，需要把他們當做一個整體看，然后做笛卡爾積
pd.merge(left,right,on=['key1','key2'],how='outer')
key1 key2 lval rval
0 foo one 1.0 4.0
1 foo one 1.0 5.0
2 foo two 2.0 NaN
3 bar one 3.0 6.0
4 bar two NaN 7.0
# suffixes用於指定附加到左右兩個dataframe對象的列標簽的名，下面是是原始的命名，默認加_x,_y
pd.merge(left,right,on='key1')
key1 key2_x lval key2_y rval
0 foo one 1 one 4
1 foo one 1 one 5
2 foo two 2 one 4
3 foo two 2 one 5
4 bar one 3 one 6
5 bar one 3 two 7
# suffixes就是指定一下名字，會形成下面的效果
pd.merge(left,right,on='key1',suffixes=('_left','_right'))
key1 key2_left lval key2_right rval
0 foo one 1 one 4
1 foo one 1 one 5
2 foo two 2 one 4
3 foo two 2 one 5
4 bar one 3 one 6
5 bar one 3 two 7
索引上的合並
left1 = pd.DataFrame({'key':['a','b','a','a','b','c'],
'value':range(6)})
right1 = pd.DataFrame({'group_val':[3.5,7]},index=['a','b'])
left1
key value
0 a 0
1 b 1
2 a 2
3 a 3
4 b 4
5 c 5
right1
group_val
a 3.5
b 7.0
# 上面是用key的列值去合並右側的行索引，right_index開啟將行索引用作連接的鍵，如果是左側的表，同理有left_index，默認交集
pd.merge(left1, right1, left_on='key',right_index=True)
key value group_val
0 a 0 3.5
2 a 2 3.5
3 a 3 3.5
1 b 1 7.0
4 b 4 7.0
# 指定並集
pd.merge(left1, right1, left_on='key',right_index=True,how='outer')
key value group_val
0 a 0 3.5
2 a 2 3.5
3 a 3 3.5
1 b 1 7.0
4 b 4 7.0
5 c 5 NaN
層次化索引的合並
lefth = pd.DataFrame({'key1':['Ohio','Ohio','Ohio','Nevada','Nevada'],
'key2':[2000,2001,2002,2001,2002],
'data':np.arange(5.)})
# lefth.set_index(['key1','key2']).T 將列轉為層次化行索引
lefth
key1 key2 data
0 Ohio 2000 0.0
1 Ohio 2001 1.0
2 Ohio 2002 2.0
3 Nevada 2001 3.0
4 Nevada 2002 4.0
right1 = pd.DataFrame(np.arange(12).reshape(6,2),index=[
['Nevada','Nevada','Ohio','Ohio','Ohio','Ohio'],
[2001,2000,2000,2000,2001,2002]
],columns=['data1','data2'])
right1
data1 data2
Nevada 2001 0 1
2000 2 3
Ohio 2000 4 5
2000 6 7
2001 8 9
2002 10 11
# 對於層次化索引的合並，左側的列名是右側的行索引，故開啟right_index
pd.merge(lefth,right1,left_on=['key1','key2'],right_index=True)
key1 key2 data data1 data2
0 Ohio 2000 0.0 4 5
0 Ohio 2000 0.0 6 7
1 Ohio 2001 1.0 8 9
2 Ohio 2002 2.0 10 11
3 Nevada 2001 3.0 0 1
# 合並雙方的索引
left2 = pd.DataFrame([[1.,2.],[3.,4.],[5.,6.]],index=['a','c','e'],columns=['Ohio','Nevada'])
right2 = pd.DataFrame([[7.,8.],[9.,10.],[11.,12.],[13,14]],index=['b','c','d','e'],columns=['Missouri','Alabama'])
pd.merge(left2,right2,how='outer',left_index=True,right_index=True)
Ohio Nevada Missouri Alabama
a 1.0 2.0 NaN NaN
b NaN NaN 7.0 8.0
c 3.0 4.0 9.0 10.0
d NaN NaN 11.0 12.0
e 5.0 6.0 13.0 14.0
# join方法，更方便實現按索引合並
left2.join(right2,how='outer')
Ohio Nevada Missouri Alabama
a 1.0 2.0 NaN NaN
b NaN NaN 7.0 8.0
c 3.0 4.0 9.0 10.0
d NaN NaN 11.0 12.0
e 5.0 6.0 13.0 14.0
left3 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3','A4'],
'B': ['B0', 'B1', 'B2', 'B3','b4'],
'key': ['K0', 'K1', 'K0', 'K1','C1']})
right3 = pd.DataFrame({'C': ['C0', 'C1'],
'D': ['D0', 'D1']},
index=['K0', 'K1'])
# 如果列的值是另一個dataframe的行索引
left3.join(right3,on='key')
A B key C D
0 A0 B0 K0 C0 D0
1 A1 B1 K1 C1 D1
2 A2 B2 K0 C0 D0
3 A3 B3 K1 C1 D1
4 A4 b4 C1 NaN NaN
# 對於簡單的索引合並，你可以向join傳入一組DataFrame
another = pd.DataFrame([[7.,8.],[9.,10.],[11.,12.],[16.,17.]],index=['a','c','e','f'],columns=['New York','Oregon'])
left2.join([right2,another])
Ohio Nevada Missouri Alabama New York Oregon
a 1.0 2.0 NaN NaN 7.0 8.0
c 3.0 4.0 9.0 10.0 9.0 10.0
e 5.0 6.0 13.0 14.0 11.0 12.0