python 【pandas】讀取excel、csv數據，提高索引速度

問題描述：數據處理，尤其是遇到大量數據且需要for循環處理時，需要消耗大量時間，如代碼1所示。通過data['trip_time'][i]的方式會占用大量的時間

代碼1

import time
t0=time.time()
for i in range(0,len(data.index)):
    data['trip_time'][i] = pd.Timestamp(data['lpep_dropoff_datetime'][i]) - pd.Timestamp(data['lpep_pickup_datetime'][i])
t1=time.time()
print(t1 - t0)

解決辦法，添加.at定位索引，data.at[i,'trip_time']

import time
t0=time.time()
for i in range(0,len(data.index)):
    data.at[i,'trip_time'] = pd.Timestamp(data.at[i,'lpep_dropoff_datetime']) - pd.Timestamp(data.at[i,'lpep_pickup_datetime'])
t1=time.time()
print(t1 - t0)

 

評價：可以看出 使用at進行索引的方法相比loc、iloc和ix要快了將近1000倍！

%timeit outdf.loc[0] = indf.loc[0]
100 loops, best of 3: 11.7 ms per loop
%timeit outdf.iloc[0] = indf.iloc[0]
100 loops, best of 3: 11.4 ms per loop
 %timeit outdf.ix[0] = indf.ix[0]
100 loops, best of 3: 11.6 ms per loop
%timeit outdf.at[0,'time'] = indf.at[0,'time']
10000 loops, best of 3: 25.3 µs per loop