python pandas使用chunksize異步拆分固定行數的文件


import pandas as pd
import asyncio
from collections import defaultdict


collect = defaultdict(list)

#######創建處理一個對象的方法,並返回期物
async def dealone(chunk,sas):
	path='/data/chaifen/testphone_%d.txt'%sas
	chunk.to_csv(path,index=False)
	return path

async def submitor(chunk,sas):
	future = await dealone(chunk,sas)
	collect['res'].append(future)
	#print(future)


chunks = pd.read_csv('/data/testphone.txt',dtype='object',chunksize=3)

loop = asyncio.get_event_loop()
tasks = asyncio.wait([submitor(chunk,index) for index,chunk in enumerate(chunks,1)])
loop.run_until_complete(tasks)
print(collect['res'])

  


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM