Python：分塊讀取大文件

Python中讀取文件的方式有許多種，往往數據量大的情況令人苦惱。之前一篇博客https://www.cnblogs.com/xiaolan-Lin/p/12014686.html我說會發大文件讀取的后續。

那么后續來啦！

import pandas as pd
# 采用塊讀取功能，讀取csv文件
path1 = "./data\\tianchi_mobile_recommend_train_user.csv"

file1 = open(path1)

user_data = pd.read_csv(path1, sep=',', iterator=True)

loop = True
chunkSize = 1000
chunks1 = []
while loop:
    try:
        chunk1 = user_data.get_chunk(chunkSize)
        chunks1.append(chunk1)
    except StopIteration:
        loop = False
user_data = pd.concat(chunks1, ignore_index=True)