用numpy處理大數據遇到的問題


在使用numpy讀取一個四百多萬行數據的.csv文件時拋出了如下異常:

numpy.core._exceptions.MemoryError: Unable to allocate array with shape (4566386, 23) and data type <U20

以下是我的源代碼:

import numpy as np
import matplotlib.pyplot as mp
import sklearn.ensemble as se
import sklearn.metrics as sm
headers = None
data = []
with open ('/home/tarena/桌面/i-80.csv','r') as f:
    for i,line in enumerate( f.readlines()):
        if i==0:
            headers=line.split(',')[2:]
        else:
            data.append(line.split(',')[2:])
headers = np.array(data)
data = np.array(data)
print(headers.shape)
print(data.shape)

以下是運行結果:

Traceback (most recent call last):
  File "/home/tarena/桌面/read_forest.py", line 13, in <module>
    headers = np.array(data)
numpy.core._exceptions.MemoryError: Unable to allocate array with shape (4566386, 23) and data type <U20

Process finished with exit code 1

雖然是報錯,但是還是拿到了結果。

各位大佬們,有沒有解決方案?


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM