y********0
发帖数: 638 | 1 【 以下文字转载自 DataSciences 讨论区 】
发信人: youyou2010 (悠悠), 信区: DataSciences
标 题: Memory Error in pd.concat with Python
发信站: BBS 未名空间站 (Mon Aug 24 15:22:34 2015, 美东)
I am more recently working with 80 dataframes in python each of which holds
650K rows with the same 11 columns varying from string to number. My initial
goal is to concatenate all dataframes and make a general analysis but I keep
hitting the dead ends in doing so.
1. I started from pd.concat([df1,df2]) but ended up with memory error and
failed to figure out a solution after searching over internet.
2.I then, checking on some online suggestions, converted the number into
float32 for the reduction of memory burden and this time only found it not
helpful either.
3. If you are asking for a little background of my PC and python,I am using
a 32-bit Python with 2.7.5, pandas 0.13.0 and little chance to switch to 64
version.
Any comments, suggestions will be really appreciated. |
|
