| Store | Cart

Speed up loading and free memory...

From: GrelEns <grel...@NOSPAMyahoo.NOTNEEDEDfr>
Thu, 11 Mar 2004 09:33:43 +0100

i need advices on this : having all files are text files, 1 is an index
(6mo) to another big data file (20mb), i can not put that in a database and
must work from raw files.

i've tried to design an oo interface to theses files, well, it took 10
minutes to just load and build the whole wrapper objects (really simple
ones - with not the full logic behind) and it tooks 446mb of memoy! ('top'
command executed while loading to check). if i use tuples instead of objects
it still need about 2 minutes and 240mb...

as i did not need all data in memory at a moment but only index and then
follow to some data, please have you any clue on how i can handle this ?
having to load only index tooks less than 1 minutes and only 60mb in memory
which is acceptable, but if i read data directly from a disk it is
desperatly slow...

thanks a lot,

Recent Messages in this Thread
GrelEns Mar 11, 2004 08:33 am
Otto Kr├╝se Mar 11, 2004 10:30 am
GrelEns Mar 11, 2004 11:41 am
benjamin schollnick Mar 11, 2004 12:25 pm
GrelEns Mar 11, 2004 05:07 pm
Josiah Carlson Mar 12, 2004 05:49 pm
Paul Rubin Mar 11, 2004 12:31 pm
Messages in this thread