[Python-Dev] Unpickling memory usage problem, and a proposed solution (original) (raw)

Dan Gindikin dgindikin at gmail.com
Fri Apr 23 23:50:50 CEST 2010

Antoine Pitrou <solipsis pitrou.net> writes:

Does cPickle bytecode have some kind of NOP instruction? You could keep track of which PUTs weren't necessary and zero them out at the end. It would be much cheaper than writing a whole other "optimized" stream.

For a large file, I'm not sure it is much faster to edit it in place than to rewrite it, and also since it's a large file, you are going to probably have it compressed, in which case you are out of luck anyway.

(of course it only works on a seekable stream )

Indeed... since I was dealing with zipped files, I had to pass in a "seek0" func, like so:

file_in_seek0_func = lambda x: os.popen('zcat ' + file_in)

More information about the Python-Dev mailing list