[Python-Dev] Unpickling memory usage problem, and a proposed solution (original) (raw)

Alexandre Vassalotti alexandre at peadrop.com
Fri Apr 23 20:49:47 CEST 2010

On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti <alexandre at peadrop.com> wrote:

Collin Winter wrote a simple optimization pass for cPickle in Unladen Swallow [1]. The code reads through the stream and remove all the unnecessary PUTs in-place.

I just noticed the code removes all PUT opcodes, regardless if they are needed or not. So, this code can only be used if there's no GET in the stream (which is unlikely for a large stream). I believe Collin made this trade-off for performance reasons. However, it wouldn't be hard to make the current code to work like pickletools.optimize().

-- Alexandre

More information about the Python-Dev mailing list