[Python-Dev] Unpickling memory usage problem, and a proposed solution (original) (raw)

Dan Gindikin dgindikin at gmail.com
Fri Apr 23 21:57:45 CEST 2010

Collin Winter <collinwinter google.com> writes:

I should add that, adding the necessary bookkeeping to remove only unused PUTs (instead of the current all-or-nothing scheme) should not be hard. I'd watch out for a further performance/memory hit; the pickling benchmarks in the benchmark suite should help assess this. The current optimization penalizes pickling to speed up unpickling, which made sense when optimizing pickles that would go into memcache and be read out 13-15x more often than they were written.

This wouldn't help our use case, your code needs the entire pickle stream to be in memory, which in our case would be about 475mb, this is on top of the 300mb+ data structures that generated the pickle stream.

More information about the Python-Dev mailing list