[Python-Dev] Unpickling memory usage problem, and a proposed solution (original) (raw)

Alexandre Vassalotti alexandre at peadrop.com
Fri Apr 23 21:24:54 CEST 2010

On Fri, Apr 23, 2010 at 3:07 PM, Collin Winter <collinwinter at google.com> wrote:

I should add that, adding the necessary bookkeeping to remove only unused PUTs (instead of the current all-or-nothing scheme) should not be hard. I'd watch out for a further performance/memory hit; the pickling benchmarks in the benchmark suite should help assess this.

I was thinking about this too. A simple boolean table could be fast, while keeping the space requirement down. This scheme would be nice to caches as well.

The current optimization penalizes pickling to speed up unpickling, which made sense when optimizing pickles that would go into memcache and be read out 13-15x more often than they were written.

This is my current impression of how pickle is most often used. Are you aware of a use case of pickle where you do more writes than reads? I can't think of any.

-- Alexandre

More information about the Python-Dev mailing list