Issue 25776: More compact pickle of iterators etc (original) (raw)

Proposed patch makes a number of classes produce more compact pickle data in common case. This includes iterators of list, tuple, str, bytes, bytearray, enumerate, array, deque, iterator object for classes with getitem, some itertools iterators, and non-iterator objects: slice, bytearray, deque. This is achieved by omitting default constructor arguments or state.

Exhausted iterators are pickled as iter(()). This is not new, exhausted bytes, and bytearray iterators, and reversed list iterator are already pickled as iter('') or iter([]) correspondingly. iter(()) is just the simplest way to create an empty iterator and it has the most compact pickle representation.

An example.

Unpatched:

import pickle, pickletools, itertools len(pickle.dumps(itertools.islice('abcdefgh', 4), 3)) 80 len(pickletools.optimize(pickle.dumps(itertools.islice('abcdefgh', 4), 3))) 66 pickletools.dis(pickletools.optimize(pickle.dumps(itertools.islice('abcdefgh', 4), 3))) 0: \x80 PROTO 3 2: c GLOBAL 'itertools islice' 20: ( MARK 21: c GLOBAL 'builtins iter' 36: X BINUNICODE 'abcdefgh' 49: \x85 TUPLE1 50: R REDUCE 51: K BININT1 0 53: b BUILD 54: K BININT1 0 56: K BININT1 4 58: K BININT1 1 60: t TUPLE (MARK at 20) 61: R REDUCE 62: K BININT1 0 64: b BUILD 65: . STOP highest protocol among opcodes = 2

Patched:

len(pickle.dumps(itertools.islice('abcdefgh', 4), 3)) 69 len(pickletools.optimize(pickle.dumps(itertools.islice('abcdefgh', 4), 3))) 55 pickletools.dis(pickletools.optimize(pickle.dumps(itertools.islice('abcdefgh', 4), 3))) 0: \x80 PROTO 3 2: c GLOBAL 'itertools islice' 20: c GLOBAL 'builtins iter' 35: X BINUNICODE 'abcdefgh' 48: \x85 TUPLE1 49: R REDUCE 50: K BININT1 4 52: \x86 TUPLE2 53: R REDUCE 54: . STOP highest protocol among opcodes = 2

There is the extra special case code for bytes and bytearray iterators and reversed list iterator. The patch just extends the optimizations to other builtin iterators.

In the example with itertools.islice() the overhead of pickling redundant data is 20%. If there are a lot of iterators, the overhead can be up to 90%.

import pickle, pickletools, itertools len(pickletools.optimize(pickle.dumps([itertools.islice('abcdefgh', 4) for i in range(1000)], 4))) Unpatched: 23059 Patched: 12059

Of course this is degenerated case. But if the data contains a lot of initial iterators of the same sequence or of short sequences, or exhausted iterators, the benefit can be not small.

Yet one benefit from the patch is that it speeds up copying and deepcopying initial and exhausted iterators.

$ ./python -m timeit -s "from itertools import islice; from copy import copy; it = islice('abcdefgh', 4)" -- "copy(it)" Unpatched: 7.37 usec per loop Patched: 6.4 usec per loop

$ ./python -m timeit -s "from itertools import islice; from copy import deepcopy; it = islice('abcdefgh', 4)" -- "deepcopy(it)" Unpatched: 41.7 usec per loop Patched: 32.6 usec per loop

$ ./python -m timeit -s "from copy import copy; it = iter('abcdefgh')" -- "copy(it)" Unpatched: 10.4 usec per loop Patched: 9.67 usec per loop

$ ./python -m timeit -s "from copy import deepcopy; it = iter('abcdefgh')" -- "deepcopy(it)" Unpatched: 21.1 usec per loop Patched: 18.3 usec per loop

$ ./python -m timeit -s "from copy import copy; it = iter(list('abcdefgh'))" -- "copy(it)" Unpatched: 10.3 usec per loop Patched: 9.54 usec per loop

$ ./python -m timeit -s "from copy import deepcopy; it = iter(list('abcdefgh'))" -- "deepcopy(it)" Unpatched: 39.7 usec per loop Patched: 36.8 usec per loop