[Python-Dev] pickle and copy discrepancy (original) (raw)

Serhiy Storchaka storchaka at gmail.com
Tue Mar 1 06:14:01 EST 2016

The pickle and the copy modules use the same protocol. The reconstruct the object by data returned by the reduce_ex/reduce method, but does it in different and incompatible way.

In general case the result of reduce includes:

  1. The class of the object and arguments to new().
  2. The state passed to setstate() (or a dict of attributes and possible a tuple of slots values).
  3. An iterator of list items that should be appended to the object by calling extend() or append().
  4. An iterator of key-value value pairs that should be set in the object by calling update() or setitem().

The difference is that the copy module sets object's state before adding items and key-value pairs, but the pickle module sets object's state after adding items and key-value pairs. If append() or setitem() depend on the state of the object, the pickling is incompatible with the copying.

The behaviour of copy was changed in issue1100562 [1] (see also issue1099746 [2]). But this caused a problem with other classes (see issue10131 [3]). Changing either pickle or copy for sure will break existing code. But keeping current discrepancy makes existing code not correct and makes too hard to write correct code that works with both pickle and copy. For sure most existing code for which it is matter is not correct.

The behaviour of default reducing method for dicts/lists subclasses is not documented [4].

We should choose in what direction we have to break backward compatibility. The behaviour of the copy module looks more natural. It allows to work correctly most naive implementations (as in [2]). The pickle module is more used and breaking it can cause more harm. But the order of object reconstruction is determined at pickling time, thus already pickled data will be reconstructed with old order. The change will only affect new pickles.

[1] http://bugs.python.org/issue1100562 [2] http://bugs.python.org/issue1099746 [3] http://bugs.python.org/issue10131 [4] http://bugs.python.org/issue4712

More information about the Python-Dev mailing list