[Python-Dev] Pickle implementation questions (original) (raw)

Tim Peters tim.peters at gmail.com
Fri Jun 30 22:24:10 CEST 2006

[Tim Peters]

I hope you've read PEP 307:

[Bruce Christensen]

I have. Thanks to you and Guido for writing it! It's been a huge help.

You're welcome -- although we were paid for that, so thanks aren't needed ;-)

The implementation is more like: [snip]

Thanks! That helps a lot. PEP 307 and the pickle module docs describe the end result pretty well, but they don't always make it clear where things are implemented.

Well, "where" and "how" are implementation details. Alas, those aren't always clearly separated from the semantics (and since Guido & I both like operational definitions, stuff we write is especially prone to muddiness on such points). The layers of backward compatibility for now out-of-favor gimmicks don't help either -- this is akin to reading the Windows API docs, finding around six functions that sound relevant, and then painfully discovering none of them actually do what you hope they do, one at a time :-)

I'm trying to make sure that I'm getting the right interaction between object.reduce(ex), pickle, and copyreg.

Alas, I'm sure I don't remember sufficient details anymore myself.

One (hopefully) last question: is object.reduce(ex) really implemented in object?

Yes, although I think you're overlooking this bit of the "acts as if" pseudo-implementation from my last note:

   elif proto < 2:
       return copy_reg._reduce_ex(self, proto)

That is, the object implementation left the proto < 2 cases coded in Python. You won't get to the (hoped to be) common path:

    else:
       # about 130 lines of C code exploiting proto 2

unless you ask for proto 2.

The tracebacks below would indicate that pickle directly implements the behavior that the specs say is implemented in object. However, that could be because frames from C code don't show up in tracebacks.

That's right, they don't, and the C object code calls back into copy_reg in proto < 2 cases.

I'm not familiar enough with CPython to know for sure.

>>> import copyreg >>> def bomb(*args, **kwargs): ... raise Exception('KABOOM! %r %r' % (args, kwargs)) ... >>> copyreg.reduceex = bomb >>> import pickle >>> pickle.dumps(object())

You're defaulting to protocol 0 there, so, as above, the object implementation actually calls copy_reg._reduce_ex(self, 0) in this case. Much the same if you do:

pickle.dumps(object(), 1)

I think it's a misfeature of pickle that it defaults to the oldest protocol instead of the newest, but not much to be done about that in Python 2.

Do one of these instead and the traceback will go away:

pickle.dumps(object(), 2) pickle.dumps(object(), -1) pickle.dumps(object(), pickle.HIGHESTPROTOCOL)

Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\lib\pickle.py", line 1386, in dumps Pickler(file, protocol, bin).dump(obj) File "C:\Python24\lib\pickle.py", line 231, in dump self.save(obj) File "C:\Python24\lib\pickle.py", line 313, in save rv = reduce(self.proto) File "", line 2, in bomb Exception: KABOOM! (<object object at 0x01E3C448>, 0) {}

It's certainly an implementation accident that the object coding happens to call back into copy_reg here. There was no intent that users be able to monkey-patch copy_reg and replace _reduce_ex(). It was left coded in Python purely as a cost/benefit tradeoff.

>>> class NewObj(object): ... def reduce(self): ... raise Exception("reducing NewObj")

In this case, it doesn't matter at all how object implements reduce or reduce_ex, because you're explicitly saying that NewObj has its own reduce method, and that overrides object's implementation. IOW, you're getting exactly what you ask for in this case, and regardless of pickle protocol specified:

>>> import pickle >>> pickle.dumps(NewObj())

Ask for protocols 1 or 2 here, and you'll get the same traceback. It would be a bug if you didn't :-)

Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\lib\pickle.py", line 1386, in dumps Pickler(file, protocol, bin).dump(obj) File "C:\Python24\lib\pickle.py", line 231, in dump self.save(obj) File "C:\Python24\lib\pickle.py", line 313, in save rv = reduce(self.proto) File "", line 3, in reduce Exception: reducing NewObj

More information about the Python-Dev mailing list