Issue 18015: python 2.7.5 fails to unpickle namedtuple pickled by 2.7.3 or 2.7.4 (original) (raw)

Change 18303391b981 breaks unpickling named tuples pickled by 2.7.3 and 2.7.4.

See closed issue #15535 for the full story. Unfortunately Raymond was wrong, when he wrote that the addition of dict was a 2.7.4 change. It was added by changeset 26d5f022eb1a in 2.7.3.

Now 2.7.5 can't unpickle any named tuples pickled by 2.7.3, which is probably one of the most widely used python versions.

Example:

Pickle a namd tuple using 2.7.3 and unpickle it using 2.7.5.

anselm@Emmy:~$ python2.7 Python 2.7.3 (default, Sep 16 2012, 21:46:37) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import collections import pickletools import pickle N=collections.namedtuple("N","a") n=N(1) p=pickle.dumps(n, 2) p2=pickletools.optimize(p) pickletools.dis(p2) 0: \x80 PROTO 2 2: c GLOBAL 'main N' 14: K BININT1 1 16: \x85 TUPLE1 17: \x81 NEWOBJ 18: c GLOBAL 'collections OrderedDict' 43: ] EMPTY_LIST 44: ] EMPTY_LIST 45: ( MARK 46: U SHORT_BINSTRING 'a' 49: K BININT1 1 51: e APPENDS (MARK at 45) 52: a APPEND 53: \x85 TUPLE1 54: R REDUCE 55: b BUILD 56: . STOP highest protocol among opcodes = 2 print repr(p2) '\x80\x02c__main__\nN\nK\x01\x85\x81ccollections\nOrderedDict\n]](U\x01aK\x01ea\x85Rb.'

anselm@Emmy:~/sc/eclipsews/fg2py$ fg2python Python 2.7.5 (default, May 18 2013, 17:02:17) [GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import pickle import collections N=collections.namedtuple("N","a") pickle.loads('\x80\x02c__main__\nN\nK\x01\x85\x81ccollections\nOrderedDict\n]](U\x01aK\x01ea\x85Rb.') Traceback (most recent call last): File "", line 1, in File "/home/anselm/sc/eclipsews/fg2py/arch/rhel4u4-x86_64/lib/python2.7/pickle.py", line 1419, in loads return Unpickler(file).load() File "/home/anselm/sc/eclipsews/fg2py/arch/rhel4u4-x86_64/lib/python2.7/pickle.py", line 895, in load dispatchkey File "/home/anselm/sc/eclipsews/fg2py/arch/rhel4u4-x86_64/lib/python2.7/pickle.py", line 1261, in load_build d = inst.dict AttributeError: 'N' object has no attribute 'dict'

As we can see from the trace back, the problem arises from the pickle op-code 'BUILD'. BUILD requires that the object to be build either has a method setstate or has an attribute dict. Therefore I propose:

or

I created a small *.pth to monkey patch collections.py until 2.7.6 gets released. Maybe this is useful for someone else. Therefore I attach it here.

The pth file runs the following code during Python startup:

import collections def _fix_issue_18015(collections): try: template = collections._class_template except AttributeError: # prior to 2.7.4 _class_template didn't exists return if not isinstance(template, basestring): return # strange if "dict" in template or "getstate" in template: return # already patched lines = template.splitlines() indent = -1 for i,l in enumerate(lines): if indent < 0: indent = l.find('def _asdict') continue if l.startswith(' '*indent + 'def '): lines.insert(i, ' '*indent + 'def getstate(self): pass') lines.insert(i, ' '*indent + 'dict = _property(_asdict)') break collections._class_template = '''\n'''.join(lines) _fix_issue_18015(collections)