| msg257524 - (view) |
Author: Anil Kulkarni (anilredshift) |
Date: 2016-01-05 07:39 |
| Pickles created with python 3.4.X will not load with python 3.5.X if they include a collections.OrderedDict To reproduce this issue, simply create a pickle of an OrderedDict on python 3.4.3 with protocol=2 and try to open it on 3.5. I have included a simple script to demonstrate this issue. I believe this is related to this bug: https://bugs.python.org/issue18473 As to the real-world implications: The python package Celery uses protocol=2 by default when serializing with pickle, and thus a celery web running 3.5 cannot receive the results of a worker running 3.4 For celery specifically, there is a workaround by setting the PICKLE_PROTOCOL environment variable, but this is a core python issue. P.S. This is the first bug I've filed with python so please let me know if there's something else I should be including. Thanks! |
|
|
| msg257526 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2016-01-05 09:18 |
| Thank you for your report Anil. Python 3.4.3 creates incorrect pickle with protocol 2 (it can't be load in Python 2). fixed pickling with protocol 2 in Python 3, but broke loading broken pickles created in unpatched versions of Python 3. Here is a patch that allows to load such broken pickles. But it is too late to fix Python 3.4, it can get only security fixes. You have Celery specific workaround. More general workaround is to update _compact_pickle.IMPORT_MAPPING, e.g.: import _compact_pickle _compact_pickle.IMPORT_MAPPING.update({ 'UserDict': 'collections', 'UserList': 'collections', 'UserString': 'collections', 'whichdb': 'dbm', 'StringIO': 'io', 'cStringIO': 'io', }) Note that you have to set mapping not just for one module name like 'UserList', but for all 'UserDict', 'UserList', and 'UserString', because it is not predicable to what module name 'collections' is mapped in Python 3.4.3. |
|
|
| msg257541 - (view) |
Author: Anil Kulkarni (anilredshift) |
Date: 2016-01-05 19:53 |
| Hi Serhiy, I have done some more investigation this morning and I have come across two distinct issues. The first is that pickles (specifically the pickle in my earlier message) created in python 3.0->3.4.3 do not load on python 2.7.10. The exception is 3.5.1, which produces a pickle that DOES load on 2.7.10 This is unfortunate but not a regression The second issue is that pickles created by python 3.0->3.4.3 do NOT load in python 3.5 This is a regression, in the sense that the pickle created by 3.0->3.4.3 is compatible with every other version of python (I haven't tested every single combination here, but several) From the language perspective, it may be that 3.4.X has incorrect code, but from a compatibility perspective, 3.5 is breaking the promise that pickles are compatible across versions of python. I write this as justification that python 3.5 should fix this regression in compatibility with 3.0 -> 3.4.3. At the very least 3.5 should have a shim behavior to fallback and allow it to import these pickles correctly. Thanks, Anil |
|
|
| msg257557 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2016-01-05 21:23 |
| I agree, and the provided is purposed to fix this issue. In the previous message I just proposed a workaround for those who can't wait 3.5.2. |
|
|
| msg257558 - (view) |
Author: Anil Kulkarni (anilredshift) |
Date: 2016-01-05 21:57 |
| Ah, sorry I misunderstood. Thanks for the quick turnaround! |
|
|
| msg258540 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2016-01-18 19:36 |
| New changeset 270add17f203 by Serhiy Storchaka in branch '3.5': Issue #26013: Added compatibility with broken protocol 2 pickles created https://hg.python.org/cpython/rev/270add17f203 New changeset 35ff0976b211 by Serhiy Storchaka in branch 'default': Issue #26013: Added compatibility with broken protocol 2 pickles created https://hg.python.org/cpython/rev/35ff0976b211 |
|
|
| msg258544 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-01-18 20:21 |
| Buildbots are unhappy. http://buildbot.python.org/all/builders/PPC64%20Fedora%203.5/builds/344/steps/test/logs/stdio ====================================================================== FAIL: test_reverse_import_mapping (test.test_pickle.CompatPickleTests) [('cStringIO', 'io')] ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/test/test_pickle.py", line 353, in test_reverse_import_mapping (module3, module2)) AssertionError: No reverse mapping from 'io' to 'cStringIO' |
|
|
| msg258545 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-01-18 20:34 |
| New changeset 7215d13baa2d by Serhiy Storchaka in branch '3.5': Added exceptins for testing non-reversible import mapping for Issue #26013. https://hg.python.org/cpython/rev/7215d13baa2d New changeset 16cfc1652844 by Serhiy Storchaka in branch 'default': Added exceptins for testing non-reversible import mapping for Issue #26013. https://hg.python.org/cpython/rev/16cfc1652844 |
|
|
| msg258546 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2016-01-18 20:46 |
| Thank you Victor. |
|
|