[Python-Dev] Module renaming and pickle mechanisms (original) (raw)

Alexandre Vassalotti alexandre at peadrop.com
Sat May 17 16:59:05 CEST 2008

On Sat, May 17, 2008 at 5:05 AM, M.-A. Lemburg <mal at egenix.com> wrote:

I'd like to bring a potential problem to attention that is caused by the recent module renaming approach:

Object serialization protocols like e.g. pickle usually store the complete module path to the object class together with the object.

Thanks for bringing this up. I was aware of the problem myself, but I hadn't yet worked out a good solution to it.

It can also happen in storage setups where Python objects are stored using e.g. pickle, ZODB being a prominent example. As soon as a Python 2.6 application starts writing to such storages, Python 2.5 and lower versions will no longer be able to read back all the data.

The opposite problem exists for Python 3.0, too. Pickle streams written by Python 2.x applications will not be readable by Python 3.0. And, one solution to this is to use Python 2.6 to regenerate pickle stream.

Another solution would be to write a 2to3 pickle converter using the pickletools module. It is surely not the most elegant or robust solution, but I could work.

Now, I think there's a way to solve this puzzle:

Instead of renaming the modules (e.g. Queue -> queue), we leave the code in the existing modules and packages and instead add the new module names and package structure with pointers and redirects to the existing 2.5 modules.

This would certainly work for simple modules, but what about packages? For packages, you can't use the sys.modules[__name__] = Queue to preserve module identity. Therefore, pickle will use the new package name when writing its streams. So, we are back to the same problem again.

A possible solution could be writing a compatibility layer for the Pickler class, which would map new module names to their old at runtime. Again, this is neither an elegant, nor robust, solution, but it should work in most cases.

-- Alexandre

More information about the Python-Dev mailing list