[Python-Dev] Module renaming and pickle mechanisms (original) (raw)

M.-A. Lemburg mal at egenix.com
Sat May 17 11:05:55 CEST 2008

I'd like to bring a potential problem to attention that is caused by the recent module renaming approach:

Object serialization protocols like e.g. pickle usually store the complete module path to the object class together with the object.

They access this module path by looking at the module attribute of the object classes.

With the renaming, all objects which use classes from the renamed modules will now refer to the renamed modules in their serialized form, e.g. queue.Queue instead of Queue.Queue (just to name one example).

While this is nice for forward compatibility, it causes rather serious problems for making object serialization backwards compatible, since the older Python versions can no longer unserialize objects due to missing modules.

This can happen in client-server setups where e.g. the server uses Python 2.6 and the clients some other Python version (e.g. Python 2.5).

It can also happen in storage setups where Python objects are stored using e.g. pickle, ZODB being a prominent example. As soon as a Python 2.6 application starts writing to such storages, Python 2.5 and lower versions will no longer be able to read back all the data.

Now, I think there's a way to solve this puzzle:

Instead of renaming the modules (e.g. Queue -> queue), we leave the code in the existing modules and packages and instead add the new module names and package structure with pointers and redirects to the existing 2.5 modules.

Code can (and probably should) still be changed to try to import the new module name. In cases where backwards compatibility is needed, this can also be done using

try: import newname except ImportError: import oldname

Later on, when porting applications to 3.0, the 2to3 script can then apply the final renaming in the source code.

Example:

queue.py:

import sys, Queue sys.modules[name] = Queue

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, May 17 2008)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::

eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
        Registered at Amtsgericht Duesseldorf: HRB 46611

More information about the Python-Dev mailing list