[Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages) (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Fri Nov 5 14🔞45 CET 2010

On Fri, Nov 5, 2010 at 10:56 AM, Steven D'Aprano <steve at pearwood.info> wrote:

Nick Coghlan wrote:

As a tool for communicating between different instances of the same version of Python though, pickle is fine. I'm using pickle to pass a list and dict of floats and strings from Python 2.6 to 3.1. I've never had any problems with it. Am I living in a state of sin or is that okay?

Builtins are generally fine, and we do try reasonably hard to keep the pickle formats properly compatible across versions. It's corner cases (like pickling unittest objects) that may sometimes break, since pickle implicitly depends on things that should be disregarded as implementation details. Specifically, without explicit directions to do otherwise, pickle encodes objects based on what they are, which may include implementation details, such as optional acceleration modules, platform specific variants of classes returned by a factory function, etc. Technically such things are bugs in an object's pickling support, but they're really non-obvious (and genuinely harmless in most cases).

As I see it, there are at least 3 levels of pickling support:

  1. Complete, version independent (implementation details are weeded out from the pickle, or deliberately kept the same across versions to preserve pickle compatibility)
  2. Partial, potentially version dependent (pickles may be infected with implementation details that affect cross-version compatibility if they happen to change)
  3. None (can't even pickle it in the first place)

Builtins are in category 1, but there are plenty of things in the standard library (like unittest classes) that rely on default pickling behaviour and hence fit into category 2 (we just very, very rarely move anything around, so such classes may as well be in category 1 most of the time).

Notably, this mostly causes problems when reading pickles generated with a new version of Python in an old version. When going the other way, we can adjust the unpickling process to cope with any discrepancies (for the "relying on implementation details case", usually by the simple expedient of keeping both sets of names around).

Cheers, Nick.

-- Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-Dev mailing list