[Python-Dev] [Python 2.2 BUG] pickle/cPickle does not find slots (original) (raw)

Kevin Jacobs jacobs@penguin.theopalgroup.com
Fri, 15 Feb 2002 09:57:19 -0500 (EST)

[I tried to post this on SourceForge, but as usual, it hates my guts]

I have been hacking on ways to make lighter-weight Python objects using the slots mechanism that came with Python 2.2 new-style class. Everything has gone swimmingly until I noticed that slots do not get pickled/cPickled at all!

Here is a simple test case:

import pickle,cPickle class Test(object): slots = ['x'] def init(self): self.x = 66666

test = Test()

pickle_str = pickle.dumps( test ) cpickle_str = cPickle.dumps( test )

untest = pickle.loads( pickle_str ) untestc = cPickle.loads( cpickle_str )

print untest.x # raises AttributeError print untextc.x # raises AttributeError

Clearly, this is incorrect behavior. The problem is due a change in object reflection semantics. Previously (before type-class unification), a standard Python object instance always contained a dict that listed all of its attributes. Now, with slots, some classes that do store attributes have no dict or one that only contains what did not fit into slots.

Unfortunately, there is no trivial way to know what slots a particular class instance really has. This is because the slots list in classes and instances can be mutable! Changing these lists does not change the object layout at all, so I am unsure why they are not stored as tuples and the 'slots' attribute is not made read-only. To be pedantic, the C implementation does have an immutable and canonical list(s) of slots, though they are well buried within the C extended type implementation.

So, IMHO this bug needs to be fixed in two steps:

First, I propose that class and instance slots read-only and the lists made immutable. Otherwise, pickle, cPickle, and any others that want to use reflection will be SOL. There is certainly good precedent in several places for this change (e.g., bases, mro, etc.) I can submit a fairly trivial patch to do so. This change requires Guido's input, since I am guessing that I am simply not privy to the method, only the madness.

Second, after the first issue is resolved, pickle and cPickle must then be modified to iterate over an instance's slots (or possibly its class's) and store any attributes that exist. i.e., some slots can be empty and thus should not be pickled. I can also whip up patches for this, though I'll wait to see how the first issue shakes out.

Regards, -Kevin

PS: There may be another problem when when one class inherits from another and both have a slot with the same name.

 e.g.:
   class Test(object):
     __slots__ = ['a']

   class Test2(Test):
     __slots__ = ['a']

   test=Test()
   test2=Test2()
   test2.__class__ = Test

This code results in this error:

  Traceback (most recent call last):
    File "<stdin>", line 1, in ?
  TypeError: __class__ assignment: 'Test' object layout differs from 'Test2'

However, Test2's slot 'a' entirely masks Test's slot 'a'.  So, either
there should be some complex attribute access scheme to make both slots
available OR (in my mind, the preferable solution) slots with the same
name can simply be re-used or coalesced.  Now that I think about it,
this has implications for pickling objects as well.  I'll likely leave
this patch for Guido -- it tickles some fairly hairy bits of typeobject.

Cool stuff, but the rabbit hole just keeps getting deeper and deeper....

-- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com