[Python-Dev] [Python 2.2 BUG] pickle/cPickle does not find slots (original) (raw)

john coppola john_coppola_r_s@yahoo.com
Fri, 15 Feb 2002 09:57:12 -0800 (PST)

Hi Kevin. I'm with you slots have great potential. First of all, I think it is a very bad idea that slots could be mutable. Here is something I wrote on the python-list. (Python list is too noisy and too many dumb questions involving people not thinking before tackle a problem. -with the exception of an interesting thread on licensing, and a spanish message.)

Anyway, here is some food for thought on slots. Perhaps long-winded and already talked about, but here it is:

(If you want to skip the intro and get to the meat and potatos go to the code, and read my remarks.

Hello Python developers! I have been reading about Python 2.2's new features and I have to say there's lots of good stuff going on. The feature I dedicate this message to is slots. This ubiquitous new feature has the potential to boost performance of python quite considerable and it is the first time python has ever hinted at, for lack of a better word, "being static".

I used this word loosely. What I mean by this is that typically one could set attributes arbitrarily on python objects, but now cometh slots, if slots is defined, one could set only attribute names defined in slots if slots exists.

In introductory essay on the new python object system, "Unifying types and classes in 2_2", I found the benevolent dictator's answer for the existence of slots unsatisfactory (or at least the example). As implied from the introductory essay, slots was used as a means to eliminate a double dictionary for the dictionary subclass in the example (please don't be offended -- SO what). That is one application of slots, but there is a more powerful way to use this new feature.

First let me discuss some notational issues I have with slots. It is possible to use any object that supports iteration as the slots object (huh does my memory serve me correctly?). I feel that this is not a good idea. Consider the following example:

class Foo(object):
    __slots__=['a','b','c']

foo=Foo()

Bad, but you could do it.

foo.__class__.__slots__.append('d') #Ouch!

Suppose someone wrote code to initialize some instance like so:

for attr in foo.__class__.__slots__:
    setattr(foo,attr,0)

Granted this is decidedly bad code but the problem this creates is that attribute "d" does not exist in foo or any instance of Foo before or after the "append event". So an attribute error will be raised. Slot attributes are read only thus del foo.slots is not possible. So neither should slots.append be legal. So the answer is to enforce slots is tuple, or for a single attribute slots is string, when compiling objects.

Another issue I have with slots is that there exists better notation to do the same thing. And part of python is to maintain a clean coding style. Why not introduce new a keyword into the language so "static attribute sets" are created like so:

class Foo(object):
    attribute a
    attribute b
    attribute c

Perhaps this syntax could be expanded further to include strict typing:

class Foo(object):
    attribute a is str
    attribute b is float
    attribute c is int

I feel that this notation is much more lucid and could lead to even more interesting notation:

class Foo(object):
attribute a is str
attribute b is float
attribute c is property
attribute g is property of float

I hope that it is understood from this context that "is" compares the type() to trailing type in the declaration. The meaning of the modifier "of" in the last line is a bit ambiguous at this moment. Perhaps it enforces the type passed into fset, and the return value of fget.

The programmer doesn't need to know what the underlying slots structure looks like, unless they are doing some heavy duty C. In which case they could probably handle it in all it's grand ugliness.

Is the python interpreter slot aware? Instead of the usual variable name lookup into dict (which is not supported by instances who's class implements slots), a slot attribute could be directly referenced via pointer indirection. Does the interpreter substitute references for slot attributes? For that matter any namespace could utilize this new approach, even the local namespace of a code segment. Slots could be built implicitly for a local namespace. This could lead to significant performance boost. Or perhaps a JIT of sorts for python. Cause all you need is a bit of type info, and those segments could be accurately translated into native code.

--- Kevin Jacobs <jacobs@penguin.theopalgroup.com> wrote:

[I tried to post this on SourceForge, but as usual, it hates my guts]

I have been hacking on ways to make lighter-weight Python objects using the slots mechanism that came with Python 2.2 new-style class. Everything has gone swimmingly until I noticed that slots do not get pickled/cPickled at all! Here is a simple test case: import pickle,cPickle class Test(object): slots = ['x'] def init(self): self.x = 66666 test = Test() picklestr = pickle.dumps( test ) cpicklestr = cPickle.dumps( test ) untest = pickle.loads( picklestr ) untestc = cPickle.loads( cpicklestr ) print untest.x # raises AttributeError print untextc.x # raises AttributeError Clearly, this is incorrect behavior. The problem is due a change in object reflection semantics. Previously (before type-class unification), a standard Python object instance always contained a dict that listed all of its attributes. Now, with slots, some classes that do store attributes have no dict or one that only contains what did not fit into slots. Unfortunately, there is no trivial way to know what slots a particular class instance really has. This is because the slots list in classes and instances can be mutable! Changing these lists does not change the object layout at all, so I am unsure why they are not stored as tuples and the 'slots' attribute is not made read-only. To be pedantic, the C implementation does have an immutable and canonical list(s) of slots, though they are well buried within the C extended type implementation. So, IMHO this bug needs to be fixed in two steps: First, I propose that class and instance slots read-only and the lists made immutable. Otherwise, pickle, cPickle, and any others that want to use reflection will be SOL. There is certainly good precedent in several places for this change (e.g., bases, mro, etc.) I can submit a fairly trivial patch to do so. This change requires Guido's input, since I am guessing that I am simply not privy to the method, only the madness. Second, after the first issue is resolved, pickle and cPickle must then be modified to iterate over an instance's slots (or possibly its class's) and store any attributes that exist. i.e., some slots can be empty and thus should not be pickled. I can also whip up patches for this, though I'll wait to see how the first issue shakes out. Regards, -Kevin PS: There may be another problem when when one class inherits from another and both have a slot with the same name. e.g.: class Test(object): slots = ['a'] class Test2(Test): slots = ['a'] test=Test() test2=Test2() test2.class = Test This code results in this error: Traceback (most recent call last): File "", line 1, in ? TypeError: class assignment: 'Test' object layout differs from 'Test2' However, Test2's slot 'a' entirely masks Test's slot 'a'. So, either there should be some complex attribute access scheme to make both slots available OR (in my mind, the preferable solution) slots with the same name can simply be re-used or coalesced. Now that I think about it, this has implications for pickling objects as well. I'll likely leave this patch for Guido -- it tickles some fairly hairy bits of typeobject. Cool stuff, but the rabbit hole just keeps getting deeper and deeper.... -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com

Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev

Do You Yahoo!? Got something to say? Say it better with Yahoo! Video Mail http://mail.yahoo.com