Issue 931877: Segfault in object_reduce_ex (original) (raw)

Shane Hathaway bumped into this, unbounded recursion in typeobject.c's object_reduce_ex(). This occurs in Python 2.3.3 and current CVS.

Assigned to Guido, to ponder whether object_reduce_ex is doing what it should; if it is (which seems likely to me), I suppose we need to inject a recursion counter to prevent the segfault.

The failing case is short, but I'll attach it (temp99.py) to avoid SF line mangling. While the test uses pickle, same symptom if it's changed to use cPickle instead.

Jim Fulton's analysis:

""" This is a very clever infinite loop. The proxy doesn't actually proxy, but it does manage to confuse reduce about what's going on.

reduce tries to figure out if it has been overridden by asking whether the class's reduce is the same as object.reduce. It doesn't expect to be lied to about the class. Things wouldn't have been so bad if the proxy had proxied reduce as well as class. """

The priority hasn't been bumped, because "the real code" from which this was whittled down wasn't doing what it needed to do anyway, and the recursion went away when the real code was repaired.

Logged In: YES user_id=31435

Hmm! The temp99.py download link doesn't work for me.
Here's the content in case it doesn't work for others either:

""" import cPickle as pickle from pickle import dumps

class SimpleItem: def reduce(self): return (self.class, None, {})

class Proxy(object): class = property(lambda self: self.__target.class)

def __init__(self, target):
    self.__target = target

def __getstate__(self):
    raise RuntimeError("No don't pickle me!  Aaarrgghh!")

p = Proxy(SimpleItem()) dumps(p) """

Logged In: YES user_id=63133

This infinite recursion also occurs in another place, that got me stumped for a couple of days when old code that worked with Python 2.2 stopped working. If class is not fidgeted with (as in original bug report), but a descriptor returns a custom reduce for the class, but not for its objects, reduce enters an infinite loop on the object:

""" class descriptor_for_reduce(object): def get(self,obj,tp=None): if obj is not None: return super(ASpecialClass,obj).reduce return self.reducer def reducer(self,proto=None): return "VerySpecial"

class ASpecialClass(object): reduce = descriptor_for_reduce()

copy.copy(ASpecialClass()) """

ASpecialClass().reduce is object.reduce, which is implemented by typeobject.c:object_reduce_ex. This function (that doesn't know if its called as the reduce or the reduce_ex method) tries to detect if the object's reduce is overridden. It does so by checking if the object's class's reduce is overridden, and in fact it is. It then assumes that the object's reduce is overridden, and calls it. But the object's reduce is the same function, causing the infinite loop.

If reduce_ex is used instead of reduce, the problem goes away, ASpecialClass().reduce_ex() return the usual tuple, and ASpecialClass.reduce_ex() return "VerySpecial". But when reduce is overridden, ASpecialClass().reduce() enters an infinite loop.

I believe this is a legitimate example that should behave just as when reduce_ex is overridden. The example doesn't lie about class, and it is certainly legitimate for define a property that behaves differently for the class and for its objects.

Where did this come up and why would I ever care about a class's reduce? The reduce attribute of a class is never used by (the standard) pickle or copy, since save_global() is called instead. However, I have a custom pickler, implemented as a subclass of pickle.Pickler, which falls back on the class's reduce when save_global() fails. This way, I can pickle certain classes that are created at run-time (and can be easily recreated, e.g. from their bases and dictionaries).