[Python-Dev] PyObject_GC_UnTrack() no longer reliable in 2.7? (original) (raw)

Jim Fulton jim at zope.com
Fri Sep 24 22:07:02 CEST 2010

On Fri, Sep 24, 2010 at 3:36 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

On Fri, 24 Sep 2010 15:14:32 -0400 Tim Peters <tim.peters at gmail.com> wrote:

Looks like 2.7 changes introduced to exempt dicts and tuples from cyclic gc if they obviously can't be in cycles has some unintended consequences.  Specifically, if an extension module calls PyObjectGCUnTrack() on a dict it does not want tracked, Python can start tracking the dict again.

I assume this is unintended because (a) the docs weren't changed to warn about this; and, (b) it's wrong ;-) It was indeed unintended. I didn't know people were using PyObjectGC(Un)Track in other places than constructors and destructors. There are two main reasons an extension module may have been calling PyObjectGCUnTrack(): 1. For correctness, if the extension is playing games with reference counts Python isn't expecting. Yikes :) 2. For speed, if the extension is creating dicts (or tuples) it knows cannot participate in cycles. The optimization is now automated in the simple cases (as you've found out!). This came up when Jim Fulton asked me for advice about assertion failures coming out of cyclic gc in a debug build when running ZODB's tests under 2.7.  Turned out to be due to the "#1 reason" above:  ZODB hand-rolled its own version of weak references long before Python had them, and has a dict mapping strings ("object IDs") to complex objects where the referenced objects' refcounts intentionally do not account for the references due to this dict. Perhaps ZODB should switch to standard weak references these days? (as a bonus, chances are it will be faster)

This is the long term plan. Switching is not going to be a small project and not high on the list of priorities.

(Actually, ZODB invented its own weakref mechanism after Python had weakrefs, but before weakrefs were subclassable. Using standard weakrefs was deemed too expensive in terms of memory use.)

For the record, I don't consider this a Python bug. This corner of ZODB is living on the edge and deserves what it gets. :) I'm just happy the fix was ultimately pretty simple.

Best if no changes had been needed.  "Better than nothing" if the docs are changed to warn that the effect of calling PyObjectGCUnTrack() may be undone by Python a nanosecond later ;-) A doc addition will be enough, hopefully.

Absolutely.

Jim

-- Jim Fulton

More information about the Python-Dev mailing list