[Python-Dev] Evil reference cycles caused Exception.traceback (original) (raw)

Martin Panter vadmium+py at gmail.com
Mon Sep 18 08:56:40 EDT 2017

On 18 September 2017 at 09:31, Victor Stinner <victor.stinner at gmail.com> wrote:

Last years, I fixed many reference cycles in various parts of the Python 3 standard library. Sometimes, it takes years to become aware of the reference cycle and finally fix it.

For example, recently, I worked on fixing all "dangling threads" leaked by tests of the Python test suite, and I found and fixed many reference cycles which probably existed since Python 3 was created (forked from Python 2)

My instinct is to suggest to make the “dangling threads” test tolerate Thread objects that are no longer running. Last time I looked, it tests a list of weak references to Thread objects. But maybe it should check “Thread.is_alive” or use “threading.enumerate”.

. . .

Ideally, CPython 3.x should never create reference cycles. Removing Exception.traceback is the obvious "fix" for the issue. But I expect that slowly, a lot of code started to rely on the attribute, maybe even for good reasons :-) A more practical solution would be to log a warning. Maybe the garbage collector can emit a warning if it detects an exception part of a reference cycle? Or maybe detect frames?

The “gc” module can do some of this already. In the past (when del used to prevent garbage collection) I used DEBUG_SAVEALL to and graphed the result to figure out where code was creating reference cycles. Or maybe if you turned on DEBUG_COLLECTABLE and filtered the objects printed out, or played with the new “gc.callbacks” API.

If the GC cannot do it, maybe we might use a debug thread (enabled manually) which checks manually if an exception is part of a reference cycle using gc.getobjects(): check if an exception remains alive longer than X seconds? I had the same idea for asyncio, to detect reference cycles or if a task is never "awaited", but I never implemented the idea.

More information about the Python-Dev mailing list