[Python-Dev] Evil reference cycles caused Exception.traceback (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Mon Sep 18 07:40:40 EDT 2017

On 18 September 2017 at 20:52, Antoine Pitrou <solipsis at pitrou.net> wrote:

On Mon, 18 Sep 2017 20:35:02 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote:

Rather than being thread local or context local state, whether or not to keep the full frames in the traceback could be a yet another setting on the exception object, whereby we tweaked the logic that drops the reference at the end of an except clause as follows: That doesn't solve the problem, since the issue is that exceptions can be raised (and then silenced) in many places, and you don't want such exception-raising code (such as socket.createconnection) to start having to set an option on the exceptions it raises.

Bleh, in trying to explain why my proposal would be sufficient to break the problematic cycles, I realised I was wrong: if we restrict the frame clearing to already terminated frames (as would be necessary to avoid breaking any still executing functions), then that means we won't clear the frame running the exception handler, and that's the frame that creates the problematic cyclic reference.

However, I still think it makes more sense to focus on the semantics of preserving an exception beyond the life of the stack being unwound, rather than on the semantics of raising the exception in the first place.

In the usual case, the traceback does keep the whole stack alive while the stack is being unwound, but then the exception gets thrown away at the end when sys.exc_info() gets reset back to (None, None, None), and then all the frames still get cleaned up fairly promptly (this is also the case in Python 2).

We only get problems when one of the exception handlers in the stack grabs an additional reference to either the traceback or the exception and hence creates a persistent cyclic reference from one of the frames back to itself. The difference in Python 3 is that saving the exception is reasonably common, while explicitly saving the traceback is relatively rare, so the "exc.traceback" is keeping tracebacks alive that would otherwise have been cleaned up more deterministically.

Putting the problem that way gave me an idea, though: what if, when the interpreter was setting "sys.exc_info()" back to (None, None, None) (or otherwise dropping an exception instance from being the "currently active exception") it automatically set exc.traceback to None?

That way, if you wanted the traceback (rather than just the exception) to live beyond the stack being unwound, you'd have to preserve the entire sys.exc_info() triple (or at least save "exc.traceback" separately from "exc"). By doing that, we'd have the opportunity to encourage folks that are considering preserving the entire traceback to extract a TracebackException instead and some themselves from some potentially nasty reference management issues: https://docs.python.org/3/library/traceback.html#tracebackexception-objects

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list