[Python-Dev] Bug in generator if the generator in created in a C thread (original) (raw)

Tim Lesher tlesher at gmail.com
Fri Mar 30 14🔞06 CEST 2012

On Thu, Mar 29, 2012 at 17:26, Victor Stinner <victor.stinner at gmail.com> wrote:

The problem is not the frame, but the Python thread state referenced by the frame. It's a private attribute. My patch just updates this reference before running the generator (and it clears the reference when the generator excution is stopped or finished).

Right--my thought was the the situation we saw might be similarly triggered because we were storing an exception with traceback (and associated frames) generated by thread A, and then re-throwing it from thread B some time after thread A has exited. The frame attached to the exception's traceback would still, then, be referencing a nonexistent thread state, correct?

My concern was that this might one instance of a broader problem for folks who embed Python and see the attractive PyGILState_Ensure() API. It already prevents us from using subinterpreters (which for us might have been a better solution than repeated initialize/finalize, with its known issues).

We recently made a change to dispose of the traceback before storing the exception, and that appears to eliminate the corruption we were seeing, so that's making me more suspicious.

You may leak memory if your threads have a short lifetime and you create many threads. For example if one thread is only used to process one request and then is destroyed.

Absolutely--this particular hack was only used for a thread created outside Python that had to call into the VM. Their behavior is well-defined in our case--two particular OS threads, with lifetimes longer than those of the interpreters we create and finalize.

-- Tim Lesher <tlesher at gmail.com>

More information about the Python-Dev mailing list