[Python-Dev] Bug in generator if the generator in created in a C thread (original) (raw)

Victor Stinner victor.stinner at gmail.com
Thu Mar 29 23:26:42 CEST 2012

On 29/03/2012 21:35, Tim Lesher wrote:

From a theoretical standpoint, I can't quite decide what the real error is:

1) the fact that PyGILStateRelease() destroys a temporary thread state that may still be referenced by some objects, or 2) the fact that some code is trying to keep frame objects after the creating thread state no longer exists. This week I've been leaning toward 2), but then I realized that keeping frames post-thread-death is not that uncommon (for example, debuggers and other diagnostic techniques like http://nedbatchelder.com/blog/200711/rethrowingexceptionsinpython.html).

The problem is not the frame, but the Python thread state referenced by the frame. It's a private attribute. My patch just updates this reference before running the generator (and it clears the reference when the generator excution is stopped or finished).

Locally we added some (unfortunate) code to our 3.1.2 port to wrap PyGILStateEnsure(), which I thought had sidestepped the issue for us:

void takeGIL() { PyGILStateEnsure(); // This has the side effect of keeping such thread states alive until // the interpreter is finalized; however, all thread state objects get // unconditionally deleted during PyFinalize, so they won't leak. PyThreadState* pThreadState = PyGILStateGetThisThreadState(); if (pThreadState->gilstatecounter == 1) { ++pThreadState->gilstatecounter; } } But clearly that can't be a correct answer (and it may not even be a functioning one, given that I'm seeing a similar issue again).

You may leak memory if your threads have a short lifetime and you create many threads. For example if one thread is only used to process one request and then is destroyed.

Victor

More information about the Python-Dev mailing list