[Python-Dev] PEP 311 Simplified Global Interpreter Lock Acquisition for Extensions (original) (raw)

Harri Pasanen harri.pasanen at trema.com
Mon Aug 18 20:39:05 EDT 2003

Hi,

Just prior to my vacation I ran into a bit of trouble with my Python embedding, as much to my dismay my C++ program that had been working fine under Linux deadlocked under Windows.

Apparently the underlying C threading API's were behaving in subtly different ways. After boiling down from my original embedded Python app which features omniORB, omniORBpy, and bunch of other goodies, I got down to the following testcase:

#include <Python.h>

class PythonLocker { public: PythonLocker () { _tstate = PyGILState_Ensure (); }

~PythonLocker() { PyGILState_Release(_tstate); } } private: PyGILState_STATE _tstate; };

int main (int argc, const char *argv) { Py_Initialize(); PyEval_InitThreads(); PyEval_ReleaseLock (); // This is the problem { PythonLocker pl; PyObject usermod = PyImport_ImportModule ("echo");
// The previous line deadlocks on win32 } return 0; }

The above works fine on Linux, but deadlocks on Win32.

Tracing the execution on Win32 showed that it hanged in thread_nt.h: DWORD EnterNonRecursiveMutex(PNRMUTEX mutex, BOOL wait) { ... ret = InterlockedIncrement(&mutex->owned) ? /* Some thread owns the mutex, let's wait... */ WaitForSingleObject(mutex->hevent, INFINITE) : WAIT_OBJECT_0 ;

The gotcha being that when entering above, mutex->owned was == -2, so it incorrectly entered the wait.

I had a private discussion with Mark about this, and he quickly pointed out that I should not call PyEval_ReleaseLock(). Although this is not exactly clear from the PEP, which says

"Apart from the existing, standard Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros, it is assumed that no additional thread state API functions will be used by the extension."

I did not think PyEval_ReleaseLock() would qualify for the above, but somehow it does, under win32 at least.

Now if I modify thread_nt.h, simple mindedly by changing the above to:

    ret = InterlockedIncrement(&mutex->owned) > 0 ?
        /* Some thread owns the mutex, let's wait... */
        WaitForSingleObject(mutex->hevent, INFINITE) : WAIT_OBJECT_0 ;

everything works for me on Win32 as well, including python regression tests. But I haven't wrestled with the code in question in detail to know what gremlins that change might unleash elsewhere.

Without modifying Python, I can make my testcase working with the following change to my test program:

{ Py_Initialize(); PyEval_InitThreads(); { PythonLocker init_pl }; // release the lock // PyEval_ReleaseLock (); { // this block could potentially be called from another thread PythonLocker pl; PyObject* usermod = PyImport_ImportModule ("echo");
// The previous line deadlocks on win32 }

In the C API world this is a bit more convoluted looking, as I need to call PyGILState_Ensure(), and PyGILState_Restore, even if after PyEval_InitThreads() I already know I have the lock.

Sorry for rambling, but this is in the hope that someone else can avoid banging their heads against this as I have. And it is a bit unclear to me also if this kind of a platform difference between Linux and Win32 is a bug, or a feature. Ideally the Python C API should be bug-compatible. So possibly wrong usage should cause all platforms to fail.

Btw., the new PyGILState_* functions do not appear to be part of the Python/C API documentation, there is only the PEP.

Anyway, thanks to Mark for this definite improvement on the state of things.

Harri

More information about the Python-Dev mailing list