[Python-Dev] Threading and callbacks - missing documentation (original) (raw)

Tim Peters tim.one@comcast.net
Fri, 12 Apr 2002 04:36:54 -0400

[Jack Jansen]

unless I'm looking in completely the wrong place (I looked in "extending and Embedding" and in "Python/C API") there is absolutely no information in the docs on the interaction between callbacks and the GIL.

Just the Prime Directive: you must not call back into Python unless you hold the GIL. Terrible things will happen otherwise.

Actually getting the GIL can be a nightmare. The worst example in the core is posixmodule.c's _PyPclose, where a thread has to get the GIL, but has no thread state to use, and has no interpreter state to use either (in order to get a thread state). It has to bootstrap all that stuff into existence, then tear it down again when it's done.

There are two saving graces in this specific routine that allow the obscure dance to work: (1) it knows for sure that Python has already been initialized; and, (2) it knows for sure that the thread calling it does not hold the GIL at the time of the call. If you can't count on both of those, it can get much harder. Anyway, the comments there will be a real help.

There is preciously little information on callbacks and threading, and it isn't very helpful at that (i.e. there doesn't seem to be a recipy on how to obtain the GIL in a callback routine that is about to call Python code,

It would take a small book to cover all the cases that can arise. You can find tens of thousands of words about how to do this in the Thread-SIG archives, some attempts at helper frameworks, and a few forgotten promises to make it easier someday. I believe Mark Hammond has a general set of C++ classes to help with this stuff on Windows, but IIRC they rely on Windows-specific TLS (thread local storage) gimmicks.

or how to test whether the current thread holds the GIL already).

Unfortunately, this isn't possible: if you need to do this, you need to build your own conventions for keeping track of who has the GIL.

[Thomas Heller, to Martin]

Can't you at least partly combine the advantages of 1 and 2 by using thread local storage for the global variable? I have no idea if there is a portable way to do this...

There isn't a portable way. The closest Python gets is the little-known PyThreadState_GetDict(), which returns a dict object unique to the thread calling it (so each thread can set up its own key->value mappings for a common set of keys). However, the calling thread must hold the GIL (or, as always, terrible things can happen ). So it's useful only if you already already know whether you have the GIL, and how to get the GIL if you don't have it.