[Python-Dev] Extension modules, Threading, and the GIL (original) (raw)

David Abrahams dave@boost-consulting.com
Sun, 29 Dec 2002 20:29:19 -0500

Recently an issue has come up on the C++-sig which I think merits a little attention here. To boil it down, the situation looks like this:

Shared library Q uses threading but not Python. It supplies a an interface by which users can supply callback functions. Some of these callbacks will be invoked directly in response to external calls into Q; others will be invoked on threads started by calls into Q.

Python extension module A calls shared library Q, but doesn't use its callback interface. It works fine by itself.

Python extension module B calls shared library Q and uses Q's callback interface. Because some of the callbacks need to use the Python API, and might be invoked by threads, they must all acquire the GIL. Because they also might be invoked by direct calls into Q, B must always release the GIL before calling anything in Q.

Problem: using B while A is loaded breaks A: because B has installed callbacks in Q that acquire the GIL, A must also release the GIL before calling into Q.

Notice that the author of A may have had no reason to believe anyone would install Python callbacks in Q!

It seems to me that for situations like these, where a function may or may not be called on Python's main thread, it would be helpful if Python supplied a "recursive mutex" GIL acquisition/release pair, for which acquisition and release on the main thread would simply bump a counter. Is this something that was considered and rejected?

TIA, Dave

-- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution