[Python-Dev] Sub-interpreters: importing numpy causes hang (original) (raw)

Stephan Reiter stephan.reiter at gmail.com
Tue Jan 22 09:32:22 EST 2019

Hi all!

I am new to the list and arriving with a concrete problem that I'd like to fix myself.

I am embedding Python (3.6) into my C++ application and I would like to run Python scripts isolated from each other using sub-interpreters. I am not using threads; everything is supposed to run in the application's main thread.

I noticed that if I create an interpreter, switch to it and execute code that imports numpy (1.13), my application will hang.

ntdll.dll!NtWaitForSingleObject() Unknown KernelBase.dll!WaitForSingleObjectEx() Unknown

python36.dll!PyCONDWAITMS(PyCONDT * cv=0x00000000748a67a0, RTLCRITICALSECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245 C [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line 370 C python36.dll!PyGILState_Ensure() Line 855 C umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C python36.dll!call_function(_object * * * pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line 4822 C

Numpy's extension umath calls PyGILState_Ensure(), which in turn calls PyEval_RestoreThread on the (auto) threadstate of the main interpreter. And that's wrong. We are already holding the GIL with the threadstate of our current sub-interpreter, so there's no need to switch.

I know that the GIL API is not fully compatible with sub-interpreters, as issues #10915 and #15751 illustrate.

But since I need to support calls to PyGILState_Ensure - numpy is the best example -, I am trying to improve the situation here: https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597

That change may be naive, but it does the trick for my use case. If totally wrong, I don't mind pursuing another alley.

Essentially, I'd like to ask for some guidance in how to tackle this problem while keeping the current GIL API unchanged (to avoid breaking modules).

I am also wondering how I can test any changes I am proposing. Is there a test suite for interpreters, for example?

Thank you very much, Stephan

More information about the Python-Dev mailing list