Issue 921077: embedding in multi-threaded & multi sub-interpreter environ (original) (raw)

I am embedding python in my C++ application. I am using Python 2.3.2 with a C++ extention DLL in multi- threaded environment. I am using SWIG-1.3.19 to generate C++ to Python interface.

Now to explain it in details,

Python initialization [Py_Initialize()] and finalization [Py_Finalize()] is done in the main thread.
For each new thread I create a separate sub- interpreter [Py_NewInterpreter()].
Using PyRun_String("import myModule"...) before execution of python script, extention module is imported.
Each thread executes multiple python script using PyEval_EvalCode() using the class objects in my extention DLL.
Each sub-interpreter is destroyed [Py_EndInterpreter ()] at the end of that particular thread.

I am observing that; As explained above when multiple threads are running. And as one of these threads finishes, in other running threads I start getting "TypeError: 'NoneType' object is not callable" error on the methods called on class objects in extention module.

The same code works fine with Python 2.2.2.

I have found these links more or less talking about the same problem migrating from 2.2 to 2.3. http://mail.python.org/pipermail/python-dev/2003- September/038237.html http://mail.python.org/pipermail/python-list/2004- February/206851.html http://mail.python.org/pipermail/python-list/2004- January/204040.html

I guess what is happening is global variables are zapped to "NoneType" when one thread finishes and other thread trying to access them through the Python script (step 4.) this error is generated. But it works sometimes when(guess) the running thread is at step 3. and by importing the module the global variables are re-initialized for "Type" information.

I tried using reload(myModule) to solve the problem but that is generating big memory leak every time it is called.

Is this a know issue with 2.3 (interpreter?) ? Or is there a change for 2.3 in the way embedding should be done in a multi-threaded and multi-sub-interpreter environment ?

Logged In: YES user_id=1003784

Hello Martin,

I am attaching the sample program which demonstrates the issue;

I compiled it on Windows 2000 with VC++ compiler.
There is a "Extension Module" [flpythonmodulesu(d).dll] which is used by "Main" program

========================================= Extension Module (extModule.dsp project) generates : (u. - release, ud. - debug) 1. dll _flpythonmodulesud.dll 2. python script flpythonmodulesud.py

Module has class FlPythonString and class Record defined
Using interface file include\flpythonmodulesud.i, SWIG- 1.3.19 generates include\flpythonmodulesud_wrap.cxx

========================================= Main sample program (multiSub.dsp project) generates : 1. executable multiSub.exe

Links in the extension DLL
Initializes and De-initializes python in main thread
Main thread creates 2 separate threads: (a) Each thread creating a new sub-interpreter (b) importing extension module (c) Calling following EXPRESSION using PyEval_EvalCode(), Thread 1:

myVar = FlPythonStringPtr(STRObject) myRecord = myVar.NewRecord() Thread=str(myVar.Get()) record=str(myRecord.GetValue()) print 'Inside EXPRESSION ' + Thread + ':' + record;myVar.DeleteRecord(myRecord)

Thread 2:

myVar = FlPythonStringPtr(STRObject) for recordNum in range(1, 11): myRecord = myVar.NewRecord() Thread=str(myVar.Get()) record=str(myRecord.GetValue()) print 'Inside EXPRESSION ' + Thread + ':' + record;myVar.DeleteRecord(myRecord) (d) Destroying sub-interpreter.

Comments in the code explain details about each step.

========================================== Console Output messages explanations (Attaching a sample console output for the issue [consoleop.txt])

-Thread ID:IMPORT Module Called:ITERATION -Inside EXPRESSION Thread:Record::GetValue() [hardcoded to 1000] -Thread ID:EXPRESSION Done:ITERATION -Thread ID:ENDINTERPRETER Called -Thread ID:EXPRESSION Failed!:ITERATION

Check the error message demonstrating the issue:

"Thread 2:EXPRESSION Failed!:3 Traceback (most recent call last): File "EXPRESSION", line 4, in ?

File "E:\VssLocal\Isis\python23_issue\multiSub\Debug\flpython modulesud.py", line 69, in Get def Get(*args): return apply (_flpythonmodulesud.FlPythonString_Get,args)

File "E:\VssLocal\Isis\python23_issue\multiSub\Debug\flpython modulesud.py", line 51, in init _swig_setattr(self, Record, 'this', this) TypeError: 'NoneType' object is not callable"

==========================================

Martin, Thanks for taking a look at the issue which has proved to be a major hurdle for our release and made us switch back to Python 2.2.2. Python 2.2.2 in itself has some issues for us; so if this can get fixed we would desperately be seeking a patch on Python 2.3.

Let me know if you need any more inputs from my side.