gh-99113: A Per-Interpreter GIL! by ericsnowcurrently · Pull Request #104210 · python/cpython (original) (raw)
well , i changed my library to use this, but end up in ...
I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with my
library. The library is thread safe and supports many languages
-> C C++ JAVA C# VB.Net Perl Python GO TCL Ruby Php
other languages with thread support
-> C C++ JAVA C# TCL GO
works fine.
even python works fine without thread support.
Task: now I started thread support in python, I had to change to Multi-phase initialization
isolate the python data on an per-thread-level and finally gent an easy thread example working
-> ok
static PyInterpreterConfig config = {
.use_main_obmalloc = 0,
.allow_fork = 0,
.allow_exec = 0,
.allow_threads = 1,
.allow_daemon_threads = 0,
.check_multi_interp_extensions = 1,
.gil = PyInterpreterConfig_OWN_GIL,
};
PyThreadState *tstate = NULL;
PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
NOW I start with an more complex example and this stuff fails with a python INTERNAL core
that I can't debug. The stack-trace comes from my MqDisasterSignal handler.
-> question: what can I do to debug this issue?
-> debugging with valgrind is not possible because with valgrind this example works fine.
Debug memory block at address p=0x7f11884def70: API '^@'
18302063728033398269 bytes originally requested
The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
at p-7: 0x00 *** OUCH
at p-6: 0x00 *** OUCH
at p-5: 0x00 *** OUCH
at p-4: 0x00 *** OUCH
at p-3: 0x00 *** OUCH
at p-2: 0x00 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.
The 8 pad bytes at tail=0xfdfe7d0f864bed6d are X> {PRINT :pid(60831):tid(0x7f118819c700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1cb99b8 ):MqDisasterSignal }: BackTrace {
:pid(60831):tid(0x7f118819c700): [ library : filename : lineno ] function
:pid(60831):tid(0x7f118819c700): [ ------- : -------- : ------ ] --------
:pid(60831):tid(0x7f118819c700): [ theLink : c/sys_mq.c : 705 ] MqDisasterSignal
:pid(60831):tid(0x7f118819c700): [ unknown : unknown : 0 ] unknown
:pid(60831):tid(0x7f118819c700): [ system : Objects/obmalloc.c : 2408 ] _PyObject_DebugDumpAddress
:pid(60831):tid(0x7f118819c700): [ system : Objects/obmalloc.c : 2326 ] _PyMem_DebugCheckAddress
:pid(60831):tid(0x7f118819c700): [ system : Objects/obmalloc.c : 2159 ] _PyMem_DebugRawFree
:pid(60831):tid(0x7f118819c700): [ system : Objects/obmalloc.c : 685 ] PyMem_RawFree
:pid(60831):tid(0x7f118819c700): [ system : Objects/obmalloc.c : 1853 ] _PyObject_Free
:pid(60831):tid(0x7f118819c700): [ system : Objects/obmalloc.c : 2163 ] _PyMem_DebugRawFree
:pid(60831):tid(0x7f118819c700): [ system : Objects/obmalloc.c : 2296 ] _PyMem_DebugFree
:pid(60831):tid(0x7f118819c700): [ system : Objects/obmalloc.c : 830 ] PyObject_Free
:pid(60831):tid(0x7f118819c700): [ system : Objects/dictobject.c : 1569 ] dictresize
:pid(60831):tid(0x7f118819c700): [ system : Objects/dictobject.c : 1194 ] insertion_resize
:pid(60831):tid(0x7f118819c700): [ system : Objects/dictobject.c : 1261 ] insertdict
:pid(60831):tid(0x7f118819c700): [ system : Objects/dictobject.c : 1865 ] _PyDict_SetItem_Take2
:pid(60831):tid(0x7f118819c700): [ system : Objects/dictobject.c : 1883 ] PyDict_SetItem
:pid(60831):tid(0x7f118819c700): [ system : Objects/typeobject.c : 7618 ] add_subclass
:pid(60831):tid(0x7f118819c700): [ system : Objects/typeobject.c : 7352 ] type_ready_add_subclasses
:pid(60831):tid(0x7f118819c700): [ system : Objects/typeobject.c : 7515 ] type_ready
:pid(60831):tid(0x7f118819c700): [ system : Objects/typeobject.c : 7553 ] PyType_Ready
:pid(60831):tid(0x7f118819c700): [ system : Objects/typeobject.c : 3795 ] type_new_impl
:pid(60831):tid(0x7f118819c700): [ system : Objects/typeobject.c : 3929 ] type_new
:pid(60831):tid(0x7f118819c700): [ system : Objects/typeobject.c : 1664 ] type_call
:pid(60831):tid(0x7f118819c700): [ system : Objects/call.c : 240 ] _PyObject_MakeTpCall
:pid(60831):tid(0x7f118819c700): [ system : Objects/typeobject.c : 3949 ] type_vectorcall
:pid(60831):tid(0x7f118819c700): [ system : Objects/call.c : 133 ] _PyObject_FastCallDictTstate
:pid(60831):tid(0x7f118819c700): [ system : Objects/call.c : 157 ] PyObject_VectorcallDict
:pid(60831):tid(0x7f118819c700): [ system : Python/bltinmodule.c : 208 ] builtin___build_class__
:pid(60831):tid(0x7f118819c700): [ system : Objects/methodobject.c : 438 ] cfunction_vectorcall_FASTCALL_KEYWORDS
:pid(60831):tid(0x7f118819c700): [ system : Include/internal/pycore_call.h : 92 ] _PyObject_VectorcallTstate
:pid(60831):tid(0x7f118819c700): [ system : Objects/call.c : 325 ] PyObject_Vectorcall
:pid(60831):tid(0x7f118819c700): [ system : Python/bytecodes.c : 2714 ] _PyEval_EvalFrameDefault
:pid(60831):tid(0x7f118819c700): [ system : nclude/internal/pycore_ceval.h : 89 ] _PyEval_EvalFrame
:pid(60831):tid(0x7f118819c700): [ system : Python/ceval.c : 1683 ] _PyEval_Vector
:pid(60831):tid(0x7f118819c700): [ system : Python/ceval.c : 578 ] PyEval_EvalCode
:pid(60831):tid(0x7f118819c700): [ system : Python/pythonrun.c : 1722 ] run_eval_code_obj
:pid(60831):tid(0x7f118819c700): [ system : Python/pythonrun.c : 1743 ] run_mod
:pid(60831):tid(0x7f118819c700): [ system : Python/pythonrun.c : 1643 ] pyrun_file
:pid(60831):tid(0x7f118819c700): [ system : Python/pythonrun.c : 433 ] _PyRun_SimpleFileObject
:pid(60831):tid(0x7f118819c700): [ system : Python/pythonrun.c : 466 ] PyRun_SimpleFileExFlags
:pid(60831):tid(0x7f118819c700): [ theLink : py/MqFactoryC_py.c : 194 ] py_mqmsgque_sFactoryCTor
:pid(60831):tid(0x7f118819c700): [ theLink : c/MqFactoryS_mq.c : 289 ] MqFactoryInvoke_RT
:pid(60831):tid(0x7f118819c700): [ theLink : c/sys_mq.c : 269 ] MqSysServerThreadMain
:pid(60831):tid(0x7f118819c700): [ theLink : c/sys_mq.c : 324 ] sSysServerThreadInit
:pid(60831):tid(0x7f118819c700): [ unknown : unknown : 0 ] unknown
:pid(60831):tid(0x7f118819c700): }
:pid(60831):tid(0x7f118819c700): ^[[1;36mglobal data -----------------------------------------------------^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mruntime : MkRuntimeRLS=0x1cb93b0, MqRuntimeRLS=0x7f11879539c8^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mgeDat : geDat=0x7f1188499de0, geMax=16, geCur=2^[[0;m
:pid(60831):tid(0x7f118819c700): ^[[1;36mevent : event=0x7f1187954610, DataLNum=16, DataLCur=0, geNum=1, fdmax=0^[[0;m
.../NHI1/bin/Nhi1Exec: line 870: 60831 Aborted '.../x86_64-suse-linux-gnu/debug/bin/python3' '.../NHI1/example/py/server.py' '--thread' '--uds' '--file' '/tmp/test.uds'