[Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC... (original) (raw)

Kristján Valur Jónsson kristjan at ccpgames.com
Fri Dec 21 10:31:44 CET 2012

I ran into this the other day. I had put in hooks in the PyMem_MALLOC to track memory per tasklet, and it crashed in those cases because it was being called without the GIL. My local patch was simply to not release the GIL. Clearly, calling PyMem_MALLOC without the GIL is an API violation.

K

-----Original Message----- From: Python-Dev [mailto:python-dev-_ _bounces+kristjan=ccpgames.com at python.org] On Behalf Of Trent Nelson Sent: 21. desember 2012 03:13 To: Gregory P. Smith Cc: Python-Dev Subject: Re: [Python-Dev] Possible GIL/threading issue involving subprocess and PyMemMALLOC...

On Thu, Dec 20, 2012 at 05:47:40PM -0800, Gregory P. Smith wrote: > On Thu, Dec 20, 2012 at 10:43 AM, Trent Nelson <trent at snakebite.org> > wrote: > > This seems odd to me so I wanted to see what others think. The unit > test Lib/unittest/test/testrunner.py:TestTextRunner.testwarnings > will eventually hit subprocess.Popen.communicate. > > The mswindows implementation of this method relies on threads to > buffer stdin/stdout. That'll eventually result in > PyOsStdioReadline > being called without the GIL being held. PyOsStdioReadline calls > PyMemMALLOC, PyMemFREE and possibly PyMemREALLOC. > > Those threads are implemented in Python so how would the GIL ever not be > held? > -gps PyOSReadline drops the GIL prior to calling PyOSStdioReadline: PyBEGINALLOWTHREADS --------^^^^^^^^^^^^^^^^^^^^^^ #ifdef WITHTHREAD PyThreadacquirelock(PyOSReadlineLock, 1); #endif /* This is needed to handle the unlikely case that the * interpreter is in interactive mode and stdin/out are not * a tty. This can happen, for example if python is run like * this: python -i < test1.py */ if (!isatty (fileno (sysstdin)) || !isatty (fileno (sysstdout))) rv = PyOSStdioReadline (sysstdin, sysstdout, prompt); ---------------- -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ else rv = (*PyOSReadlineFunctionPointer)(sysstdin, sysstdout, prompt); PyENDALLOWTHREADS

Trent.

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python- dev/kristjan%40ccpgames.com

More information about the Python-Dev mailing list