Issue 20318: subprocess.Popen can hang in threaded applications in Python 2 (original) (raw)

Created on 2014-01-20 20:30 by Andrew.Lutomirski, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
python_thread_bug.py Andrew.Lutomirski,2014-01-20 20:30
Messages (7)
msg208577 - (view) Author: Andrew Lutomirski (Andrew.Lutomirski) Date: 2014-01-20 20:30
Running python python_thread_bug.py -j4 often results in one of the threads failing to start until another thread finishes. The bug appears to be that subprocess's pipe_cloexec function is racy: if another thread forks between os.pipe() and _set_cloexec_flag, then the resulting process could hang on to the write end of the pipe. That will cause the Popen call that got rudely interrupted to wait until the whole resulting process tree dies.
msg208578 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-01-20 20:35
Hmm. I thought someone had already reported this, but I can't find an issue. I suspect it is fixed in 3.4 and it may not be practical to fix it in earlier versions.
msg208582 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-01-20 22:07
2.7 subprocess is fundamentally flawed and cannot be fixed. Install subprocess32 from PyPI. It's fixed in 3.2 and later.
msg208584 - (view) Author: Andrew Lutomirski (Andrew.Lutomirski) Date: 2014-01-20 22:19
Would it be worth adding something to the Python 2.7 subprocess docs indicating that subprocess is known to be broken?
msg208585 - (view) Author: Andrew Lutomirski (Andrew.Lutomirski) Date: 2014-01-20 22:26
FWIW, sticking a mutex in Popen.__init__ (wrapping the whole thing) seems to work around this issue (for programs that aren't using multiprocessing or fork, for example). This might be a good-enough fix and be safe enough to stick in the standard library.
msg208606 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-01-21 03:55
A pointer in the 2.7 subprocess docs to subprocess32 does seem like a good idea, its what i tell everyone to do anyways. :) If you've got a patch for this in 2.7 feel free to add it here and I can take a look. leaving this open as a reminder to me to update the docs at the very least.
msg216305 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-04-15 15:10
I added a pointer to subprocess32 in the 2.7 subprocess docs in dd52365c8721.
History
Date User Action Args
2022-04-11 14:57:57 admin set github: 64517
2016-06-28 14:12:07 r.david.murray link issue27406 superseder
2014-04-15 15:10:31 gregory.p.smith set status: open -> closedresolution: wont fixmessages: + stage: resolved
2014-02-10 06:14:01 martin.panter set nosy: + martin.panter
2014-02-03 01:32:35 vstinner set title: subprocess.Popen can hang in threaded applications -> subprocess.Popen can hang in threaded applications in Python 2
2014-01-21 03:55:13 gregory.p.smith set status: closed -> openassignee: gregory.p.smithmessages: +
2014-01-20 22:26:31 Andrew.Lutomirski set status: open -> closedmessages: +
2014-01-20 22:19:39 Andrew.Lutomirski set status: closed -> openresolution: wont fix -> (no value)messages: +
2014-01-20 22:07:40 gregory.p.smith set status: open -> closednosy: + gregory.p.smithmessages: + resolution: wont fix
2014-01-20 20:35:09 r.david.murray set nosy: + vstinner, r.david.murray, gpsmessages: +
2014-01-20 20:30:19 Andrew.Lutomirski create