(original) (raw)

Index: Misc/NEWS =================================================================== --- Misc/NEWS (revision 73699) +++ Misc/NEWS (working copy) @@ -337,6 +337,10 @@ Library ------- +- Issues #5155, 5313, 5331: multiprocessing.Process._bootstrap was + unconditionally calling "os.close(sys.stdin.fileno())" resulting in file + descriptor errors + - Issue #6365: Distutils build_ext inplace mode was copying the compiled extension in a subdirectory if the extension name had dots. Index: Misc/ACKS =================================================================== --- Misc/ACKS (revision 73699) +++ Misc/ACKS (working copy) @@ -46,6 +46,7 @@ Ulf Bartelt Nick Bastin Jeff Bauer +Mike Bayer Michael R Bax Anthony Baxter Samuel L. Bayer @@ -183,6 +184,7 @@ Dean Draayer John DuBois Paul Dubois +Graham Dumpleton Quinn Dunkan Robin Dunn Luke Dunstan Index: Doc/library/multiprocessing.rst =================================================================== --- Doc/library/multiprocessing.rst (revision 73699) +++ Doc/library/multiprocessing.rst (working copy) @@ -2101,7 +2101,27 @@ for i in range(10): Process(target=f, args=(lock,)).start() +Beware replacing sys.stdin with a "file like object" + :mod:`multiprocessing` originally unconditionally called:: + + os.close(sys.stdin.fileno()) + + In the :meth:`multiprocessing.Process._bootstrap` method of - this resulted + in issues with processes-in-processes. This has been changed to:: + + sys.stdin.close() + sys.stdin = open(os.devnull) + + Which solves the fundamental issue of processes colliding with each other + resulting in a bad file descriptor error, but introduces a potential danger + to applications which replace :func:`sys.stdin` with a "file-like object" + with output buffering, this danger is that if multiple processes call + :func:`close()` on this file-like object, it could result in the same + data being flushed to the object multiple times, resulting in corruption. + + For more information, see :issue:`5155`, :issue:`5313` and :issue:`5331` + Windows ~~~~~~~ Index: Lib/multiprocessing/process.py =================================================================== --- Lib/multiprocessing/process.py (revision 73699) +++ Lib/multiprocessing/process.py (working copy) @@ -220,7 +220,8 @@ self._children = set() self._counter = itertools.count(1) try: - os.close(sys.stdin.fileno()) + sys.stdin.close() + sys.stdin = open(os.devnull) except (OSError, ValueError): pass _current_process = self Index: Lib/test/test_multiprocessing.py =================================================================== --- Lib/test/test_multiprocessing.py (revision 73699) +++ Lib/test/test_multiprocessing.py (working copy) @@ -1861,8 +1861,46 @@ p.join() self.assertEqual(self.ns.test, 1) -testcases_other = [OtherTest, TestInvalidHandle, TestInitializers] +# +# Issue 5155, 5313, 5331: Test process in processes +# Verifies os.close(sys.stdin.fileno) vs. sys.stdin.close() behavior +# +def _ThisSubProcess(q): + try: + item = q.get(block=False) + except Queue.Empty: + pass + +def _TestProcess(q): + queue = multiprocessing.Queue() + subProc = multiprocessing.Process(target=_ThisSubProcess, args=(queue,)) + subProc.start() + subProc.join() + +def _afunc(x): + return x*x + +def pool_in_process(): + pool = multiprocessing.Pool(processes=4) + x = pool.map(_afunc, [1, 2, 3, 4, 5, 6, 7]) + +class TestStdinBadfiledescriptor(unittest.TestCase): + + def test_queue_in_process(self): + queue = multiprocessing.Queue() + proc = multiprocessing.Process(target=_TestProcess, args=(queue,)) + proc.start() + proc.join() + + def test_pool_in_process(self): + p = multiprocessing.Process(target=pool_in_process) + p.start() + p.join() + +testcases_other = [OtherTest, TestInvalidHandle, TestInitializers, + TestStdinBadfiledescriptor] + # # #