| msg231073 - (view) |
Author: Florian Finkernagel (ffinkernagel) * |
Date: 2014-11-12 12:29 |
| If you import a module that creates a multiprocessing.Queue, puts a value, and then waits for to be received again from the queue, you run into a deadlock. The issue is that Queue._feed does 'from .util import is_existing' - which needs the import lock, but is still being held by the main thread. Attached a script that illustrates this. Patch is a two line change, import is_exiting in line 49, remove the import inside the thread: 49c49 < from multiprocessing.util import debug, info, Finalize, register_after_fork --- > from multiprocessing.util import debug, info, Finalize, register_after_fork, is_exiting 232d231 < from .util import is_exiting |
|
|
| msg236790 - (view) |
Author: Mark Lawrence (BreamoreBoy) * |
Date: 2015-02-27 17:05 |
| @Davin I believe that you're interested in multiprocessing issues. |
|
|
| msg236854 - (view) |
Author: Davin Potts (davin) *  |
Date: 2015-02-27 23:12 |
| Confirmed that the issue can be reproduced under 2.7.9 on OS X 10.10. It is not possible to reproduce the issue with default (3.5) -- taking a look at what's different there, notably the import of is_exiting has moved to the top of the queues module and is no longer only imported at the time Queue._feed is invoked, which is just as Florian advocates doing for 2.7. I should take a closer look to understand what else has changed. @Mark: Cool -- thanks. |
|
|
| msg236865 - (view) |
Author: Davin Potts (davin) * |
Date: 2015-02-28 05:40 |
| Attaching a patch for 2.7 that applies Florian's fix and provides a test for it as well. Although the issue is not triggered on 3.4 or default (3.5), there is the potential for regression there -- attaching a single patch that works for both 3.4 and 3.5 to provide a regression test (only a test, nothing to "fix"). These patches have been tested on OS X 10.10 and Ubuntu 12.04.5 64-bit for each of 2.7, 3.4, and default (3.5). |
|
|
| msg237375 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2015-03-06 19:59 |
| I'm not sure there is a need to fix this issue. Using multiprocessing Queue at import time looks as yet one way to shoot yourself in the foot. But the patch looks harmful, I'll commit it. |
|
|
| msg237376 - (view) |
Author: Mark Lawrence (BreamoreBoy) * |
Date: 2015-03-06 20:06 |
| That's all we need, harmful patches being committed :) Harmless possibly? |
|
|
| msg237377 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2015-03-06 20:21 |
| Oh, right. Harmless. Thanks for all the fish Mark. I meant the bug. |
|
|
| msg237378 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2015-03-06 20:34 |
| New changeset 069c13ca7a70 by Serhiy Storchaka in branch '2.7': Issue #22853: Fixed a deadlock when use multiprocessing.Queue at import time. https://hg.python.org/cpython/rev/069c13ca7a70 |
|
|
| msg237379 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2015-03-06 20:49 |
| As for 3.x, underscored test does not run, and when remove the underscore it runs, but produce a warning (you should run regrtests with -vv to see detailed warnings): $ ./python -m test.regrtest -vv -m '*no_import_lock_contention*' test_multiprocessing_spawn ... Warning -- threading._dangling was modified by test_multiprocessing_spawn Before: <_weakrefset.WeakSet object at 0xb6a960ac> After: <_weakrefset.WeakSet object at 0xb6c4cc0c> 1 test altered the execution environment: test_multiprocessing_spawn |
|
|
| msg237382 - (view) |
Author: Davin Potts (davin) * |
Date: 2015-03-06 21:25 |
| Corrected patch for 3.4 and default/3.5 -- newly introduced test is now turned on this time and the dangling weak references are properly addressed as well as the reference to Empty. Nastiness. Good save, Serhiy. |
|
|
| msg237384 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2015-03-06 21:35 |
| New changeset cf12856bde17 by Serhiy Storchaka in branch '3.4': Issue #22853: Added regression test for using multiprocessing.Queue at import https://hg.python.org/cpython/rev/cf12856bde17 New changeset dcd6d41f2c9a by Serhiy Storchaka in branch 'default': Issue #22853: Added regression test for using multiprocessing.Queue at import https://hg.python.org/cpython/rev/dcd6d41f2c9a |
|
|
| msg237385 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2015-03-06 21:36 |
| Thank you Florian and Davin for your contribution. |
|
|