| msg118905 - (view) |
Author: Michael Olson (Michael.Olson) |
Date: 2010-10-16 22:41 |
| In an application with an entry point of __main__.py, multiprocessing.Pool throws the following: Traceback (most recent call last): File "", line 1, in File "D:\Dev\Python27\lib\multiprocessing\forking.py", line 346, in main prepare(preparation_data) File "D:\Dev\Python27\lib\multiprocessing\forking.py", line 454, in prepare assert main_name not in sys.modules, main_name AssertionError: __main__ These messages repeat as long as the application is running. Demonstration Code, must be in file named __main__.py: -------------------- import multiprocessing import time if __name__ == '__main__': pool = multiprocessing.Pool() time.sleep(2) -------------------- |
|
|
| msg118969 - (view) |
Author: Michael Olson (Michael.Olson) |
Date: 2010-10-17 19:36 |
| I wrapped the offending assertion in a if main_name != '__main__'. I considered not checking the module_name against built-in modules but that seemed likely to be the sort of thing being guarded against, so I left it at an exception for __main__. Ran a few trivial testing using Pool and Process and it seems to work fine. |
|
|
| msg119134 - (view) |
Author: Ask Solem (asksol)  |
Date: 2010-10-19 12:36 |
| Is this on Windows? Does it work for you now? |
|
|
| msg119186 - (view) |
Author: Michael Olson (Michael.Olson) |
Date: 2010-10-20 03:35 |
| Sorry about that, yes, this is on Windows XP and 7, 32 bit. And with the if statement it seems to work fine. v/r -- Michael Olson |
|
|
| msg119204 - (view) |
Author: Michael Olson (Michael.Olson) |
Date: 2010-10-20 13:09 |
| Ummm, I think I've been unclear on where I was making changes, I changed lib\multiprocessing\forking.py to fix the issue. Patch attached. |
|
|
| msg119205 - (view) |
Author: Michael Olson (Michael.Olson) |
Date: 2010-10-20 13:13 |
| As a note, I didn't attach a patch at first because I was fairly sure I was kludging it into submission, but at least this makes it clear as to what I did. v/r -- Michael Olson |
|
|
| msg149989 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2011-12-21 14:22 |
| The fix from #10845 should be backported to Python 2.7 (bypassing the assertion isn't enough - you want to skip re-executing __main__ entirely) |
|
|
| msg213269 - (view) |
Author: Martin Dengler (mdengler) * |
Date: 2014-03-12 16:12 |
| I applied the patch from #10845 to 2.7.6 and it worked well for me. |
|
|
| msg236443 - (view) |
Author: Marc Schlaich (schlamar) * |
Date: 2015-02-23 13:55 |
| Please fix this. Scripts with multiprocessing bundled as wheels are broken with Python 2.7 on Windows: https://github.com/pypa/pip/issues/1891 |
|
|
| msg236485 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2015-02-24 10:06 |
| Marc's reference to pip meant I noticed something that I had missed previously - that this issue is referring specifically to the use of setuptools/pip entry points, not to the -m switch. Entry points shouldn't be hitting this if they're importing the module containing the entry point normally - #10845 specifically related to invoking multiprocessing from a module run directly with the -m switch, which was worked around in 3.2 and 3.3, and then finally fixed properly in 3.4 by the implementation of #19946 So if there's an incompatibility between multiprocessing and entry points, it would be preferable to fix it in pip/setuptools, as that will reach many more installations than just fixing multiprocessing to better tolerate that situation. |
|
|
| msg254840 - (view) |
Author: Marc Schlaich (schlamar) * |
Date: 2015-11-18 08:39 |
| Please see my latest comments to https://github.com/pypa/pip/issues/1891. tl;dr It is related to the -m switch as pip's wheel launcher does PYTHONPATH=script.exe python -m __main__ |
|
|
| msg254872 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2015-11-19 01:53 |
| Thanks for the investigation Marc. I'd been hesitant to backport the mitigation patch in #10845 to 2.7.x, (as it *does* represent a behavioural change), but if that code path is currently hitting an assert statement anyway, it seems reasonable to make the change for 2.7.11+ |
|
|
| msg254875 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2015-11-19 02:59 |
| New changeset 5d88c1d413b9 by Nick Coghlan in branch '2.7': Close #10128: don't rerun __main__.py in multiprocessing https://hg.python.org/cpython/rev/5d88c1d413b9 |
|
|