| msg128768 - (view) |
Author: Matt Chaput (mattchaput) |
Date: 2011-02-17 23:56 |
| If you start unit tests with a command line such as "python setup.py test" or "nosetests", if the tested code starts a multiprocessing.Process on Windows, each new process will act as if it was started as "python setup.py test"/"nosetests", leading to an infinite explosion of processes that eventually locks up the entire machine. |
|
|
| msg128770 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *  |
Date: 2011-02-18 00:28 |
| Using multiprocessing on Windows can be different; please read http://docs.python.org/library/multiprocessing.html#windows especially the part named "Safe importing of main module". On Windows, fork() does not exist, so a new interpreter must be started, which will import the current module; this must not start the test suite again! Adding "if __name__ == '__main__'" somewhere is probably the solution. If not, you should move the target function to another module. |
|
|
| msg128771 - (view) |
Author: Matt Chaput (mattchaput) |
Date: 2011-02-18 00:33 |
| Thank you, I understand all that, but I don't think you understand the issue. My code is not __main__. I am not starting the test suite. It's the distutils/nose code that's doing that. It seems as if the multiprocessing module is starting new Windows processes by duplicating the command line of the original process. That doesn't seem to work very well, given the example of running test suites, hence the bug. |
|
|
| msg128772 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2011-02-18 00:51 |
| > It seems as if the multiprocessing module is starting new Windows > processes by duplicating the command line of the original process. It does not. The spawned processes use the command:: python.exe -c 'from multiprocessing.forking import main; main()' --multiprocessing-fork [handle#] And only after, the multiprocessing machinery overrides sys.argv with the same value as the initial process. There is certainly some code in one of your modules that starts running the tests. |
|
|
| msg128794 - (view) |
Author: Matt Chaput (mattchaput) |
Date: 2011-02-18 16:49 |
| I don't know what to tell you... to the best of my knowledge there's absolutely no way for my code to kick off the entire test suite -- I always do that through PyDev (which doesn't cause the bug, by the way). The closest thing is the boilerplate at the bottom of every test file: if __name__ == "__main__": unittest.main() ...but even that would only start the tests in that file, not the entire suite. Another thing that makes me think multiprocessing is re-running the original command line is that if I use "python setup.py test" to start the tests, when it gets to the MP tests it seems to run that command for each Process that gets started, but if I use "nosetests", it seems to run "nosetests" for each started Process. |
|
|
| msg128797 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2011-02-18 17:51 |
| Nose works correctly for me with multiprocessing. In a directory, I have: == run_nose.py ================= from nose import main if __name__ == '__main__': main() ================================ == test_me.py ================== from multiprocessing import Pool import os, time def foo(x): time.sleep(0.1) return (x, os.getpid()) def test_me(): pool = Pool(processes=4) x = pool.map(foo, range(10)) a, b = zip(*x) print a, b assert list(a) == range(10) assert 1 < len(set(b)) <= 4 ================================ Now when I do: "c:\python27\python run_nose.py" the test runs correctly. Can you try this test in your environment? |
|
|
| msg128799 - (view) |
Author: Matt Chaput (mattchaput) |
Date: 2011-02-18 18:55 |
| If I do "c:\python27\python run_nose.py" it works correctly. If I do "nosetests" I get the process explosion. Maybe the bug is in how distutils and nose work from the command line? I'm confused. |
|
|
| msg128803 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2011-02-18 19:17 |
| Ah, is 'nosetests' a .exe file? A frozen executable? In this case, can you try the solution proposed at http://docs.python.org/library/multiprocessing.html#multiprocessing.freeze_support |
|
|
| msg128804 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2011-02-18 19:23 |
| > If I do "c:\python27\python run_nose.py" it works correctly. If I do > "nosetests" I get the process explosion. Maybe the bug is in how > distutils and nose work from the command line? I'm confused. The bug is in nose itself. You should report a bug there. |
|
|
| msg151479 - (view) |
Author: Chris Jones (Chris.Jones) |
Date: 2012-01-17 17:58 |
| You can work around this issue by using: python.exe -c "import nose; nose.main()" instead of nosetests Note that nose.main() with no args parses sys.argv |
|
|
| msg212742 - (view) |
Author: Matt Chaput (mattchaput) |
Date: 2014-03-04 22:04 |
| IIRC the root issue turned out to be that when you execute any multiprocessing statements at the module/script level on Windows, you need to put it under if __name__ == "__main__", otherwise it will cause infinite spawning. I think this is mentioned in the multiprocessing docs but should probably be in giant blinking red letters ;) |
|
|
| msg212745 - (view) |
Author: Martin Dengler (mdengler) * |
Date: 2014-03-04 22:37 |
| > the root issue turned out to be that when you execute any multiprocessing statements at the module/script level on Windows, you need to put it under if __name__ == "__main__", otherwise it will cause infinite spawning. Same for me. The error message and failure mode are completely unhelpful, though. > I think this is mentioned in the multiprocessing docs but should probably be in giant blinking red letters ;) Indeed. It would be even better if I or someone else had time to contribute a patch to fix the behaviour and, or at least the failure mode / error message. In a large codebase with multiple contributors it might not be so simple to track down the commit that caused the issue, especially if one is just starting out and the tests aren't clean. |
|
|
| msg235692 - (view) |
Author: Davin Potts (davin) * |
Date: 2015-02-10 16:00 |
| In some (but not necessarily all) circumstances, the multiprocessing module is now (in 2.7 and 3.x) able to detect the "infinite spawning" behavior described due to unsafe importing of the main module on Windows. The resulting error message looks like this: RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable. Hopefully this behavior and the resulting message will both alert developers when such a problem is introduced in a project and communicate to them what the problem is. Giant blinking red letters aside, hopefully this pragmatic solution will do a better job of educating and helping developers notice and deal with such issues. In the combinatoric explosion of possibilities that combine nosetest-binaries with PyDev with other useful dev tools, if this error message is being suppressed or otherwise not shared with the end user of those tools, that would warrant opening an issue with that tool. |
|
|