| msg82534 - (view) |
Author: mike bayer (zzzeek) * |
Date: 2009-02-20 16:06 |
| this occurs for me running on Mac OSX Leopard. The equivalent code using "processing" in python 2.5 works fine, which is how I found this bug - my code hung when upgraded to 2.6. Basically initiating a multiprocessing.Pool inside of multiprocessing.Process hangs the application. Below is code which illustrates the issue on both py2.6 and py3.0: from multiprocessing import Pool, Process import sys def f(x): return x*x def go(): pool = Pool(processes=4) x = pool.map(f, [1, 2, 3, 4, 5, 6, 7]) sys.stdout.write("result: %s\n" % x) def go2(): x = map(f, [1,2,3,4,5,6,7]) sys.stdout.write("result: %s\n" % x) # pool alone, fine go() # process alone, fine p = Process(target=go2) p.start() p.join() # use both, hangs p = Process(target=go) p.start() p.join() |
|
|
| msg82540 - (view) |
Author: David W. Lambert (LambertDW) |
Date: 2009-02-20 18:57 |
| Fails also on this system. $ p3 Python 3.0.1 (r301:69556, Feb 13 2009, 23:52:55) [GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2 (...) >>> # use both hangs (...) >>> p.start() >>> p.join() (hmm la do diii laaa...)^C Traceback (most recent call last): File "", line 1, in File "/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/process.py", line 119, in join res = self._popen.wait(timeout) File "/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py", line 118, in wait return self.poll(0) File "/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py", line 107, in poll pid, sts = os.waitpid(self.pid, flag) KeyboardInterrupt >>> |
|
|
| msg89192 - (view) |
Author: OG7 (OG7) |
Date: 2009-06-10 08:58 |
| Apparently the pool workers die all at once, just after the pool creates them and before the pool is used. I added a few lines to multiprocessing/pool.py to get the stack and the exception backtrace. except (EOFError, IOError): import traceback debug(traceback.format_exc()) debug(''.join(traceback.format_stack())) debug('worker got EOFError or IOError -- exiting') break INFO::Rule dispatcher::multiprocessing::child process calling self.run() DEBUG::Rule dispatcher::multiprocessing::created semlock with handle 3082559488 DEBUG::Rule dispatcher::multiprocessing::created semlock with handle 3082104832 DEBUG::Rule dispatcher::multiprocessing::created semlock with handle 3081826304 DEBUG::Rule dispatcher::multiprocessing::created semlock with handle 3081822208 INFO::PoolWorker-3:1::multiprocessing::child process calling self.run() DEBUG::PoolWorker-3:1::multiprocessing::Traceback (most recent call last): File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py", line 57, in worker task = get() File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/queues.py", line 339, in get return recv() IOError: [Errno 9] Bad file descriptor DEBUG::PoolWorker-3:1::multiprocessing:: File "/home/who/var/co/git-svn/what-base/correlator/bin/correlator", line 30, in what.corr.actors.main.main() File "/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/main.py", line 47, in main rrp.start() File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py", line 109, in start self._popen = Popen(self) File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py", line 99, in __init__ code = process_obj._bootstrap() File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py", line 236, in _bootstrap self.run() File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/rule_dispatcher.py", line 26, in main initargs=(manager.agg_msgs_queue, )) File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/__init__.py", line 232, in Pool return Pool(processes, initializer, initargs) File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py", line 107, in __init__ w.start() File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py", line 109, in start self._popen = Popen(self) File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py", line 99, in __init__ code = process_obj._bootstrap() File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py", line 236, in _bootstrap self.run() File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py", line 61, in worker debug(''.join(traceback.format_stack())) DEBUG::PoolWorker-3:1::multiprocessing::worker got EOFError or IOError -- exiting INFO::PoolWorker-3:1::multiprocessing::process shutting down DEBUG::PoolWorker-3:1::multiprocessing::running all "atexit" finalizers with priority >= 0 |
|
|
| msg89194 - (view) |
Author: OG7 (OG7) |
Date: 2009-06-10 10:42 |
| It seems the root cause is at http://bugs.python.org/issue5155 . A workaround is to use a duplex Pipe in SimpleQueue. |
|
|
| msg89901 - (view) |
Author: Jesse Noller (jnoller) *  |
Date: 2009-06-30 03:04 |
| Patch attached to issue 5313, please review |
|
|
| msg89945 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2009-06-30 17:14 |
| Committed in r73708 on trunk |
|
|