Issue 6433: multiprocessing: pool.map hangs on empty list (original) (raw)

Created on 2009-07-07 11:39 by ede, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (6)
msg90228 - (view)	Author: Eric Eisner (ede)	Date: 2009-07-07 11:39
In multiprocessing, if you give a pool.map a zero-length iterator and specify a nonzero chunksize, the process hangs indefinitely. Example: import multiprocessing pool = multiprocessing.Pool() pool.map(len, [], chunksize=1) # hang forever Attached simple testcase and simple fix. I observed this behavior on 2.6 and 3.1, but only verified the patch on 3.1. Unless the line numbers changed it will probably fix it on 2.6 as well.
msg90231 - (view)	Author: Ezio Melotti (ezio.melotti) *	Date: 2009-07-07 12:06
I'm not familiar with multiprocessing but I gave a quick look at the patch. If the 'iterable' is an iterator/generator "if len(iterable) == 0:" may fail since these objects don't usually have a __len__. More tests for this situation are probably needed too.
msg90232 - (view)	Author: Eric Eisner (ede)	Date: 2009-07-07 12:19
A few lines before this patch the code turns iterable into a list if it does not hasattr('__len__') for the purpose of calculating the chunksize.
msg90483 - (view)	Author: Eric Eisner (ede)	Date: 2009-07-13 14:08
Can anyone review this patch? It is a very simple fix to catch this one edge case. As this can cause multiprocessing to enter a state where it needs to be externally killed, it would be good if this made 2.6.3.
msg90484 - (view)	Author: Jesse Noller (jnoller) *	Date: 2009-07-13 14:25
It's on my list for this week.
msg90571 - (view)	Author: Jesse Noller (jnoller) *	Date: 2009-07-16 14:23
committed r74023 on trunk