| msg280833 - (view) |
Author: Lev Veshnyakov (lev-veshnyakov) |
Date: 2016-11-15 12:52 |
| It's only in imap, in map it's ok. The following code explains the issue: from multiprocessing.pool import ThreadPool pool = ThreadPool(10) def gen(): yield 1 + '1' # here is an error try: next((pool.imap(str, gen()))) except: # Will be catched using pool.map normally, but using pool.imap will be not. # Instead it hangs. This is the same for ThreadPool and Pool. print('this will not be printed because thread is hanging') |
|
|
| msg280838 - (view) |
Author: Davin Potts (davin) *  |
Date: 2016-11-15 13:52 |
| Using the supplied example, I was unable to reproduce what you described using either 3.5 or 3.6-beta on OS X 10.11. What platform are you using? (Perhaps it is platform specific.) |
|
|
| msg280840 - (view) |
Author: Lev Veshnyakov (lev-veshnyakov) |
Date: 2016-11-15 13:55 |
| Ubuntu 14.04 LTS, 3.13.0-83-generic, x86_64 |
|
|
| msg280842 - (view) |
Author: Lev Veshnyakov (lev-veshnyakov) |
Date: 2016-11-15 14:04 |
| So, I've checked twice, it's presented by me on python 3.4.3, and not by 3.5.2. So I will go deaper |
|
|
| msg280863 - (view) |
Author: Lev Veshnyakov (lev-veshnyakov) |
Date: 2016-11-15 16:08 |
| I've reproduced it on 2 different machines: - on a MacBook in Docker (debian:jessie, python 3.4.2) - on another desktop (Ubuntu 14.04.1, 3.16.0-77-generic, x86_64, python 3.4.3) |
|
|
| msg280866 - (view) |
Author: Lev Veshnyakov (lev-veshnyakov) |
Date: 2016-11-15 16:32 |
| It's hanging in a while loop in _handle_workers, /usr/lib/python3.4/multiprocessingpool.py:365. I can't figure out what is the reason. @staticmethod def _handle_workers(pool): thread = threading.current_thread() # Keep maintaining workers until the cache gets drained, unless the pool # is terminated. while thread._state == RUN or (pool._cache and thread._state != TERMINATE): pool._maintain_pool() time.sleep(0.1) # send sentinel to stop workers pool._taskqueue.put(None) util.debug('worker handler exiting') |
|
|
| msg280867 - (view) |
Author: Davin Potts (davin) * |
Date: 2016-11-15 16:37 |
| If it only occurs in 3.4.x, is moving to 3.5 an option for you? |
|
|
| msg280868 - (view) |
Author: Lev Veshnyakov (lev-veshnyakov) |
Date: 2016-11-15 17:19 |
| Yes, I'm free to move to 3.5, now I'm seeing isn't there any problems in 3.5 according to this issue. |
|
|
| msg281032 - (view) |
Author: Xiang Zhang (xiang.zhang) * |
Date: 2016-11-17 10:46 |
| In Py3.6, it raises error: >>> next((pool.imap(str, gen()))) Traceback (most recent call last): File "/opt/lib/python3.7/multiprocessing/pool.py", line 684, in next item = self._items.popleft() IndexError: pop from an empty deque During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/lib/python3.7/multiprocessing/pool.py", line 690, in next item = self._items.popleft() IndexError: pop from an empty deque During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "/opt/lib/python3.7/multiprocessing/pool.py", line 693, in next raise StopIteration StopIteration But this doesn't looks good. I think it should raise TypeError. It can be fixed in #28699. |
|
|
| msg281743 - (view) |
Author: Davin Potts (davin) * |
Date: 2016-11-25 22:06 |
| Closing this issue -- all further discussion moves to |
|
|