[Python-Dev] Usage of the multiprocessing API and object lifetime (original) (raw)

Pablo Galindo Salgado pablogsal at gmail.com
Tue Dec 11 12:47:35 EST 2018

> Pablo's issue35378 evolved to add a weak reference in iterators to try > to detect when the Pool is destroyed: raise an exception from the > iterator, if possible.

That's an ok fix for me.

I am playing with weakreferences inside the iterator and result objects, but this may not be enough/a complete solution. For example, take the code of ApplyResult.get:

def get(self, timeout=None): if self._pool() is None: # self._pool is a weakref raise RuntimeError("The pool is dead. Aborting") <--- new code self.wait(timeout)

It can be that the pool is alive when we check for it (self._pool() is None) but while the code is waiting with no timeout (timeout=None) the pool dies, effectively leaving the program deadlocked with no error.

--

I agree that misusage of the pool should not be encouraged but in this situation the fact that this code hangs:

import multiprocessing

for x in multiprocessing.Pool().imap(int, ["4", "3"]): print(x)

is a bit worriying because although is incorrect and an abuse of the API, users can do this easily with no error message other than a misterious hang. I have found this on several places and people were very confused because usually the interpreter throws some kind of error indication. In my humble opinion, we should try to avoid hanging as a consequence of the misusage, whatever we do.

Pablo -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20181211/5a6ee3b2/attachment.html>

More information about the Python-Dev mailing list