bpo-33056 FIX leaking fd in concurrent.futures.ProcessPoolExecutor by tomMoral · Pull Request #6084 · python/cpython (original) (raw)

@tomMoral, I believe that either this change and/or #3895 causes OSError: handle is closed when shutting down ProcessPoolExecutor with wait=False due to self._queue_management_thread_wakeup being closed, but still referenced in _threads_wakeups on exit of Python, e.g. with script:

from concurrent.futures import ProcessPoolExecutor


class ObjectWithPickleError():
    """Triggers a RuntimeError when sending job to the workers"""

    def __reduce__(self):
        raise RuntimeError()


if __name__ == "__main__":
    e = ProcessPoolExecutor()
    f = e.submit(id, ObjectWithPickleError())
    e.shutdown(wait=False)
    # f.result()  # Deadlock on get

I'm not sure what the right solution is here, obviously this change is to prevent leaking the file descriptors which are taken care of in the close(), but perhaps the _python_exit method needs a review. Hopefully you'll have some insight please?