bpo-30006 More robust concurrent.futures.ProcessPoolExecutor by tomMoral · Pull Request #1013 · python/cpython (original) (raw)

This set of patches intends to fix some silent deadlocks that can happened with
the present ProcessPoolExecutor implementation. The deadlock situations
notably include:

• Pickling and Unpickling errors in task definitions and results.
• Processes that get killed in non pythonic ways (kill -9 / segfault).

The commit message of each patch gives more details but in essence:

1. Support for multiprocessing context instead of forcing the use of fork on
   posix systems.
2. Avoid deadlock when a process dies with the result_queue.wlock acquired.
3. Deadlock free clean up of failures in the queue_manager_thread and the
   queue_feeder_thread.
4. Deadlock free termination of workers and threads when the
   ProcessPoolExecutor instance has beeen gc'ed.

The commits are incremental, each one adds new fixes on top of the previous
ones to handle extra deadlock situations. If you think that some changes in the
last commits need more discussion or refinement, let me now so I can separate
them and open different tickets/ PR.

Each commit passes the test suit on its own but the addition of the 3 context
testing makes test_concurrent_futures longer (~140s on my computer). One
fix would be to split the test for each context in seprate files (allowing test
parallelization). I did not do it as it changes the structure of the test suit but
I can implement it if it is necessary.

This work was done as part of the loky project in collaboration with
@ogrisel. It provides a backport of the same features for older versions of
python including 2.7 for legacy users and reusable executors.

https://bugs.python.org/issue30006