[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension (original) (raw)

[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

Sean Harrington seanharr11 at gmail.com
Fri Sep 28 17:07:33 EDT 2018

I am proposing an extension to the multiprocessing.Pool API that allows for an alternative way to pass data to Pool worker processes, without using globals.

A PR has been opened <https://github.com/python/cpython/pull/9627>, extensive test coverage is also included, with all tests & CI passing on github.

Please see this blog post <https://thelaziestprogrammer.com/python/multiprocessing-pool-expect-initret-proposal> for details, motivation, and use cases of the API extension before reading on.

In short, the implementation of the feature works as follows:

  1. Exposes a kwarg on Pool.init called expect_initret, that defaults to False. When set to True:
    1. Capture the return value of the initializer kwarg of Pool
    2. Pass this value to the function being applied, as a kwarg.

Again, in short, the motivation of the feature is to provide an explicit "flow of data" from parent process to worker process, and to avoid being forced to using the global keyword in initializer, or being forced to create global variables in the parent process.

The interface is 100% backwards compatible through Python3.x (and perhaps beyond). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20180928/03fcc55f/attachment.html>

More information about the Python-Dev mailing list