[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension (original) (raw)

[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

Antoine Pitrou solipsis at pitrou.net
Fri Sep 28 18:44:54 EDT 2018

Hi,

On Fri, 28 Sep 2018 17:07:33 -0400 Sean Harrington <seanharr11 at gmail.com> wrote:

In short, the implementation of the feature works as follows: 1. Exposes a kwarg on Pool.init called expectinitret, that defaults to False. When set to True: 1. Capture the return value of the initializer kwarg of Pool 2. Pass this value to the function being applied, as a kwarg. Again, in short, the motivation of the feature is to provide an explicit "flow of data" from parent process to worker process, and to avoid being forced to using the global keyword in initializer, or being forced to create global variables in the parent process.

Thanks for taking the time to explain your use case and write a proposal.

My reactions to this are:

  1. The proposed API is ugly. This basically allows you to pass an argument which changes with which arguments another function is later called...
  2. A global variable seems like the adequate way to represent a process-global object (which is exactly your use case).
  3. If you don't like globals, you could probably do something like lazily-initialize the resource when a function needing it is executed; this also avoids creating the resource if the child doesn't use it at all. Would that work for you?

As a more general remark, I understand the desire to make the Pool object more flexible, but we can also not pile up features until it satisfies all use cases.

As another general remark, concurrent.futures is IMHO the preferred API for the future, and where feature work should probably concentrate.

Regards

Antoine.

More information about the Python-Dev mailing list