[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension (original) (raw)

[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

Michael Selik mike at selik.org
Fri Sep 28 21:27:13 EDT 2018

On Fri, Sep 28, 2018 at 2:11 PM Sean Harrington <seanharr11 at gmail.com> wrote:

kwarg on Pool.init called expectinitret, that defaults to False. When set to True: Capture the return value of the initializer kwarg of Pool Pass this value to the function being applied, as a kwarg.

The parameter name you chose, "initret" is awkward, because nowhere else in Python does an initializer return a value. Initializers mutate an encapsulated scope. For a class init, that scope is an instance's attributes. For a subprocess managed by Pool, that encapsulated scope is its "globals". I'm using quotes to emphasize that these "globals" aren't shared.

On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington <seanharr11 at gmail.com> wrote:

On Fri, Sep 28, 2018 at 6:45 PM Antoine Pitrou <solipsis at pitrou.net> wrote:

3. If you don't like globals, you could probably do something like lazily-initialize the resource when a function needing it is executed if initializing the resource is expensive, we only want to do this ONE time per worker process.

We must have a different concept of "lazily-initialize". I understood Antoine's suggestion to be a one-time initialize per worker process.

On Fri, Sep 28, 2018 at 4:39 PM Sean Harrington <seanharr11 at gmail.com> wrote:

My simple argument is that the developer should not be constrained to make the objects passed globally available in the process, as this MAY break encapsulation for large projects.

I could imagine someone switching from Pool to ThreadPool and getting into trouble, but in my mind using threads is caveat emptor. Are you worried about breaking encapsulation in a different scenario?

More information about the Python-Dev mailing list