[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension (original) (raw)

[Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals

Sean Harrington seanharr11 at gmail.com
Fri Oct 12 10:49:28 EDT 2018

Yes - “func” (and “self” which func is bound to) would be copied to each child worker process, where they are stored and applied to each element of the iterable being mapped over. On Fri, Oct 12, 2018 at 10:41 AM Antoine Pitrou <solipsis at pitrou.net> wrote:

On Fri, 12 Oct 2018 09:42:50 -0400 Sean Harrington <seanharr11 at gmail.com> wrote: > I would contend that this is much more granular than Dask - this is just an > optimization of Pool.map() to avoid redundantly passing the same func > repeatedly, once per task, to each worker, with the primary goal of > eliminating redundant serialization of large-memory-footprinted Callables. > This is a different use case than Dask - I don't intend to approach the > shared memory or distributed computing realms. > > And the second call to Pool.map would update the cached "self" as a part of > its initialization workflow, s.t. "the latest version of self when map() is > called is taken into account".

I still don't understand how that works. If you "updated the cached self", then surely you must transmit it to the child, right? Regards Antoine.

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/seanharr11%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20181012/ac9abaa6/attachment.html>

More information about the Python-Dev mailing list