[Python-Dev] multiprocessing not compatible with functional.partial (original) (raw)

Neal Becker ndbecker2 at gmail.com
Thu Feb 12 16:53:56 CET 2009

Hrvoje Niksic wrote:

Calvin Spealman wrote:

I don't think it would be unreasonable to consider either 1) making functools.partial picklable (I don't know how feasible this is) It's not only feasible, but quite easy and, I think, useful. A "partial" instance is a simple triplet of (function, args, kwds), and it can be pickled as such. For example: >>> import copyreg, functools >>> def reconstructpartial(f, args, kwds): ... return functools.partial(f, *args, **(kwds or {})) ... >>> def reducepartial(p): ... return reconstructpartial, (p.func, p.args, p.keywords) ... >>> copyreg.pickle(functools.partial, reducepartial) Test: >>> import operator, cPickle as cp >>> p = functools.partial(operator.add, 3) >>> p(10) 13 >>> cp.dumps(p) 'c__main__\n_reconstruct_partial\np1\n(coperator\nadd\np2\n(I3\ntp3\nNtRp4\n.' >>> p2 = cp.loads() >>> p2(10) 13 Iedally this should be implemented in the functools.partial object itself. Confirmed:

from multiprocessing import Pool

def power (x, pwr=2): return x**pwr

import functools run_test = functools.partial (power, pwr=3)

import copy_reg, functools def _reconstruct_partial(f, args, kwds): return functools.partial(f, *args, **(kwds or {}))

def _reduce_partial(p): return _reconstruct_partial, (p.func, p.args, p.keywords)

copy_reg.pickle(functools.partial, _reduce_partial)

if name == "main":

pool = Pool()
cases = [3,4,5]
results = pool.map (run_test, cases)
print results

$python test_multi.py [27, 64, 125]

More information about the Python-Dev mailing list