Issue 941881: PEP309 Partial implementation (original) (raw)
Created on 2004-04-25 18:05 by hyeshik.chang, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Messages (24)
Author: Hyeshik Chang (hyeshik.chang) * 
Date: 2004-04-25 18:05
This patch implements functional module which is introduced by PEP309. It has only 'partial' function as its member in this stage.
Unittest code is copied and modified slightly from Patch #931010 by Peter Harris.
Author: Paul Moore (paul.moore) *
Date: 2004-04-26 19:30
Logged In: YES user_id=113328
Why implement this in C? I can't imagine that the performance improvement will be that significant. A pure Python module in the standard library seems to me to be a far better idea. As the PEP says, "the case for a built-in coded in C is not very strong". And a Python module is good self-documentation.
I prefer the function version suggested in the PEP (credited to Carl Banks) over the class-based one. You need to take a little care to avoid capturing argument names:
def partial(*args, **kwds):
def callit(*moreargs, **morekwds):
kw = kwds.copy()
kw.update(morekwds)
return args[0](*(args[1:]+moreargs), **kw)
return callitAuthor: Hyeshik Chang (hyeshik.chang) *
Date: 2004-04-27 02:05
Logged In: YES user_id=55188
Python-version (function) ... 1.19 2.69 Python-version (class) ... 2.61 2.38 C-version ... 0.50 0.37 (former value is for 100000 instanciations and latter is for 100000 calls.)
And, C version have a facility that changing attributes after the instantiation that is supported by class version only.
Author: Paul Moore (paul.moore) *
Date: 2004-04-27 08:03
Logged In: YES user_id=113328
Yes, that looks like a significant speed improvement :-) I was basing my assumptions on the comments made in the PEP. Sorry.
But I still wonder if having the implementation in Python wouldn't be better from a maintenance point of view. (As well as all the arguments about usefulness as sample code, ability to backport, etc etc, that have come up on python-dev regarding moving other Python library modules into C...).
Author: Bob Ippolito (bob.ippolito) *
Date: 2004-05-18 04:05
Logged In: YES user_id=139309
I would use partial in situations where speed can matter (imap, commonly used event handlers, etc.), so a fast C implementation would be nice to have.
Author: Paul Moore (paul.moore) *
Date: 2004-08-03 20:23
Logged In: YES user_id=113328
OK, a real need beats my theoretical worries :-) Consider me convinced.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2004-08-11 00:11
Logged In: YES user_id=1038590
I have tested this patch on Windows, and it passes its own test suite, without affecting any other tests.
However, PCBuild\pythoncore.vcproj & PC\config.c require modification to allow Python to pick up the new module correctly.
Patch #1006948 created with the needed changes (also removes unneeded ODBC references from pythoncore.vcproj as I am using the free MS toolkits to build here)
My patch definitely needs to be checked by someone with a copy of Vis Studio 2003!
Author: Steven Bethard (bethard) *
Date: 2005-02-19 20:38
Logged In: YES user_id=945502
It might be nice if partial objects were usable as instancemethods, like functions are. Note the following behavior with the current patch:
import functional class C(object): ... pass ... def func(self, arg): ... return arg ... for item in ['a', 'b', 'c']: ... setattr(C, item, functional.partial(func, item)) ... C().a() Traceback (most recent call last): File "", line 1, in ? TypeError: func() takes exactly 2 arguments (1 given)
If functional.partial was instead defined like:
class partial(object): def init(*args, **kwargs): self = args[0] try: self.func = args[1] except IndexError: raise TypeError('expected 2 or more arguments, got ' % len(args)) self.obj = () self.args = args[2:] self.kwargs = kwargs def call(self, *args, *kwargs): if kwargs and self.kwargs: d = self.kwargs.copy() d.update(kwargs) else: d = kwargs or self.kwargs return self.func((self.obj + self.args + args), **d) def get(self, obj, type=None): if obj is None: return self result = partial(self.func, *self.args, **self.kwargs) result.obj = (obj,) return result
where an appropriate get method is provided, then partial objects behave properly when set as attributes to a class:
py> def test(): ... class C(object): ... pass ... def func(self, arg): ... return arg ... for item in ['a', 'b', 'c']: ... setattr(C, item, functional.partial(func, item)) ... c = C() ... return c.a(), c.b(), c.c() ... py> test() ('a', 'b', 'c')
Author: Martin v. Löwis (loewis) *
Date: 2005-02-25 20:34
Logged In: YES user_id=21627
In this form, the patch cannot be applied, since it lacks documentation changes. It might be that perky does not consider himself fluent enough in English to write the documentation - any other volunteers?
I also think there is value to bediviere's point, even though the PEP currently specifies partial() differently.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2005-02-26 03:17
Logged In: YES user_id=1038590
Peter Harris already wrote docs in Patch 931007. Presumably they are still accurate, or perky would have said something.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2005-02-26 03:27
Logged In: YES user_id=1038590
Steve's suggestion does sound good, but can't it simply be implemented with the PEP implementation and the following get method?:
def get(self, obj, type=None): if obj is None: return self return partial(self.func, obj, *self.args, **self.kwargs)
Author: Alyssa Coghlan (ncoghlan) *
Date: 2005-02-26 03:45
Logged In: YES user_id=1038590
If the get method is added to make partial functions behave like instance methods, the documentation should probably mention that classmethod() and staticmethod() can then be used to alter the behaviour:
Py> def f(*args): ... print args ... Py> class C: ... a = functional.partial(f) ... b = classmethod(functional.partial(f)) ... c = staticmethod(functional.partial(f)) ... Py> C().a <functional.partial object at 0x009E0DD0> Py> C().a() (<__main__.C instance at 0x009E6698>,) Py> C().b() (<class __main__.C at 0x009DDB40>,) Py> C().c() ()
(This example used the simpler get implementation I posted earlier)
Author: Steven Bethard (bethard) *
Date: 2005-02-26 03:53
Logged In: YES user_id=945502
Yes, that's much clearer. I don't remember why I couldn't get it to work this way before (I tried something quite similar to this), but I'm glad someone could figure out the simpler way to do it. ;-)
One potential problem with my proposal -- it doesn't work for nested partials:
py> class C(object): ... pass ... py> def func(self, arg1, arg2): ... return self, arg1, arg2 ... py> setattr(C, 'a', partial(func, 'a', 'b')) py> C().a() (<__main__.C object at 0x01186470>, 'a', 'b') py> setattr(C, 'a', partial(partial(func, 'a'), 'b')) py> C().a() ('a', <__main__.C object at 0x01186370>, 'b')
One possible solution would be to merge nested partials, but I'm not at all certain this solves all the problems:
py> class partial(object): ... def init(*args, **kw): ... self = args[0] ... func = args[1] ... if isinstance(func, partial): ... self.fn = func.fn ... self.args = args[2:] + func.args ... d = func.kw.copy() ... d.update(kw) ... self.kw = d ... else: ... self.fn, self.args, self.kw = (args[1], args[2:], kw) ... def call(self, *args, *kw): ... if kw and self.kw: ... d = self.kw.copy() ... d.update(kw) ... else: ... d = kw or self.kw ... return self.fn((self.args + args), **d) ... def get(self, obj, type=None): ... if obj is None: ... return self ... return partial(self.fn, obj, *self.args, **self.kw) ... py> class C(object): ... pass ... py> def func(self, arg1, arg2): ... return self, arg1, arg2 ... py> setattr(C, 'a', partial(func, 'a', 'b')) py> C().a() (<__main__.C object at 0x01186BB0>, 'a', 'b') py> setattr(C, 'a', partial(partial(func, 'a'), 'b')) py> C().a() (<__main__.C object at 0x01186650>, 'b', 'a')
On the other hand, merging nested partials does reduce the number of function calls, so perhaps it's useful in and of itself...
Author: Steven Bethard (bethard) *
Date: 2005-02-26 03:58
Logged In: YES user_id=945502
Oops -- wrong order of association.
... self.args = args[2:] + func.args ... d = func.kw.copy() ... d.update(kw)
should have been
... self.args = func.args + args[2:] ... kw.update(func.kw) ... self.kw = kw
hopefully these errors didn't confuse my intent too much.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2005-02-26 05:27
Logged In: YES user_id=1038590
Moving the discussion to python-dev :)
Author: Paul Moore (paul.moore) *
Date: 2005-02-27 20:43
Logged In: YES user_id=113328
To build on Windows (at least with mingw gcc, I don't know about MSVC) you need to assign members of partial_type which are symbols in the Python DLL at runtime, not compile time. The following diff probably explains better:
--- functionalmodule.c.orig 2005-02-27 20:41:41.000000000 +000 +++ functionalmodule.c 2005-02-26 21:41:14.000000000 +0000 @@ -197,7 +197,7 @@ 0, /* tp_hash / (ternaryfunc)partial_call, / tp_call / 0, / tp_str */
PyObject_GenericGetAttr, /* tp_getattro */
0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC |
@@ -220,7 +220,7 @@ 0, /* tp_init / 0, / tp_alloc / partial_new, / tp_new */
PyObject_GC_Del, /* tp_free */
0, /* tp_free */
};
@@ -239,6 +239,8 @@ int i; PyObject *m; char *name;
partial_type.tp_getattro = PyObject_GenericGetAttr;partial_type.tp_free = PyObject_GC_Del; PyTypeObject *typelist[] = { &partial_type, NULL
Author: Raymond Hettinger (rhettinger) *
Date: 2005-02-27 20:43
Logged In: YES user_id=80475
Exposing the structure members for dynamic alteration exceeds the PEP specification. While making the tool more flexible, it slows it down (requiring type checks for every call) and it encourages strange uses. The notion of giving partial objects a changable state is at odds with the basis for functional programming (i.e. statelessness).
The traverse function should use the new Py_VISIT macro.
The PyCallable_Check() is unnecessary as it duplicates the check already existing in PyObject_Call().
I am not certain whether the PyDict_Copy() is necessary. Calls to f(**d) already pass a copy of d. Likewise, calling f(a=1) will create a new dictionary. If so, that dictionary can be updated in- place with pto->kw rather than creating a new dictionary.
None of these issues are critical. Feel free to go forward with the patch.
Author: Martin v. Löwis (loewis) *
Date: 2005-02-27 23:05
Logged In: YES user_id=21627
It turns out that the documentation is incorrect, wrt. this patch, as it mentions a non-existing class functional.Partial.
Author: Raymond Hettinger (rhettinger) *
Date: 2005-02-28 05:25
Logged In: YES user_id=80475
- Significantly beefed-up the test suite.
- Brought in docs from another patch and heavily editted.
- Brought in setup/build code from another patch and enhanced for MSC6.0.
- Made attributes read-only and improved their names
- Did NOT change dict copy/update logic.
- Improved docstring wording
- Applied the Py_VISIT macro
- Added a NEWS announcement
- Added lib.tex
Suggest that if Nick wants to add get, that it occur separately after this patch is applied. It would be a nice improvement.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2005-02-28 12:41
Logged In: YES user_id=1038590
I agree with Raymond - put the basic left-binding partial in place, and we can look at options like partialmethod, rightpartial and so forth that give more control over the positional arguments later.
Author: Raymond Hettinger (rhettinger) *
Date: 2005-02-28 14:21
Logged In: YES user_id=80475
Martin, if there are no objections, I'll go ahead an apply this one so we can move on.
Author: Martin v. Löwis (loewis) *
Date: 2005-02-28 18:02
Logged In: YES user_id=21627
The patch looks good, so please apply.
Author: George Yoshida (quiver)
Date: 2005-03-02 11:59
Logged In: YES user_id=671362
"New in version 2.5." statement is missing from the document.
Author: George Yoshida (quiver)
Date: 2005-03-03 15:29
Logged In: YES user_id=671362
Noted that the module is new in version 2.5. Thanks, Raymond.