Issue 5135: Expose simplegeneric function in functools module (original) (raw)
Created on 2009-02-02 19:57 by paul.moore, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Messages (30)
Author: Paul Moore (paul.moore) * 
Date: 2009-02-02 19:57
This patch takes the existing "simplegeneric" decorator, currently an internal implementation detail of the pkgutil module, and exposes it as a feature of the functools module.
Documentation and tests have been added, and the pkgutil code has been updated to use the functools implementation.
Open issue: The syntax for registering an overload is rather manual:
def xxx_impl(xxx):
pass
generic_fn.register(XXX, xxx_impl)It might be better to make the registration function a decorator:
@generic_fn.register(XXX)
def xxx_impl(xxx):
passHowever, this would involve changing the existing (working) code, and I didn't want to do that before there was agreement that the general idea (of exposing the functionality) was sound.
Author: Ryan Freckleton (ryan.freckleton)
Date: 2009-02-04 02:43
PJE seems to have borrowed the time machine :-). Based on the code the register function is already a decorator:
def register(typ, func=None):
if func is None:
return lambda f: register(typ, f)
registry[typ] = func
return funcThe returned lambda is a one argument decorator. so your syntax:
@generic_fn.register(XXX)
def xxx_impl(xxx):
passAlready works. A test to validate this behavior should probably be added.
I don't mean to bikeshed, but could we call this function functools.generic instead of functools.simplegeneric? The only reason I can think of for keeping it simplegeneric would be to avoid a future name clash with the Generic Function PEP and if/when that PEP get's implemented, I would think that the functionality would live in builtins, not functools.
Author: Paul Moore (paul.moore) *
Date: 2009-02-04 08:10
Well spotted! I missed that when I checked. I will add tests and documentation.
I agree that generic is better. I only left it as it was because the original intent was simply to move the existing code - but that's not a particularly good reason for keeping a clumsy name. There shouldn't be a clash, as any more general mechanism can either be in its own module or the existing function can be extended in a compatible manner. I'll make this change too.
Thanks for the feedback!
Author: Paul Moore (paul.moore) *
Date: 2009-02-04 08:27
Here's an updated patch.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2009-02-04 11:48
The reason I like the simplegeneric name is that that is exactly what this feature is: a simple generic implementation that is deliberately limited to dispatching on the first argument (because that is easily explained to users that are already familiar with OOP and especially the existing Python magic method dispatch mechanism.
So the name isn't just about avoiding name clashes, it's also about setting appropriate expectations as to what is supported. Yes, the name is a little clumsy but one thing I do not want to see happen is a swathe of feature requests asking that this become an all-singing all-dancing generic function mechanism like RuleDispatch.
Don't forget that actually writing generic functions (i.e. using the @functools.simplegeneric decorator itself) should be far less common than using the .register() method of existing generic functions.
Author: Paul Moore (paul.moore) *
Date: 2009-02-04 12:03
Fair comment. As Ryan said, it's a bit of a bikeshed issue. I prefer "generic", on the basis that I'd prefer to keep the simple name for the simple use - something as complex as the RuleDispatch version could use the name "dispatch" (if they want to keep it the name simple) or "multimethod" (to emphasize that it dispatches on more than just one argument).
If the consensus is for keeping it as "simplegeneric", I'll go with that, but I'll wait for other views first.
BTW, another option is simply to clarify the limitations in the documentation. I can add something there if that would be enough for you.
Author: Walter Dörwald (doerwalter) *
Date: 2009-02-04 20:19
The patch looks fine to me. Tests pass.
I have no opinion about the name. Both "simplegeneric" and "generic" are OK to me.
I wonder if being able to use register() directly instead of as a decorator should be dropped.
Also IMHO the Python 2.3 backwards compatibility (name isn't setable) can be dropped.
Author: Paul Moore (paul.moore) *
Date: 2009-02-04 22:52
Agreed about the compatibility. It's there from pkgutil, where to be honest, it's even less necessary, as simplegeneric was for internal use only, there. I'm certainly not aware of any backward compatibility requirements for functools.
Assuming nobody speaks up to the contrary, I'll rip out the compatibility bits in the next version of the patch.
I'm unsure about the non-decorator version of register. I can imagine use cases for it - consider pprint, for example, where you might want to register str as the overload for your particular type. But it's not a big deal either way.
Author: Ryan Freckleton (ryan.freckleton)
Date: 2009-02-05 03:30
I think that registering existing functions is an important use case, so I vote for keeping the non-decorator version of register.
Another thing that we may want to document is that [simple]generic doesn't dispatch based on registered abstract base classes.
class A: ... pass ... class C: ... metaclass = abc.ABCMeta ... C.register(A) @generic ... def pprint(obj): ... print str(obj) ... @pprint.register(C) ... def pprint_C(obj): ... print "Charlie", obj ... pprint(C()) Charlie <__main__.C object at 0xb7c5336c> pprint(A()) <__main__.A instance at 0xb7c5336c>
Author: Alyssa Coghlan (ncoghlan) *
Date: 2009-02-05 10:58
Failure to respect isinstance() should be fixed, not documented :)
As far as registering existing functions goes, I also expect registering lambdas and functools.partial will be popular approaches, so keeping direct registration is a good idea. There isn't any ambiguity between the one-argument and two-argument forms.
Author: Paul Moore (paul.moore) *
Date: 2009-02-05 11:59
Agreed (in principle). However, in practice the subtleties of override order must be documented (and a method of implementation must be established!!!) Consider:
class A: ... pass ... class C: ... metaclass = abc.ABCMeta ... class D: ... metaclass = abc.ABCMeta ... C.register(A) D.register(A) @generic ... def pprint(obj): ... print "Base", str(obj) ... @pprint.register(C) ... def pprint_C(obj): ... print "Charlie", obj ... @pprint.register(D) ... def pprint_D(obj): ... print "Delta", obj ... pprint(A())
What should be printed? A() is a C and a D, but which takes precedence? There is no concept of a MRO for ABCs, so how would the "correct" answer be defined? "Neither" may not be perfect, but at least it's clearly defined. Relying on order of registration for overloads of the generic function seems to me to be unacceptable, before anyone suggests it, as it introduces a dependency on what order code is imported.
So while the theory makes sense, the practice is not so clear. Respecting ABCs seems to me to contradict the "simple" aspect of simplegeneric, so a documented limitation is appropriate.
(But given the above, I'm more inclined now to leave the name as "simplegeneric", precisely to make this point :-))
Author: Alyssa Coghlan (ncoghlan) *
Date: 2009-02-05 12:16
Hmm, there is such a thing as being too simple... a generic function implementation that doesn't even respect ABCs seems pretty pointless to me (e.g. I'd like to be able to register a default Sequence implementation for pprint and have all declared Sequences use it automatically if there isn't a more specific override).
I'll wait until I have a chance to actually play with the code a bit before I comment further though.
Author: Paul Moore (paul.moore) *
Date: 2009-02-05 13:20
Very good point. Registering for the standard ABCs seems like an important use case. Unfortunately, it seems to me that ABCs simply don't provide that capability - is there a way, for a given class, of listing all the ABCs it's registered under? Even if the order is arbitrary, that's OK.
Without that, I fail to see how any generic function implementation ("simple" or not) could support ABCs. (Excluding obviously broken approaches such as registration-order dependent overload resolution).
The problem is that ABCs are all about isinstance testing, where generic functions are all about avoiding isinstance testing. (As a compromise, you could have a base generic function that did isinstance testing for the sequence ABC).
Author: Alyssa Coghlan (ncoghlan) *
Date: 2009-02-05 20:03
Even more inconveniently, the existence of unregister() on ABCs makes it difficult for the generic to cache the results of the isinstance() checks (you don't want to be going through the chain of registered ABCs every time calling isinstance(), since that would be painfully slow).
That said, it is already the case that if you only register with an ABC, you don't get any of the methods - you have to implement them yourself. It's only when you actually inherit from the ABC that the methods are provided "for free". I guess the case isn't really any different here - if you changed your example so that A inherited from C and D rather than merely registering with them, then C & D would appear in the MRO and the generic would recognise them.
So perhaps just documenting the limitation is the right answer after all.
Author: Paul Moore (paul.moore) *
Date: 2009-02-06 19:03
Here's an updated patch. I've reverted to the name "simplegeneric" and documented the limitation around ABCs (I've tried to give an explanation why it's there, as well as a hint on now to work around the limitation - let me know if I'm overdoing it, or the text needs rewording).
I've also fixed the wrapper to use update_wrapper to copy the attributes. That way, there's no duplication.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2009-03-01 21:32
Unassigning - the lack of support for ABC registration still bothers me, but a) I don't have a good answer for it, and b) I'm going to be busy for a while working on some proposed changes to the with statement.
Author: Kevin Teague (kteague)
Date: 2009-03-02 09:39
The problem with generic functions supporting ABCs is it's a bug with the way ABCs work and not a problem with the generic function implementation. The register() method of an ABC only fakes out isinstance checks, it doesn't actually make the abstract base class a base class of the class. It doesn't make any sense for a class to say it is an instance of an ABC, but not have that ABC in it's MRO. It's not a base class if it's not in the MRO!
The documentation for lack of ABC support should read something like:
- Note that generic functions do not work with classes which have
- been declared as an abstract base class using the
- abc.ABCMeta.register() method because this method doesn't make
- that abstract base class a base class of the class - it only fakes
- out instance checks.
Perhaps a bug should be opened for the abc.ABCMeta.register() method.
However, I'd say that just because virtual abstract base classes are wonky doesn't mean that a solid generic function implementation shouldn't be added to standard library.
Author: Paul Moore (paul.moore) *
Date: 2009-03-02 13:21
I raised issue 5405. Armin Ronacher commented over there that it's not even possible in principle to enumerate the ABCs a class implements because ABCs can do semantic checks (e.g., checking for the existence of a special method).
So documenting the limitation is all we can manage, I guess.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2009-03-02 20:53
Given the point Armin raised, I once again agree that documenting the limitation is a reasonable approach. Longer-term, being able to subcribe to ABCs (and exposing the registration list if it isn't already visible) is likely to be the ultimate solution.
Author: Guido van Rossum (gvanrossum) *
Date: 2009-09-13 17:07
Please don't introduce this without a PEP.
Author: Mark Lawrence (BreamoreBoy) *
Date: 2010-07-21 13:53
Changes as Guido has stated that he wants a PEP.
Author: Paul Moore (paul.moore) *
Date: 2010-07-22 11:24
I don't propose to raise a PEP myself. The issue with ABCs seems to me to be a fundamental design issue, and I think it's better to leave raising any PEP, and managing the subsequent discussion, to someone with a deeper understanding of, and interest in, generic functions.
Not sure if the lack of a champion means that this issue should be closed. I'm happy if that's the consensus (but I'm also OK with it being left open indefinitely, until someone cares enough to pick it up).
Author: Antoine Pitrou (pitrou) *
Date: 2010-07-22 11:45
Generic functions are a lesser-known paradigm than OO, and nowhere do common Python documents (including the official docs) try to teach them. That means the first public appearance of generic functions in the stdlib should really be well thought out if we don't want to encourage poor practices. I agree with Guido that a PEP is required to flesh out all the details.
Author: Ryan Freckleton (ryan.freckleton)
Date: 2010-07-22 19:18
An elaborate PEP for generic functions already exists, PEP 3124 [ http://www.python.org/dev/peps/pep-3124/]. Also note the reasons for deferment. I'd be interested in creating a "more limited" generic function implementation based on this PEP, minus func_code rewriting and the other fancier items. Sadly I won't have any bandwidth to work on it until January of next year.
I'd vote for keeping this issue open because of that.
On Thu, Jul 22, 2010 at 5:45 AM, Antoine Pitrou <report@bugs.python.org>wrote:
Antoine Pitrou <pitrou@free.fr> added the comment:
Generic functions are a lesser-known paradigm than OO, and nowhere do common Python documents (including the official docs) try to teach them. That means the first public appearance of generic functions in the stdlib should really be well thought out if we don't want to encourage poor practices. I agree with Guido that a PEP is required to flesh out all the details.
nosy: +pitrou
Python tracker <report@bugs.python.org> <http://bugs.python.org/issue5135>
Author: Alyssa Coghlan (ncoghlan) *
Date: 2010-07-30 10:46
A couple more relevant links.
I brought this issue up in the context of a JSON serialisation discussion on python-ideas: http://mail.python.org/pipermail/python-ideas/2010-July/007854.html
Andrey Popp mentioned his pure Python generic functions library in that thread: http://pypi.python.org/pypi/generic
Author: Éric Araujo (eric.araujo) *
Date: 2011-03-25 22:45
The register() method of an ABC only fakes out isinstance checks, it doesn't actually make the abstract base class a base class of the class. It doesn't make any sense for a class to say it is an instance of an ABC, but not have that ABC in [its] MRO.
I disagree. If someone writes a class and registers them with an ABC, it is their duty to make sure that the class actually complies. Virtual subclasses are provided for use by consenting adults IMO.
Author: PJ Eby (pje) *
Date: 2011-03-26 01:37
Just as an FYI, it is possible to do generic functions that work with Python's ABCs (PEAK-Rules supports it for Python 2.6), but it requires caching, and a way of handling ambiguities. In PEAK-Rules' case, unregistering is simply ignored, and ambiguity causes an error at call time. But simplegeneric can avoid ambiguities, since it's strictly single-dispatch. Basically, you just have two dictionaries instead of one.
The first dictionary is the same registry that's used now, but the second is a cache of "virtual MROs" you'll use in place of a class' real MRO. The virtual MRO is built by walking the registry for classes that the class is a subclass of, but which are not found in the class's MRO, e.g.:
for rule_cls in registry:
if issubclass(cls, rule_cls) and rule_cls not in real_mro:
# insert rule_cls into virtual_mro for clsYou then insert those classes (abcs) in the virtual MRO at the point just after the last class in the MRO that says it's a subclass of the abc in question.
IOW, you implement it such that an abc declaration appears in the MRO just after the class that was registered for it. (This has to be recursive, btw, and the MRO cache has to be cleared when a new method is registered with the generic function.)
This approach, while not trivial, is still "simple", in that it has a consistent, unambiguous resolution order. Its main downside is that it holds references to the types of objects it has been called with. (But that could be worked around with a weak key dictionary, I suppose.) It also doesn't reset the cache on unregistration of an abc subclass, and it will be a bit slower on the first call with a previously-unseen type.
The downside of a PEP, to me, is that it will be tempting to go the full overloading route -- which isn't necessarily a bad thing, but it won't be a simple thing, and it'll be harder to get agreement on what it should do and how -- especially with respect to resolution order.
Still, if someone wants to do a PEP on simple generics -- especially one that can replace pkgutil.simplegeneric, and could be used to refactor things like copy.copy, pprint.pprint, et al to use a standardized registration mechanism, I'm all in favor -- with or without abc registration support.
Btw, the current patch on this issue includes code that is there to support classic classes, and metaclasses written in C. Neither should be necessary in 3.x. Also, a 3.x version could easily take advantage of type signatures, so that:
@foo.register def foo_bar(baz: bar): ...
could be used instead of @foo.register(bar, foo_bar).
But all that would be PEP territory, I suppose.
Author: Éric Araujo (eric.araujo) *
Date: 2011-03-26 01:40
Thanks for the detailed explanation. Note that type annotations are disallowed in the stdlib, as per PEP 8.
Author: Terry J. Reedy (terry.reedy) *
Date: 2013-03-25 21:04
Guido said "Please don't introduce this without a PEP." and that has not happened and if it did, the result would probably look quite different from the patches. So this is 'unripe' and no current action here is possible.
Author: Łukasz Langa (lukasz.langa) *
Date: 2013-06-07 20:09
For the record, this has been implemented as PEP 443.
History
Date
User
Action
Args
2022-04-11 14:56:45
admin
set
github: 49385
2013-06-07 20:09:35
lukasz.langa
set
nosy: + lukasz.langa
messages: +
2013-03-26 02:44:26
eric.snow
set
nosy: + eric.snow
2013-03-25 21:04:56
terry.reedy
set
stage: resolved
2013-03-25 21:04:43
terry.reedy
set
status: open -> closed
nosy: + terry.reedy
messages: +
resolution: later
2011-03-26 08:23:12
daniel.urban
set
nosy: + daniel.urban
2011-03-26 01:40:55
eric.araujo
set
messages: +
2011-03-26 01:37:33
pje
set
messages: +
2011-03-25 22:45:57
eric.araujo
set
nosy: + pje, - BreamoreBoy
2011-03-25 22:45:18
eric.araujo
set
nosy: + rhettinger
messages: +
2010-10-19 18:16:25
eric.araujo
set
files: - unnamed
2010-07-30 12:01:06
eric.araujo
set
nosy: + eric.araujo
2010-07-30 10:46:52
ncoghlan
set
messages: +
2010-07-22 19🔞15
ryan.freckleton
set
files: + unnamed
messages: +
2010-07-22 11:45:17
pitrou
set
nosy: + pitrou
messages: +
2010-07-22 11:24:03
paul.moore
set
messages: +
2010-07-21 13:53:16
BreamoreBoy
set
versions: + Python 3.3, - Python 2.7
nosy: + BreamoreBoy
messages: +
stage: patch review -> (no value)
2009-09-13 17:07:30
gvanrossum
set
nosy: + gvanrossum
messages: +
2009-03-02 20:53:36
ncoghlan
set
messages: +
2009-03-02 13:21:27
paul.moore
set
messages: +
2009-03-02 13:21:10
paul.moore
set
messages: -
2009-03-02 13:20:43
paul.moore
set
messages: +
2009-03-02 09:39:10
kteague
set
nosy: + kteague
messages: +
2009-03-01 21:32:10
ncoghlan
set
assignee: ncoghlan ->
messages: +
2009-02-06 19:03:18
paul.moore
set
files: + generic.patch
messages: +
2009-02-05 20:03:22
ncoghlan
set
messages: +
2009-02-05 13:20:02
paul.moore
set
messages: +
2009-02-05 12:16:24
ncoghlan
set
messages: +
2009-02-05 11:59:36
paul.moore
set
messages: +
2009-02-05 10:58:19
ncoghlan
set
messages: +
2009-02-05 03:30:09
ryan.freckleton
set
messages: +
2009-02-04 22:52:47
paul.moore
set
messages: +
2009-02-04 20:19:41
doerwalter
set
nosy: + doerwalter
messages: +
2009-02-04 12:03:16
paul.moore
set
messages: +
2009-02-04 11:48:08
ncoghlan
set
messages: +
2009-02-04 08:27:39
paul.moore
set
files: + generic.patch
messages: +
2009-02-04 08:10:11
paul.moore
set
messages: +
2009-02-04 02:43:20
ryan.freckleton
set
nosy: + ryan.freckleton
messages: +
2009-02-02 19:57:38
paul.moore
create