[Python-Dev] PEP 443 - request for pronouncement (original) (raw)

Gustavo Carneiro gjcarneiro at gmail.com
Fri May 31 12🔞07 CEST 2013

Sorry, maybe I am too late to comment on this, but,

@singledispatch ... def fun(arg, verbose=False): ... if verbose: ... print("Let me just say,", end=" ") ... print(arg)

It is not clear from the PEP (up until the end of the User API section at least) when, if ever, is this implementation of fun ever called. I mean, what type of 'arg' triggers a dispatch to this function body? I am guessing that when the arg does not match the type of any of the other registered functions, this function body is used by default. But it is only a guess, the PEP doesn't state this clearly.

If my guess is true, would it be reasonable to update the example "def fun" code to reflect this, e.g., to print("Warning: I do not know what to do with arg {} of type {}".format(arg, type(arg)).

So my comment is just about clarity of the PEP text. I do not wish to interfere with pronouncement.

Thanks.

On Fri, May 31, 2013 at 10:46 AM, Łukasz Langa <lukasz at langa.pl> wrote:

Hello python-dev,

PEP 443 is ready for final review. I'm attaching the latest version below for convenience. The full history of changes is available here: http://hg.python.org/peps/log/tip/pep-0443.txt A reference implementation for PEP 443 is available at: http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l363 with relevant tests here: http://hg.python.org/features/pep-443/file/tip/Lib/test/testfunctools.py#l855 and documentation here: http://hg.python.org/features/pep-443/file/tip/Doc/library/functools.rst#l189 There's also an official backport for 2.6 - 3.3 already up: https://pypi.python.org/pypi/singledispatch

PEP: 443 Title: Single-dispatch generic functions Version: RevisionRevisionRevision Last-Modified: DateDateDate Author: Łukasz Langa <lukasz at langa.pl> Discussions-To: Python-Dev <python-dev at python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 22-May-2013 Post-History: 22-May-2013, 25-May-2013, 31-May-2013 Replaces: 245, 246, 3124 Abstract ======== This PEP proposes a new mechanism in the functools standard library module that provides a simple form of generic programming known as single-dispatch generic functions. A generic function is composed of multiple functions implementing the same operation for different types. Which implementation should be used during a call is determined by the dispatch algorithm. When the implementation is chosen based on the type of a single argument, this is known as single dispatch. Rationale and Goals =================== Python has always provided a variety of built-in and standard-library generic functions, such as len(), iter(), pprint.pprint(), copy.copy(), and most of the functions in the operator module. However, it currently: 1. does not have a simple or straightforward way for developers to create new generic functions, 2. does not have a standard way for methods to be added to existing generic functions (i.e., some are added using registration functions, others require defining _special_ methods, possibly by monkeypatching). In addition, it is currently a common anti-pattern for Python code to inspect the types of received arguments, in order to decide what to do with the objects. For example, code may wish to accept either an object of some type, or a sequence of objects of that type. Currently, the "obvious way" to do this is by type inspection, but this is brittle and closed to extension. Abstract Base Classes make it easier to discover present behaviour, but don't help adding new behaviour. A developer using an already-written library may be unable to change how their objects are treated by such code, especially if the objects they are using were created by a third party. Therefore, this PEP proposes a uniform API to address dynamic overloading using decorators. User API ======== To define a generic function, decorate it with the @singledispatch decorator. Note that the dispatch happens on the type of the first argument, create your function accordingly:: >>> from functools import singledispatch >>> @singledispatch ... def fun(arg, verbose=False): ... if verbose: ... print("Let me just say,", end=" ") ... print(arg) To add overloaded implementations to the function, use the register() attribute of the generic function. It is a decorator, taking a type parameter and decorating a function implementing the operation for that type:: >>> @fun.register(int) ... def (arg, verbose=False): ... if verbose: ... print("Strength in numbers, eh?", end=" ") ... print(arg) ... >>> @fun.register(list) ... def (arg, verbose=False): ... if verbose: ... print("Enumerate this:") ... for i, elem in enumerate(arg): ... print(i, elem) To enable registering lambdas and pre-existing functions, the register() attribute can be used in a functional form:: >>> def nothing(arg, verbose=False): ... print("Nothing.") ... >>> fun.register(type(None), nothing) The register() attribute returns the undecorated function which enables decorator stacking, pickling, as well as creating unit tests for each variant independently:: >>> @fun.register(float) ... @fun.register(Decimal) ... def funnum(arg, verbose=False): ... if verbose: ... print("Half of your number:", end=" ") ... print(arg / 2) ... >>> funnum is fun False When called, the generic function dispatches on the type of the first argument:: >>> fun("Hello, world.") Hello, world. >>> fun("test.", verbose=True) Let me just say, test. >>> fun(42, verbose=True) Strength in numbers, eh? 42 >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True) Enumerate this: 0 spam 1 spam 2 eggs 3 spam >>> fun(None) Nothing. >>> fun(1.23) 0.615 Where there is no registered implementation for a specific type, its method resolution order is used to find a more generic implementation. To check which implementation will the generic function choose for a given type, use the dispatch() attribute:: >>> fun.dispatch(float) <function funnum at 0x104319058> >>> fun.dispatch(dict) <function fun at 0x103fe4788> To access all registered implementations, use the read-only registry attribute:: >>> fun.registry.keys() dictkeys([<class 'NoneType'>, <class 'int'>, <class 'object'>, <class 'decimal.Decimal'>, <class 'list'>, <class 'float'>]) >>> fun.registry[float] <function funnum at 0x1035a2840> >>> fun.registry[object] <function fun at 0x103170788> The proposed API is intentionally limited and opinionated, as to ensure it is easy to explain and use, as well as to maintain consistency with existing members in the functools module. Implementation Notes ==================== The functionality described in this PEP is already implemented in the pkgutil standard library module as simplegeneric. Because this implementation is mature, the goal is to move it largely as-is. The reference implementation is available on hg.python.org [#ref-impl]. The dispatch type is specified as a decorator argument. An alternative form using function annotations has been considered but its inclusion has been deferred. As of May 2013, this usage pattern is out of scope for the standard library [#pep-0008] and the best practices for annotation usage are still debated. Based on the current pkgutil.simplegeneric implementation and following the convention on registering virtual subclasses on Abstract Base Classes, the dispatch registry will not be thread-safe. Abstract Base Classes --------------------- The pkgutil.simplegeneric implementation relied on several forms of method resultion order (MRO). @singledispatch removes special handling of old-style classes and Zope's ExtensionClasses. More importantly, it introduces support for Abstract Base Classes (ABC). When a generic function implementation is registered for an ABC, the dispatch algorithm switches to a mode of MRO calculation for the provided argument which includes the relevant ABCs. The algorithm is as follows:: def composemro(cls, haystack): """Calculates the MRO for a given class cls, including relevant abstract base classes from haystack.""" bases = set(cls.mro) mro = list(cls.mro) for regcls in haystack: if regcls in bases or not issubclass(cls, regcls): continue # either present in the mro or unrelated for index, base in enumerate(mro): if not issubclass(base, regcls): break if base in bases and not issubclass(regcls, base): # Conflict resolution: put classes present in mro # and their subclasses first. index += 1 mro.insert(index, regcls) return mro In its most basic form, it returns the MRO for the given type:: >>> composemro(dict, []) [<class 'dict'>, <class 'object'>] When the haystack consists of ABCs that the specified type is a subclass of, they are inserted in a predictable order:: >>> composemro(dict, [Sized, MutableMapping, str, ... Sequence, Iterable]) [<class 'dict'>, <class 'collections.abc.MutableMapping'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Sized'>, <class 'object'>] While this mode of operation is significantly slower, all dispatch decisions are cached. The cache is invalidated on registering new implementations on the generic function or when user code calls register() on an ABC to register a new virtual subclass. In the latter case, it is possible to create a situation with ambiguous dispatch, for instance:: >>> from collections import Iterable, Container >>> class P: ... pass >>> Iterable.register(P) <class '_main_.P'> >>> Container.register(P) <class '_main_.P'> Faced with ambiguity, @singledispatch refuses the temptation to guess:: >>> @singledispatch ... def g(arg): ... return "base" ... >>> g.register(Iterable, lambda arg: "iterable") <function at 0x108b49110> >>> g.register(Container, lambda arg: "container") <function at 0x108b491c8> >>> g(P()) Traceback (most recent call last): ... RuntimeError: Ambiguous dispatch: <class 'collections.abc.Container'> or <class 'collections.abc.Iterable'> Note that this exception would not be raised if Iterable and Container had been provided as base classes during class definition. In this case dispatch happens in the MRO order:: >>> class Ten(Iterable, Container): ... def iter(self): ... for i in range(10): ... yield i ... def contains(self, value): ... return value in range(10) ... >>> g(Ten()) 'iterable' Usage Patterns ============== This PEP proposes extending behaviour only of functions specifically marked as generic. Just as a base class method may be overridden by a subclass, so too may a function be overloaded to provide custom functionality for a given type. Universal overloading does not equal arbitrary overloading, in the sense that we need not expect people to randomly redefine the behavior of existing functions in unpredictable ways. To the contrary, generic function usage in actual programs tends to follow very predictable patterns and registered implementations are highly-discoverable in the common case. If a module is defining a new generic operation, it will usually also define any required implementations for existing types in the same place. Likewise, if a module is defining a new type, then it will usually define implementations there for any generic functions that it knows or cares about. As a result, the vast majority of registered implementations can be found adjacent to either the function being overloaded, or to a newly-defined type for which the implementation is adding support. It is only in rather infrequent cases that one will have implementations registered in a module that contains neither the function nor the type(s) for which the implementation is added. In the absence of incompetence or deliberate intention to be obscure, the few implementations that are not registered adjacent to the relevant type(s) or function(s), will generally not need to be understood or known about outside the scope where those implementations are defined. (Except in the "support modules" case, where best practice suggests naming them accordingly.) As mentioned earlier, single-dispatch generics are already prolific throughout the standard library. A clean, standard way of doing them provides a way forward to refactor those custom implementations to use a common one, opening them up for user extensibility at the same time. Alternative approaches ====================== In PEP 3124 [#pep-3124] Phillip J. Eby proposes a full-grown solution with overloading based on arbitrary rule sets (with the default implementation dispatching on argument types), as well as interfaces, adaptation and method combining. PEAK-Rules [#peak-rules] is a reference implementation of the concepts described in PJE's PEP. Such a broad approach is inherently complex, which makes reaching a consensus hard. In contrast, this PEP focuses on a single piece of functionality that is simple to reason about. It's important to note this does not preclude the use of other approaches now or in the future. In a 2005 article on Artima [#artima2005] Guido van Rossum presents a generic function implementation that dispatches on types of all arguments on a function. The same approach was chosen in Andrey Popp's generic package available on PyPI [#pypi-generic], as well as David Mertz's gnosis.magic.multimethods [#gnosis-multimethods]. While this seems desirable at first, I agree with Fredrik Lundh's comment that "if you design APIs with pages of logic just to sort out what code a function should execute, you should probably hand over the API design to someone else". In other words, the single argument approach proposed in this PEP is not only easier to implement but also clearly communicates that dispatching on a more complex state is an anti-pattern. It also has the virtue of corresponding directly with the familiar method dispatch mechanism in object oriented programming. The only difference is whether the custom implementation is associated more closely with the data (object-oriented methods) or the algorithm (single-dispatch overloading). PyPy's RPython offers extendabletype [#pairtype], a metaclass which enables classes to be externally extended. In combination with pairtype() and pair() factories, this offers a form of single-dispatch generics. Acknowledgements ================ Apart from Phillip J. Eby's work on PEP 3124 [#pep-3124] and PEAK-Rules, influences include Paul Moore's original issue [#issue-5135] that proposed exposing pkgutil.simplegeneric as part of the functools API, Guido van Rossum's article on multimethods [#artima2005], and discussions with Raymond Hettinger on a general pprint rewrite. Huge thanks to Nick Coghlan for encouraging me to create this PEP and providing initial feedback. References ========== .. [#ref-impl] http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l359 .. [#pep-0008] PEP 8 states in the "Programming Recommendations" section that "the Python standard library will not use function annotations as that would result in a premature commitment to a particular annotation style". (http://www.python.org/dev/peps/pep-0008) .. [#pep-3124] http://www.python.org/dev/peps/pep-3124/ .. [#peak-rules] http://peak.telecommunity.com/DevCenter/PEAK2dRules .. [#artima2005] http://www.artima.com/weblogs/viewpost.jsp?thread=101605 .. [#pypi-generic] http://pypi.python.org/pypi/generic .. [#gnosis-multimethods] http://gnosis.cx/publish/programming/charmingpythonb12.html .. [#pairtype] https://bitbucket.org/pypy/pypy/raw/default/rpython/tool/pairtype.py .. [#issue-5135] http://bugs.python.org/issue5135 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com

-- Gustavo J. A. M. Carneiro "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130531/bcfd1e3c/attachment-0001.html>

More information about the Python-Dev mailing list