[Python-3000] PEP 3124 - Overloading, Generic Functions, Interfaces, etc. (original) (raw)

Chris Monson monpublic at gmail.com
Wed May 9 03:58:13 CEST 2007

On 4/30/07, Phillip J. Eby <pje at telecommunity.com> wrote:

This is just the first draft (also checked into SVN), and doesn't include the details of how the extension API works (so that third-party interfaces and generic functions can interoperate using the same decorators, annotations, etc.). Comments and questions appreciated, as it'll help drive better explanations of both the design and rationales. I'm usually not that good at guessing what other people will want to know (or are likely to misunderstand) until I get actual questions.

PEP: 3124 Title: Overloading, Generic Functions, Interfaces, and Adaptation Version: Revision:55029Revision: 55029 Revision:55029 Last-Modified: Date:2007−04−3018:48:06−0400(Mon,30Apr2007)Date: 2007-04-30 18:48:06 -0400 (Mon, 30 Apr 2007) Date:2007−04−3018:48:06−0400(Mon,30Apr2007) Author: Phillip J. Eby <pje at telecommunity.com> Discussions-To: Python 3000 List <python-3000 at python.org> Status: Draft Type: Standards Track Requires: 3107, 3115, 3119 Replaces: 245, 246 Content-Type: text/x-rst Created: 28-Apr-2007 Post-History: 30-Apr-2007

[snip]

"Before" and "After" Methods ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In addition to the simple next-method chaining shown above, it is sometimes useful to have other ways of combining methods. For example, the "observer pattern" can sometimes be implemented by adding extra methods to a function, that execute before or after the normal implementation. To support these use cases, the overloading module will supply @before, @after, and @around decorators, that roughly correspond to the same types of methods in the Common Lisp Object System (CLOS), or the corresponding "advice" types in AspectJ. Like @when, all of these decorators must be passed the function to be overloaded, and can optionally accept a predicate as well:: def begintransaction(db): print "Beginning the actual transaction"

@before(begintransaction) def checksingleaccess(db: SingletonDB): if db.inuse: raise TransactionError("Database already in use") @after(begintransaction) def startlogging(db: LoggableDB): db.setloglevel(VERBOSE)

If we are looking at doing Design By Contract using @before and @after (preconditions and postconditions), shouldn't there be some way of getting at the return value in functions decorated with @after? For example, it seems reasonable to require an extra argument, perhaps at the beginning:

def successor(num): return num + 1

@before(successor) def check_positive(num: int): if num < 0: raise PreconditionError("Positive integer inputs required")

@after(successor) def check_successor(returned, num:int): if returned != num + 1: raise PostconditionError("successor failed to do its job")

Or am I missing something about how @after works?

+1, BTW, on this whole idea.

@before and @after methods are invoked either before or after

the main function body, and are never considered ambiguous. That is, it will not cause any errors to have multiple "before" or "after" methods with identical or overlapping signatures. Ambiguities are resolved using the order in which the methods were added to the target function.

"Before" methods are invoked most-specific method first, with ambiguous methods being executed in the order they were added. All "before" methods are called before any of the function's "primary" methods (i.e. normal @overload methods) are executed. "After" methods are invoked in the reverse order, after all of the function's "primary" methods are executed. That is, they are executed least-specific methods first, with ambiguous methods being executed in the reverse of the order in which they were added. The return values of both "before" and "after" methods are ignored, and any uncaught exceptions raised by any methods (primary or other) immediately end the dispatching process. "Before" and "after" methods cannot have _proceed_ arguments, as they are not responsible for calling any other methods. They are simply called as a notification before or after the primary methods. Thus, "before" and "after" methods can be used to check or establish preconditions (e.g. by raising an error if the conditions aren't met) or to ensure postconditions, without needing to duplicate any existing functionality.

"Around" Methods ~~~~~~~~~~~~~~~~ The @around decorator declares a method as an "around" method. "Around" methods are much like primary methods, except that the least-specific "around" method has higher precedence than the most-specific "before" or method. Unlike "before" and "after" methods, however, "Around" methods are responsible for calling their _proceed_ argument, in order to continue the invocation process. "Around" methods are usually used to transform input arguments or return values, or to wrap specific cases with special error handling or try/finally conditions, e.g.:: @around(committransaction) def lockwhilecommitting(proceed, db: SingletonDB): with db.globallock: return proceed(db) They can also be used to replace the normal handling for a specific case, by not invoking the _proceed_ function. The _proceed_ given to an "around" method will either be the next applicable "around" method, a DispatchError instance, or a synthetic method object that will call all the "before" methods, followed by the primary method chain, followed by all the "after" methods, and return the result from the primary method chain. Thus, just as with normal methods, _proceed_ can be checked for DispatchError-ness, or simply invoked. The "around" method should return the value returned by _proceed_, unless of course it wishes to modify or replace it with a different return value for the function as a whole. Custom Combinations ~~~~~~~~~~~~~~~~~~~ The decorators described above (@overload, @when, @before, @after, and @around) collectively implement what in CLOS is called the "standard method combination" -- the most common patterns used in combining methods. Sometimes, however, an application or library may have use for a more sophisticated type of method combination. For example, if you would like to have "discount" methods that return a percentage off, to be subtracted from the value returned by the primary method(s), you might write something like this:: from overloading import alwaysoverrides, mergebydefault from overloading import Around, Before, After, Method, MethodList class Discount(MethodList): """Apply return values as discounts""" def call(self, *args, **kw): retval = self.tail(*args, **kw) for sig, body in self.sorted(): retval -= retval * body(*args, **kw) return retval # merge discounts by priority mergebydefault(Discount) # discounts have precedence over before/after/primary methods alwaysoverrides(Discount, Before) alwaysoverrides(Discount, After) alwaysoverrides(Discount, Method) # but not over "around" methods alwaysoverrides(Around, Discount) # Make a decorator called "discount" that works just like the # standard decorators... discount = Discount.makedecorator('discount') # and now let's use it... def price(product): return product.listprice @discount(price) def tenpercentoffshoes(product: Shoe) return Decimal('0.1') Similar techniques can be used to implement a wide variety of CLOS-style method qualifiers and combination rules. The process of creating custom method combination objects and their corresponding decorators is described in more detail under the Extension API section. Note, by the way, that the @discount decorator shown will work correctly with any new predicates defined by other code. For example, if zope.interface were to register its interface types to work correctly as argument annotations, you would be able to specify discounts on the basis of its interface types, not just classes or overloading-defined interface types. Similarly, if a library like RuleDispatch or PEAK-Rules were to register an appropriate predicate implementation and dispatch engine, one would then be able to use those predicates for discounts as well, e.g.:: from somewhere import Pred # some predicate implementation @discount( price, Pred("isinstance(product,Shoe) and" " product.material.name=='Blue Suede'") ) def fortyoffbluesuedeshoes(product): return Decimal('0.4') The process of defining custom predicate types and dispatching engines is also described in more detail under the Extension API section. Overloading Inside Classes -------------------------- All of the decorators above have a special additional behavior when they are directly invoked within a class body: the first parameter (other than _proceed_, if present) of the decorated function will be treated as though it had an annotation equal to the class in which it was defined. That is, this code:: class And(object): # ... @when(getconjuncts) _def conjuncts(self): return self.conjuncts produces the same effect as this (apart from the existence of a private method):: class And(object): # ... @when(getconjuncts) def getconjunctsofand(ob: And): return ob.conjuncts This behavior is both a convenience enhancement when defining lots of methods, and a requirement for safely distinguishing multi-argument overloads in subclasses. Consider, for example, the following code:: class A(object): def foo(self, ob): print "got an object" @overload def foo(proceed, self, ob:Iterable): print "it's iterable!" return proceed(self, ob) class B(A): foo = A.foo # foo must be defined in local namespace @overload def foo(proceed, self, ob:Iterable): print "B got an iterable!" return proceed(self, ob) Due to the implicit class rule, calling B().foo([]) will print "B got an iterable!" followed by "it's iterable!", and finally, "got an object", while A().foo([]) would print only the messages defined in A. Conversely, without the implicit class rule, the two "Iterable" methods would have the exact same applicability conditions, so calling either A().foo([]) or B().foo([]) would result in an AmbiguousMethods error. It is currently an open issue to determine the best way to implement this rule in Python 3.0. Under Python 2.x, a class' metaclass was not chosen until the end of the class body, which means that decorators could insert a custom metaclass to do processing of this sort. (This is how RuleDispatch, for example, implements the implicit class rule.) PEP 3115, however, requires that a class' metaclass be determined before the class body has executed, making it impossible to use this technique for class decoration any more. At this writing, discussion on this issue is ongoing. Interfaces and Adaptation ------------------------- The overloading module provides a simple implementation of interfaces and adaptation. The following example defines an IStack interface, and declares that list objects support it:: from overloading import abstract, Interface class IStack(Interface): @abstract def push(self, ob) """Push 'ob' onto the stack""" @abstract def pop(self): """Pop a value and return it""" when(IStack.push, (list, object))(list.append) when(IStack.pop, (list,))(list.pop) mylist = [] mystack = IStack(mylist) mystack.push(42) assert mystack.pop()==42 The Interface class is a kind of "universal adapter". It accepts a single argument: an object to adapt. It then binds all its methods to the target object, in place of itself. Thus, calling mystack.push(42) is the same as calling IStack.push(mylist, 42). The @abstract decorator marks a function as being abstract: i.e., having no implementation. If an @abstract function is called, it raises NoApplicableMethods. To become executable, overloaded methods must be added using the techniques previously described. (That is, methods can be added using @when, @before, @after, @around, or any custom method combination decorators.) In the example above, the list.append method is added as a method for IStack.push() when its arguments are a list and an arbitrary object. Thus, IStack.push(mylist, 42) is translated to list.append(mylist, 42), thereby implementing the desired operation. (Note: the @abstract decorator is not limited to use in interface definitions; it can be used anywhere that you wish to create an "empty" generic function that initially has no methods. In particular, it need not be used inside a class.) Subclassing and Re-assembly ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interfaces can be subclassed:: class ISizedStack(IStack): @abstract def len(self): """Return the number of items on the stack""" # define len support for ISizedStack when(ISizedStack.len, (list,))(list.len) Or assembled by combining functions from existing interfaces:: class Sizable(Interface): len_ = ISizedStack._len # list now implements Sizable as well as ISizedStack, without # making any new declarations! A class can be considered to "adapt to" an interface at a given point in time, if no method defined in the interface is guaranteed to raise a NoApplicableMethods error if invoked on an instance of that class at that point in time. In normal usage, however, it is "easier to ask forgiveness than permission". That is, it is easier to simply use an interface on an object by adapting it to the interface (e.g. IStack(mylist)) or invoking interface methods directly (e.g. ``IStack.push(mylist, 42)``), than to try to figure out whether the object is adaptable to (or directly implements) the interface. Implementing an Interface in a Class ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is possible to declare that a class directly implements an interface, using the declareimplementation() function:: from overloading import declareimplementation class Stack(object): def init(self): self.data = [] def push(self, ob): self.data.append(ob) def pop(self): return self.data.pop() declareimplementation(IStack, Stack) The declareimplementation() call above is roughly equivalent to the following steps:: when(IStack.push, (Stack,object))(lambda self, ob: self.push(ob)) when(IStack.pop, (Stack,))(lambda self, ob: self.pop()) That is, calling IStack.push() or IStack.pop() on an instance of any subclass of Stack, will simply delegate to the actual push() or pop() methods thereof. For the sake of efficiency, calling IStack(s) where s is an instance of Stack, may return s rather than an IStack adapter. (Note that calling IStack(x) where x is already an IStack adapter will always return x unchanged; this is an additional optimization allowed in cases where the adaptee is known to directly implement the interface, without adaptation.) For convenience, it may be useful to declare implementations in the class header, e.g.:: class Stack(metaclass=Implementer, implements=IStack): ... Instead of calling declareimplementation() after the end of the suite. Interfaces as Type Specifiers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interface subclasses can be used as argument annotations to indicate what type of objects are acceptable to an overload, e.g.:: @overload def traverse(g: IGraph, s: IStack): g = IGraph(g) s = IStack(s) # etc.... Note, however, that the actual arguments are not changed or adapted in any way by the mere use of an interface as a type specifier. You must explicitly cast the objects to the appropriate interface, as shown above. Note, however, that other patterns of interface use are possible. For example, other interface implementations might not support adaptation, or might require that function arguments already be adapted to the specified interface. So the exact semantics of using an interface as a type specifier are dependent on the interface objects you actually use. For the interface objects defined by this PEP, however, the semantics are as described above. An interface I1 is considered "more specific" than another interface I2, if the set of descriptors in I1's inheritance hierarchy are a proper superset of the descriptors in I2's inheritance hierarchy. So, for example, ISizedStack is more specific than both ISizable and ISizedStack, irrespective of the inheritance relationships between these interfaces. It is purely a question of what operations are included within those interfaces -- and the names of the operations are unimportant. Interfaces (at least the ones provided by overloading) are always considered less-specific than concrete classes. Other interface implementations can decide on their own specificity rules, both between interfaces and other interfaces, and between interfaces and classes. Non-Method Attributes in Interfaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The Interface implementation actually treats all attributes and methods (i.e. descriptors) in the same way: their _get_ (and _set_ and _delete_, if present) methods are called with the wrapped (adapted) object as "self". For functions, this has the effect of creating a bound method linking the generic function to the wrapped object. For non-function attributes, it may be easiest to specify them using the property built-in, and the corresponding fget, fset, and fdel attributes:: class ILength(Interface): @property @abstract def length(self): """Read-only length attribute""" # ILength(aList).length == list.len(aList) when(ILength.length.fget, (list,))(list.len) Alternatively, methods such as getfoo() and setfoo() may be defined as part of the interface, and the property defined in terms of those methods, but this a bit more difficult for users to implement correctly when creating a class that directly implements the interface, as they would then need to match all the individual method names, not just the name of the property or attribute. Aspects ------- The adaptation system provided assumes that adapters are "stateless", which is to say that adapters have no attributes or storage apart from those of the adapted object. This follows the "typeclass/instance" model of Haskell, and the concept of "pure" (i.e., transitively composable) adapters. However, there are occasionally cases where, to provide a complete implementation of some interface, some sort of additional state is required. One possibility of course, would be to attach monkeypatched "private" attributes to the adaptee. But this is subject to name collisions, and complicates the process of initialization. It also doesn't work on objects that don't have a _dict_ attribute. So the Aspect class is provided to make it easy to attach extra information to objects that either: 1. have a _dict_ attribute (so aspect instances can be stored in it, keyed by aspect class), 2. support weak referencing (so aspect instances can be managed using a global but thread-safe weak-reference dictionary), or 3. implement or can be adapt to the overloading.IAspectOwner interface (technically, #1 or #2 imply this) Subclassing Aspect creates an adapter class whose state is tied to the life of the adapted object. For example, suppose you would like to count all the times a certain method is called on instances of Target (a classic AOP example). You might do something like:: from overloading import Aspect class Count(Aspect): count = 0 @after(Target.somemethod) def countaftercall(self, *args, **kw): Count(self).count += 1 The above code will keep track of the number of times that Target.somemethod() is successfully called (i.e., it will not count errors). Other code can then access the count using Count(someTarget).count. Aspect instances can of course have _init_ methods, to initialize any data structures. They can use either _slots_ or dictionary-based attributes for storage. While this facility is rather primitive compared to a full-featured AOP tool like AspectJ, persons who wish to build pointcut libraries or other AspectJ-like features can certainly use Aspect objects and method-combination decorators as a base for more expressive AOP tools. XXX spec out full aspect API, including keys, N-to-1 aspects, manual attach/detach/delete of aspect instances, and the IAspectOwner interface. Extension API ============= TODO: explain how all of these work implies(o1, o2) declareimplementation(iface, class) predicatesignatures(ob) parserule(ruleset, body, predicate, actiontype, localdict, globaldict) combineactions(a1, a2) rulesfor(f) Rule objects ActionDef objects RuleSet objects Method objects MethodList objects IAspectOwner

Implementation Notes ==================== Most of the functionality described in this PEP is already implemented in the in-development version of the PEAK-Rules framework. In particular, the basic overloading and method combination framework (minus the @overload decorator) already exists there. The implementation of all of these features in peak.rules.core is 656 lines of Python at this writing. peak.rules.core currently relies on the DecoratorTools and BytecodeAssembler modules, but both of these dependencies can be replaced, as DecoratorTools is used mainly for Python 2.3 compatibility and to implement structure types (which can be done with named tuples in later versions of Python). The use of BytecodeAssembler can be replaced using an "exec" or "compile" workaround, given a reasonable effort. (It would be easier to do this if the funcclosure attribute of function objects was writable.) The Interface class has been previously prototyped, but is not included in PEAK-Rules at the present time. The "implicit class rule" has previously been implemented in the RuleDispatch library. However, it relies on the _metaclass_ hook that is currently eliminated in PEP 3115. I don't currently know how to make @overload play nicely with classmethod and staticmethod in class bodies. It's not really clear if it needs to, however. Copyright ========= This document has been placed in the public domain.

Python-3000 mailing list Python-3000 at python.org http://mail.python.org/mailman/listinfo/python-3000 Unsubscribe: http://mail.python.org/mailman/options/python-3000/monpublic%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20070508/fcf166af/attachment.html

More information about the Python-3000 mailing list