[Python-3000] pep 3124 plans (original) (raw)

Talin talin at acm.org
Sat Jul 21 07:55:24 CEST 2007

Phillip J. Eby wrote:

At 07:49 AM 7/20/2007 -0700, Guido van Rossum wrote:

On 7/19/07, Joe Smith <unknownkevcat at hotmail.com> wrote:

So the state of the PEP? From the rest of the posts so far, it sounds like there is no real objection to the basic end user API as described in the PEP, Actually I want to reserve judgment on that until the PEP is rewritten to explain and document the underlying mechanisms. It is currently impossible (for me, anyway) to understand how the machinery to support the described features could be built. Without that I cannot approve the PEP. Phillip knows this but is too busy to work on it. Actually, I was under the impression you didn't want the API described in the PEP, and wanted the following changes in addition to dropping method combination, aspects, and interfaces:

I'd like to clarify these requirements a little bit:

On the issue of method combination, aspects, and interfaces: Guido has not made a pronouncement on whether these things may or may not be accepted at some time in the future. What he has said is that he doesn't yet understand the use case for them, and that these should be separate PEPs so that we can argue their merits independently. What he's strongly against (if my understanding is correct) is a "package deal" where he is forced to accept all of the features, or none.

I get the sense that the need for some of these advanced features becomes apparent only after having worked with generics for a while. If that's the case, then the best hope for including them in the stdlib is to get an implementation of generics into the hands of lots of Python programmers so that they can become familiar with them.

* :nextmethod as a keyword-only argument

* @somegeneric.overload as the standard decorator (w/no @overload or @when)

You mentioned earlier that there was a design reason for preferring @overload and @when vs. the earlier RuleDispatch syntax, but the explanation you gave wasn't very clear (to me anyway).

(I personally prefer the @somegeneric.overload, but that's purely an aesthetic value judgement - if there's a strong architectural advantage of the other syntax, I'd like to hear it.)

* advance declaration of a function as overloadable (which is also required by the previous change and by your preference not to modify functions in-place)

Right. There are two reasons that I think that post-hoc overloading runs into problems. The first, as you mentioned, is that it's difficult to implement without some kind of trickery.

The second reason - this is my opinion - is that it too much resembles the mythical "comefrom" statement (the opposite of "goto"). The "comefrom" statement is intended to be a joke - the worst possible language feature from the standpoint of being able to manually trace the flow of execution of a program.

I do think that there are use cases for being able to 'decorate' (in the broader sense) the execution of a function, in an aspect-like way; But I also think that such power should not be used casually, and places where its used should stick out in a way that makes them visually obvious and searchable.

Also, I didn't know you wanted an explanation of how the underlying mechanisms work in general. I thought the only piece you were looking for more explanation of was the method combination machinery -- which would be moot if we're scaling back the API as described by the above.

Just to be sure I'm clear as to what you want, is that the only mechanism you're unclear on, or is the whole thing unclear? The whole thing was inspired by your overloading prototype, I've just made all the concrete bits of it more... "generic".

It seems to me that PEPs should only be required to explain their mechanisms if there's some doubt or controversy about the implementation. It seems to me that this PEP pushes the bounds of what is efficiently doable, so some extra explanation is required.

One issue that hasn't been satisfactorily resolved is the handling of the 'self' parameter. At least, let me give my explanation of what I think the issue is and see if we're on the same page:

Overloading a class method requires special treatment of the 'self' parameter because there's an implicit constraint on what types of objects can be passed as 'self': for any method defined in any class, the 'self' parameter must be an instance of the class (or a subclass) in which the method is defined. Now, this would be trivial if we required the programmer to explicitly declare the type of 'self', but this violates DRY and has the potential to cause mischief if the programmer forgets to update the method signature when they change the class.

In order to avoid this syntactical redundancy, there is a desire to be able to automatically detect the type of the class in which the overload is declared.

This is hard to do, because the "overload" machinery is handled by a function decorator, which runs before the class is actually constructed. Various methods for deducing the class have been proposed, but they have all so far been somewhat problematic, especially in light of "new-style" metaclasses.

I can think of only two approaches for solving this cleanly.

The first is that the overload decorator should be given some C-code help. Now, I recognize that part of your goal was to make the initial prototype a "pure Python" implementation in order to make life easier for Jython/IronPython and friends. That is certainly laudable. However, if the C-code help is a relatively small function that can be reimplemented for the other interpreters, then the impact on portability will be small.

The other approach is to somehow defer the work until after the class is fully constructed. The question then is when will the work be done - in other words, where should the decorator hook its fixup callback?

Even assuming we had some sort of hook that would be triggered when a class has finished construction, then the question is what about non-member generic functions? Since they are not contained in a class body, this hypothetical hook will never be called, and thus the methods won't be "finished". (A way around this would be to say that the only thing that the class-construction hook does is to add the additional type information for 'self', and the method is otherwise finished and ready to go as soon as the decorator is completed.)

If it turns out that there's no way to get a callback when the class has finished being built, then we may have to defer finishing the construction until the first time the generic function is called. This wouldn't be too bad, considering that there's a bunch of other stuff that is lazily calculated on first call anyway, from what I understand.

That is, instead of using issubclass or other explicit relationship tests between overload signatures, I use a generic function implies(). Instead of simply storing a method added as an overload, I use a "combineactions()" generic function to combine it with any method that's already there (possibly including a method type for "No Method Found"). Instead of simply finding the most-specific matching signature on cache misses, I use combineactions() to combine *all applicable* actions (i.e., all those that the calling signature implies()).

The combineactions() function uses another generic function, overrides(), to compare method priorities. overrides() is defined so that Around beats Before beats After beats regular methods beats no method found. The overrides() of two methods of the same type is determined by which signature implies() the other, without also being implied by the other. If there is no overrides() order between two methods, you get an AmbiguousMethod combining the two -- which can be overridden by any method whose signature implies() everything in the AmbiguousMethod. All this is pretty much the same as in your prototype, except that it's done by adding these rules to the generic functions, rather than by hardcoding them. That's why it's bigger than your prototype, but also why it's extensible in terms of adding new method types or ways to specify signatures. I then also added the ability to attach different dispatchers to a function, so that you could replace the simple "tuple of types" matching with more sophisticated engines like RuleDispatch's, while still retaining the ability to use the same method combinations and existing overloads registered for a function. That is, it lets you keep the same API for defining overloads and method combinations as the basic implementation, while allowing the actual overload targets and dispatching mechanisms to vary. That's pretty much it except for Aspects and Interfaces. I've ended up making my Aspect implementation available separately in the ObjectRoles cheeseshop package, renaming them Roles instead of Aspects. (And yes, I will add all the above explanation to the PEP.)

AFAIK Phillip has declared that his implementation only uses (or could be made to only use) isinstance()/issubclass(), and the overriding of these two used by the ABCs is actually very convenient for the GF PEP. Yep. The overload of "implies(c1:type, c2:type)" is "issubclass". "isinstance()" isn't used, since that would render your type-tuple caching strategy unusable.

Python-3000 mailing list Python-3000 at python.org http://mail.python.org/mailman/listinfo/python-3000 Unsubscribe: http://mail.python.org/mailman/options/python-3000/talin%40acm.org

More information about the Python-3000 mailing list