[Python-3000] New section for PEP 3124 (original) (raw)

Phillip J. Eby pje at telecommunity.com
Tue Jul 24 23:56:28 CEST 2007

Taking the recent threads here, and Guido's comments off-list, I've attempted to put together a coherent response as a new section for the PEP, which I've checked in and included a copy of here. If I have misrepresented anyone's argument, or if you spot something where you have a question or need a clarification, please let me know. Thanks.

Overloading Usage Patterns

In discussion on the Python-3000 list, the proposed feature of allowing arbitrary functions to be overloaded has been somewhat controversial, with some people expressing concern that this would make programs more difficult to understand.

The general thrust of this argument is that one cannot rely on what a function does, if it can be changed from anywhere in the program at any time. Even though in principle this can already happen through monkeypatching or code substitution, it is considered poor practice to do so.

However, providing support for overloading any function (or so the argument goes), is implicitly blessing such changes as being an acceptable practice.

This argument appears to make sense in theory, but it is almost entirely mooted in practice for two reasons.

First, people are generally not perverse, defining a function to do one thing in one place, and then summarily defining it to do the opposite somewhere else! The principal reasons to extend the behavior of a function that has not been specifically made generic are to:

None of these reasons for adding overloads imply any change to the intended default or overall behavior of the existing function, however. Just as a base class method may be overridden by a subclass for these same two reasons, so too may a function be overloaded to provide for such enhancements.

In other words, universal overloading does not equal arbitrary overloading, in the sense that we need not expect people to randomly redefine the behavior of existing functions in illogical or unpredictable ways. If they did so, it would be no less of a bad practice than any other way of writing illogical or unpredictable code!

However, to distinguish bad practice from good, it is perhaps necessary to clarify further what good practice for defining overloads is. And that brings us to the second reason why generic functions do not necessarily make programs harder to understand: overloading patterns in actual programs tend to follow very predictable patterns. (Both in Python and in languages that have no non-generic functions.)

If a module is defining a new generic operation, it will usually also define any required overloads for existing types in the same place. Likewise, if a module is defining a new type, then it will usually define overloads there for any generic functions that it knows or cares about.

As a result, the vast majority of overloads can be found adjacent to either the function being overloaded, or to a newly-defined type for which the overload is adding support. Thus, overloads are highly- discoverable in the common case, as you are either looking at the function or the type, or both.

It is only in rather infrequent cases that one will have overloads in a module that contains neither the function nor the type(s) for which the overload is added. This would be the case if, say, a third-party created a bridge of support between one library's types and another library's generic function(s). In such a case, however, best practice suggests prominently advertising this, especially by way of the module name.

For example, PyProtocols defines such bridge support for working with Zope interfaces and legacy Twisted interfaces, using modules called protocols.twisted_support and protocols.zope_support. (These bridges are done with interface adapters, rather than generic functions, but the basic principle is the same.)

In short, understanding programs in the presence of universal overloading need not be any more difficult, given that the vast majority of overloads will either be adjacent to a function, or the definition of a type that is passed to that function.

And, in the absence of incompetence or deliberate intention to be obscure, the few overloads that are not adjacent to the relevant type(s) or function(s), will generally not need to be understood or known about outside the scope where those overloads are defined. (Except in the "support modules" case, where best practice suggests naming them accordingly.)

More information about the Python-3000 mailing list