[Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Sun Feb 10 13:32:50 CET 2013

Previous message: [Python-Dev] checking what atexit handlers are registered in Python 3
Next message: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

I wrote PEP 422 (simple class initialization hook) last year after PJE pointed out that Python 3 style metaclasses made some Python 2 code impossible to migrate (since the class body could no longer request modifications to be made to the class after initialization was complete).

It then languished, as I never found the time to actually implement it. Fortunately, Daniel Urban has now agreed to be a co-author on the PEP, and has fixed a couple of lingering technical errors in the PEP, as well as providing a reference implementation at http://bugs.python.org/issue17044

For those that don't recall the original discussion, the proposal is to add a new init_class hook, invoked after the class object is created, but before the class decorators are applied. This provides a simple approach to inherited post-creation modification of classes, without the need for a custom metaclass.

There's currently no BDFL delegate assigned, so this is a request for a pronouncement from Guido.

Regards, Nick.

======================================= PEP: 422 Title: Simple class initialisation hook Version: RevisionRevisionRevision Last-Modified: DateDateDate Author: Nick Coghlan <ncoghlan at gmail.com>, Daniel Urban <urban.dani+py at gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-Jun-2012 Python-Version: 3.4 Post-History: 5-Jun-2012, 10-Feb-2012

Abstract

In Python 2, the body of a class definition could modify the way a class was created (or simply arrange to run other code after the class was created) by setting the __metaclass__ attribute in the class body. While doing this implicitly from called code required the use of an implementation detail (specifically, sys._getframes()), it could also be done explicitly in a fully supported fashion (for example, by passing locals() to a function that calculated a suitable __metaclass__ value)

There is currently no corresponding mechanism in Python 3 that allows the code executed in the class body to directly influence how the class object is created. Instead, the class creation process is fully defined by the class header, before the class body even begins executing.

This PEP proposes a mechanism that will once again allow the body of a class definition to more directly influence the way a class is created (albeit in a more constrained fashion), as well as replacing some current uses of metaclasses with a simpler, easier to understand alternative.

Background

For an already created class cls, the term "metaclass" has a clear meaning: it is the value of type(cls).

During class creation, it has another meaning: it is also used to refer to the metaclass hint that may be provided as part of the class definition. While in many cases these two meanings end up referring to one and the same object, there are two situations where that is not the case:

If the metaclass hint refers to an instance of type, then it is considered as a candidate metaclass along with the metaclasses of all of the parents of the class being defined. If a more appropriate metaclass is found amongst the candidates, then it will be used instead of the one given in the metaclass hint.
Otherwise, an explicit metaclass hint is assumed to be a factory function and is called directly to create the class object. In this case, the final metaclass will be determined by the factory function definition. In the typical case (where the factory functions just calls type, or, in Python 3.3 or later, types.new_class) the actual metaclass is then determined based on the parent classes.

It is notable that only the actual metaclass is inherited - a factory function used as a metaclass hook sees only the class currently being defined, and is not invoked for any subclasses.

In Python 3, the metaclass hint is provided using the metaclass=Meta keyword syntax in the class header. This allows the __prepare__ method on the metaclass to be used to create the locals() namespace used during execution of the class body (for example, specifying the use of collections.OrderedDict instead of a regular dict).

In Python 2, there was no __prepare__ method (that API was added for Python 3 by PEP 3115). Instead, a class body could set the __metaclass__ attribute, and the class creation process would extract that value from the class namespace to use as the metaclass hint. There is published code_ that makes use of this feature.

Another new feature in Python 3 is the zero-argument form of the super() builtin, introduced by PEP 3135. This feature uses an implicit __class__ reference to the class being defined to replace the "by name" references required in Python 2. Just as code invoked during execution of a Python 2 metaclass could not call methods that referenced the class by name (as the name had not yet been bound in the containing scope), similarly, Python 3 metaclasses cannot call methods that rely on the implicit __class__ reference (as it is not populated until after the metaclass has returned control to the class creation machiner).

Proposal

This PEP proposes that a mechanism be added to Python 3 that meets the following criteria:

Restores the ability for class namespaces to have some influence on the class creation process (above and beyond populating the namespace itself), but potentially without the full flexibility of the Python 2 style __metaclass__ hook
Integrates nicely with class inheritance structures (including mixins and multiple inheritance)
Integrates nicely with the implicit __class__ reference and zero-argument super() syntax introduced by PEP 3135
Can be added to an existing base class without a significant risk of introducing backwards compatibility problems

One mechanism that can achieve this goal is to add a new class initialisation hook, modelled directly on the existing instance initialisation hook, but with the signature constrained to match that of an ordinary class decorator.

Specifically, it is proposed that class definitions be able to provide a class initialisation hook as follows::

class Example: @classmethod def init_class(cls): # This is invoked after the class is created, but before any # explicit decorators are called # The usual super() mechanisms are used to correctly support # multiple inheritance. The class decorator style signature helps # ensure that invoking the parent class is as simple as possible.

If present on the created object, this new hook will be called by the class creation machinery after the __class__ reference has been initialised. For types.new_class(), it will be called as the last step before returning the created class object.

If a metaclass wishes to block class initialisation for some reason, it must arrange for cls.__init_class__ to trigger AttributeError.

Note, that when __init_class__ is called, the name of the class is not bound to the new class object yet. As a consequence, the two argument form of super() cannot be used to call methods (e.g., super(Example, cls) wouldn't work in the example above). However, the zero argument form of super() works as expected, since the __class__ reference is already initialised.

This general proposal is not a new idea (it was first suggested for inclusion in the language definition more than 10 years ago, and a similar mechanism has long been supported by Zope's ExtensionClass), but the situation has changed sufficiently in recent years that the idea is worth reconsidering.

Key Benefits

Replaces many use cases for dynamic setting of `metaclass`

For use cases that don't involve completely replacing the defined class, Python 2 code that dynamically set __metaclass__ can now dynamically set __init_class__ instead. For more advanced use cases, introduction of an explicit metaclass (possibly made available as a required base class) will still be necessary in order to support Python 3.

Easier inheritance of definition time behaviour

Understanding Python's metaclasses requires a deep understanding of the type system and the class construction process. This is legitimately seen as challenging, due to the need to keep multiple moving parts (the code, the metaclass hint, the actual metaclass, the class object, instances of the class object) clearly distinct in your mind. Even when you know the rules, it's still easy to make a mistake if you're not being extremely careful. An earlier version of this PEP actually included such a mistake: it stated "subclass of type" for a constraint that is actually "instance of type".

Understanding the proposed class initialisation hook only requires understanding decorators and ordinary method inheritance, which isn't quite as daunting a task. The new hook provides a more gradual path towards understanding all of the phases involved in the class definition process.

Reduced chance of metaclass conflicts

One of the big issues that makes library authors reluctant to use metaclasses (even when they would be appropriate) is the risk of metaclass conflicts. These occur whenever two unrelated metaclasses are used by the desired parents of a class definition. This risk also makes it very difficult to add a metaclass to a class that has previously been published without one.

By contrast, adding an __init_class__ method to an existing type poses a similar level of risk to adding an __init__ method: technically, there is a risk of breaking poorly implemented subclasses, but when that occurs, it is recognised as a bug in the subclass rather than the library author breaching backwards compatibility guarantees. In fact, due to the constrained signature of __init_class__, the risk in this case is actually even lower than in the case of __init__.

Integrates cleanly with \PEP 3135

Unlike code that runs as part of the metaclass, code that runs as part of the new hook will be able to freely invoke class methods that rely on the implicit __class__ reference introduced by PEP 3135, including methods that use the zero argument form of super().

Alternatives

The Python 3 Status Quo

The Python 3 status quo already offers a great deal of flexibility. For changes which only affect a single class definition and which can be specified at the time the code is written, then class decorators can be used to modify a class explicitly. Class decorators largely ignore class inheritance and can make full use of methods that rely on the __class__ reference being populated.

Using a custom metaclass provides the same level of power as it did in Python 2. However, it's notable that, unlike class decorators, a metaclass cannot call any methods that rely on the __class__ reference, as that reference is not populated until after the metaclass constructor returns control to the class creation code.

One major use case for metaclasses actually closely resembles the use of class decorators. It occurs whenever a metaclass has an implementation that uses the following pattern::

class Metaclass(type):
    def __new__(meta, *args, **kwds):
        cls = super(Metaclass, meta).__new__(meta, *args, **kwds)
        # Do something with cls
        return cls

The key difference between this pattern and a class decorator is that it is automatically inherited by subclasses. However, it also comes with a major disadvantage: Python does not allow you to inherit from classes with unrelated metaclasses.

Thus, the status quo requires that developers choose between the following two alternatives:

Use a class decorator, meaning that behaviour is not inherited and must be requested explicitly on every subclass
Use a metaclass, meaning that behaviour is inherited, but metaclass conflicts may make integration with other libraries and frameworks more difficult than it otherwise would be

If this PEP is ultimately rejected, then this is the existing design that will remain in place by default.

Restoring the Python 2 metaclass hook

One simple alternative would be to restore support for a Python 2 style metaclass hook in the class body. This would be checked after the class body was executed, potentially overwriting the metaclass hint provided in the class header.

The main attraction of such an approach is that it would simplify porting Python 2 applications that make use of this hook (especially those that do so dynamically).

However, this approach does nothing to simplify the process of adding inherited class definition time behaviour, nor does it interoperate cleanly with the PEP 3135 __class__ and super() semantics (as with any metaclass based solution, the __metaclass__ hook would have to run before the __class__ reference has been populated.

Dynamic class decorators

The original version of this PEP was called "Dynamic class decorators" and focused solely on a significantly more complicated proposal than that presented in the current version.

As with the current version, it proposed that a new step be added to the class creation process, after the metaclass invocation to construct the class instance and before the application of lexical decorators. However, instead of a simple process of calling a single class method that relies on normal inheritance mechanisms, it proposed a far more complicated procedure that walked the class MRO looking for decorators stored in iterable __decorators__ attributes.

Using the current version of the PEP, the scheme originally proposed could be implemented as::

class DynamicDecorators(Base): @classmethod def init_class(cls): # Process any classes later in the MRO try: mro_chain = super().init_class except AttributeError: pass else: mro_chain() # Process any decorators attributes in the MRO for entry in reversed(cls.mro()): decorators = entry.dict.get("decorators", ()) for deco in reversed(decorators): cls = deco(cls)

Any subclasses of DynamicDecorators would then automatically have the contents of any __decorators__ attributes processed and invoked.

The mechanism in the current PEP is considered superior, as many issues to do with ordering and the same decorator being invoked multiple times just go away, as that kind of thing is taken care of through the use of an ordinary class method invocation.

Automatic metaclass derivation

When no appropriate metaclass is found, it's theoretically possible to automatically derive a metaclass for a new type based on the metaclass hint and the metaclasses of the bases.

While adding such a mechanism would reduce the risk of spurious metaclass conflicts, it would do nothing to improve integration with PEP 3135, would not help with porting Python 2 code that set __metaclass__ dynamically and would not provide a more straightforward inherited mechanism for invoking additional operations after the class invocation is complete.

In addition, there would still be a risk of metaclass conflicts in cases where the base metaclasses were not written with multiple inheritance in mind. In such situations, there's a chance of introducing latent defects if one or more metaclasses are not invoked correctly.

Calling the new hook from `type.init`

Calling the new hook automatically from type.__init__, would achieve most of the goals of this PEP. However, using that approach would mean that __init_class__ implementations would be unable to call any methods that relied on the __class__ reference (or used the zero-argument form of super()), and could not make use of those features themselves.