[Python-ideas] An even simpler customization of class creation (original) (raw)

Martin Teichmann lkb.teichmann at gmail.com
Fri Feb 27 17:56:08 CET 2015

Previous message: [Python-ideas] Reprs of classes and functions
Next message: [Python-ideas] An even simpler customization of class creation
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hello everybody,

recently, I started a discussion on how metaclasses could be improved. This lead to a major discussion about PEP 422, where I discussed some possible changes to PEP 422. Nick proposed I write a competing PEP, so that the community may decide.

Here it comes. I propose that a class may define a classmethod called init_subclass which initialized subclasses of this class, as a lightweight version of metaclasses.

Compared to PEP 422, this proposal is much simpler to implement, but I do believe that it also makes more sense. PEP 422 has a method called autodecorate which acts on the class currently being generated. This is normally not what one wants: when writing a registering class, for example, one does not want to register the registering class itself. Most evident it become if one wants to introduce such a decorator as a mixin: certainly one does not want to tamper with the mixin class, but with the class it is mixed in!

A lot of the text in my new PEP has been stolen from PEP 422. This is why I hesitated writing myself as the only author of my new PEP. On the other hand, I guess the original authors don't want to be bothered with my stuff, so I credited them in the body of my new PEP.

Greetings

Martin

And here it comes, the draft for a new PEP:

PEP: 422a Title: Simpler customisation of class creation Version: RevisionRevisionRevision Last-Modified: DateDateDate Author: Martin Teichmann <lkb.teichmann at gmail.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Feb-2015 Python-Version: 3.5 Post-History: 27-Feb-2015 Replaces: 422

Abstract

Currently, customising class creation requires the use of a custom metaclass. This custom metaclass then persists for the entire lifecycle of the class, creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation scenarios through a new namespace parameter in the class header, and a new __init_subclass__ hook in the class body.

The new mechanism should be easier to understand and use than implementing a custom metaclass, and thus should provide a gentler introduction to the full power Python's metaclass machinery.

Connection to other PEP

This is a competing proposal to PEP 422 by Nick Coughlan and Daniel Urban. It shares both most of the PEP text and proposed code, but has major differences in how to achieve its goals.

Background

For an already created class cls, the term "metaclass" has a clear meaning: it is the value of type(cls).

During class creation, it has another meaning: it is also used to refer to the metaclass hint that may be provided as part of the class definition. While in many cases these two meanings end up referring to one and the same object, there are two situations where that is not the case:

If the metaclass hint refers to an instance of type, then it is considered as a candidate metaclass along with the metaclasses of all of the parents of the class being defined. If a more appropriate metaclass is found amongst the candidates, then it will be used instead of the one given in the metaclass hint.
Otherwise, an explicit metaclass hint is assumed to be a factory function and is called directly to create the class object. In this case, the final metaclass will be determined by the factory function definition. In the typical case (where the factory functions just calls type, or, in Python 3.3 or later, types.new_class) the actual metaclass is then determined based on the parent classes.

It is notable that only the actual metaclass is inherited - a factory function used as a metaclass hook sees only the class currently being defined, and is not invoked for any subclasses.

In Python 3, the metaclass hint is provided using the metaclass=Meta keyword syntax in the class header. This allows the __prepare__ method on the metaclass to be used to create the locals() namespace used during execution of the class body (for example, specifying the use of collections.OrderedDict instead of a regular dict).

In Python 2, there was no __prepare__ method (that API was added for Python 3 by PEP 3115). Instead, a class body could set the __metaclass__ attribute, and the class creation process would extract that value from the class namespace to use as the metaclass hint. There is published code_ that makes use of this feature.

Another new feature in Python 3 is the zero-argument form of the super() builtin, introduced by PEP 3135. This feature uses an implicit __class__ reference to the class being defined to replace the "by name" references required in Python 2. Just as code invoked during execution of a Python 2 metaclass could not call methods that referenced the class by name (as the name had not yet been bound in the containing scope), similarly, Python 3 metaclasses cannot call methods that rely on the implicit __class__ reference (as it is not populated until after the metaclass has returned control to the class creation machinery).

Finally, when a class uses a custom metaclass, it can pose additional challenges to the use of multiple inheritance, as a new class cannot inherit from parent classes with unrelated metaclasses. This means that it is impossible to add a metaclass to an already published class: such an addition is a backwards incompatible change due to the risk of metaclass conflicts.

Proposal

This PEP proposes that a new mechanism to customise class creation be added to Python 3.5 that meets the following criteria:

Integrates nicely with class inheritance structures (including mixins and multiple inheritance),
Integrates nicely with the implicit __class__ reference and zero-argument super() syntax introduced by PEP 3135,
Can be added to an existing base class without a significant risk of introducing backwards compatibility problems, and
Restores the ability for class namespaces to have some influence on the class creation process (above and beyond populating the namespace itself), but potentially without the full flexibility of the Python 2 style __metaclass__ hook.

Those goals can be achieved by adding two functionalities:

A __init_subclass__ hook that initializes all subclasses of a given class, and
A new keyword parameter namespace to the class creation statement, that gives an initialization of the namespace.

As an example, the first proposal looks as follows::

class SpamBase: # this is implicitly a @classmethod def init_subclass(cls, ns, **kwargs): # This is invoked after a subclass is created, but before # explicit decorators are called. # The usual super() mechanisms are used to correctly support # multiple inheritance.

ns is the classes namespace

       # **kwargs are the keyword arguments to the subclasses'
       # class creation statement
       super().__init_subclass__(cls, ns, **kwargs)

class Spam(SpamBase): pass

the new hook is called on Spam

To simplify the cooperative multiple inheritance case, object will gain a default implementation of the hook that does nothing::

class object: def init_subclass(cls, ns): pass

Note that this method has no keyword arguments, meaning that all methods which are more specialized have to process all keyword arguments.

This general proposal is not a new idea (it was first suggested for inclusion in the language definition more than 10 years ago, and a similar mechanism has long been supported by Zope's ExtensionClass), but the situation has changed sufficiently in recent years that the idea is worth reconsidering for inclusion.

The second part of the proposal is to have a namespace keyword argument to the class declaration statement. If present, its value will be called without arguments to initialize a subclasses namespace, very much like a metaclass __prepare__ method would do.

In addition, the introduction of the metaclass __prepare__ method in PEP 3115 allows a further enhancement that was not possible in Python 2: this PEP also proposes that type.__prepare__ be updated to accept a factory function as a namespace keyword-only argument. If present, the value provided as the namespace argument will be called without arguments to create the result of type.__prepare__ instead of using a freshly created dictionary instance. For example, the following will use an ordered dictionary as the class namespace::

class OrderedBase(namespace=collections.OrderedDict): pass

class Ordered(OrderedBase): # cls.dict is still a read-only proxy to the class namespace, # but the underlying storage is an OrderedDict instance

.. note::

This PEP, along with the existing ability to use  __prepare__ to share a
single namespace amongst multiple class objects, highlights a possible
issue with the attribute lookup caching: when the underlying mapping is
updated by other means, the attribute lookup cache is not invalidated
correctly (this is a key part of the reason class ``__dict__`` attributes
produce a read-only view of the underlying storage).

Since the optimisation provided by that cache is highly desirable,
the use of a preexisting namespace as the class namespace may need to
be declared as officially unsupported (since the observed behaviour is
rather strange when the caches get out of sync).

Key Benefits

Easier use of custom namespaces for a class

Currently, to use a different type (such as collections.OrderedDict) for a class namespace, or to use a pre-populated namespace, it is necessary to write and use a custom metaclass. With this PEP, using a custom namespace becomes as simple as specifying an appropriate factory function in the class header.

Easier inheritance of definition time behaviour

Understanding Python's metaclasses requires a deep understanding of the type system and the class construction process. This is legitimately seen as challenging, due to the need to keep multiple moving parts (the code, the metaclass hint, the actual metaclass, the class object, instances of the class object) clearly distinct in your mind. Even when you know the rules, it's still easy to make a mistake if you're not being extremely careful.

Understanding the proposed implicit class initialization hook only requires ordinary method inheritance, which isn't quite as daunting a task. The new hook provides a more gradual path towards understanding all of the phases involved in the class definition process.

Reduced chance of metaclass conflicts

One of the big issues that makes library authors reluctant to use metaclasses (even when they would be appropriate) is the risk of metaclass conflicts. These occur whenever two unrelated metaclasses are used by the desired parents of a class definition. This risk also makes it very difficult to add a metaclass to a class that has previously been published without one.

By contrast, adding an __init_subclass__ method to an existing type poses a similar level of risk to adding an __init__ method: technically, there is a risk of breaking poorly implemented subclasses, but when that occurs, it is recognised as a bug in the subclass rather than the library author breaching backwards compatibility guarantees.

Integrates cleanly with \PEP 3135

Given that the method is called on already existing classes, the new hook will be able to freely invoke class methods that rely on the implicit __class__ reference introduced by PEP 3135, including methods that use the zero argument form of super().

Replaces many use cases for dynamic setting of `metaclass`

For use cases that don't involve completely replacing the defined class, Python 2 code that dynamically set __metaclass__ can now dynamically set __init_subclass__ instead. For more advanced use cases, introduction of an explicit metaclass (possibly made available as a required base class) will still be necessary in order to support Python 3.

A path of introduction into Python

Most of the benefits of this PEP can already be implemented using a simple metaclass. For the __init_subclass__ hook this works all the way down to python 2.7, while the namespace needs python 3.0 to work. Such a class has been uploaded to PyPI_.

The only drawback of such a metaclass are the mentioned problems with metaclasses and multiple inheritance. Two classes using such a metaclass can only be combined, if they use exactly the same such metaclass. This fact calls for the inclusion of such a class into the standard library, let's call it SubclassMeta, with a base class using it called SublassInit. Once all users use this standard library metaclass, classes from different packages can easily be combined.

But still such classes cannot be easily combined with other classes using other metaclasses. Authors of metaclasses should bear that in mind and inherit from the standard metaclass if it seems useful for users of the metaclass to add more functionality. Ultimately, if the need for combining with other metaclasses is strong enough, the proposed functionality may be introduced into python's type.

Those arguments strongly hint to the following procedure to include the proposed functionality into python:

The metaclass implementing this proposal is put onto PyPI, so that it can be used and scrutinized.
Once the code is properly mature, it can be added to the python standard library. There should be a new module called metaclass which collects tools for metaclass authors, as well as a documentation of the best practices of how to write metaclasses.
If the need of combining this metaclass with other metaclasses is strong enough, it may be included into python itself.

New Ways of Using Classes

This proposal has many usecases like the following. In the examples, we still inherit from the SubclassInit base class. This would become unnecessary once this PEP is included in Python directly.

Subclass registration

Especially when writing a plugin system, one likes to register new subclasses of a plugin baseclass. This can be done as follows::

class PluginBase(SubclassInit): subclasses = []

   def __init_subclass__(cls, ns, **kwargs):
       super().__init_subclass__(ns, **kwargs)
       cls.subclasses.append(cls)

One should note that this also works nicely as a mixin class.

Trait descriptors

There are many designs of python descriptors in the wild which, for example, check boundaries of values. Often those "traits" need some support of a metaclass to work. This is how this would look like with this PEP::

class Trait: def get(self, instance, owner): return instance.dict[self.key]

   def __set__(self, instance, value):
       instance.__dict__[self.key] = value

class Int(Trait): def set(self, instance, value): # some boundary check code here super().set(instance, value)

class HasTraits(SubclassInit): def init_subclass(cls, ns, **kwargs): super().init_subclass(ns, **kwargs) for k, v in ns.items(): if isinstance(v, Trait): v.key = k

The new namespace keyword in the class header enables a number of interesting options for controlling the way a class is initialised, including some aspects of the object models of both Javascript and Ruby.

Order preserving classes

:: class OrderedClassBase(namespace=collections.OrderedDict): pass

class OrderedClass(OrderedClassBase):
    a = 1
    b = 2
    c = 3

Prepopulated namespaces

:: seed_data = dict(a=1, b=2, c=3) class PrepopulatedClass(namespace=seed_data.copy): pass

Cloning a prototype class

:: class NewClass(namespace=Prototype.dict.copy): pass

Rejected Design Options

Calling the hook on the class itself

Adding an __autodecorate__ hook that would be called on the class itself was the proposed idea of PEP 422. Most examples work the same way or even better if the hook is called on the subclass. In general, it is much easier to explicitly call the hook on the class in which it is defined (to opt-in to such a behavior) than to opt-out, meaning that one does not want the hook to be called on the class it is defined in.

This becomes most evident if the class in question is designed as a mixin: it is very unlikely that the code of the mixin is to be executed for the mixin class itself, as it is not supposed to be a complete class on its own.

The original proposal also made major changes in the class initialization process, rendering it impossible to back-port the proposal to older python versions.

Other variants of calling the hook

Other names for the hook were presented, namely __decorate__ or __autodecorate__. This proposal opts for __init_subclass__ as it is very close to the __init__ method, just for the subclass, while it is not very close to decorators, as it does not return the class.

Requiring an explicit decorator on `__init_subclass__`

One could require the explicit use of @classmethod on the __init_subclass__ decorator. It was made implicit since there's no sensible interpretation for leaving it out, and that case would need to be detected anyway in order to give a useful error message.

This decision was reinforced after noticing that the user experience of defining __prepare__ and forgetting the @classmethod method decorator is singularly incomprehensible (particularly since PEP 3115 documents it as an ordinary method, and the current documentation doesn't explicitly say anything one way or the other).

Passing in the namespace directly rather than a factory function

At one point, PEP 422 proposed that the class namespace be passed directly as a keyword argument, rather than passing a factory function. However, this encourages an unsupported behaviour (that is, passing the same namespace to multiple classes, or retaining direct write access to a mapping used as a class namespace), so the API was switched to the factory function version.

Possible Extensions

Some extensions to this PEP are imaginable, which are postponed to a later pep:

A __new_subclass__ method could be defined which acts like a __new__ for classes. This would be very close to __autodecorate__ in PEP 422.
__subclasshook__ could be made a classmethod in a class instead of a method in the metaclass.