[Python-Dev] PEP 487 vs 422 (dynamic class decoration) (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Fri Apr 3 10:21:10 CEST 2015

On 3 April 2015 at 11:32, PJ Eby <pje at telecommunity.com> wrote:

On Thu, Apr 2, 2015 at 6:24 PM, Martin Teichmann <lkb.teichmann at gmail.com> wrote:

The whole point of PEP 487 was to reduce PEP 422 so much that it can be written in python and back-ported. As I said earlier, it's a fine feature and should be in the stdlib for Python 3. (But it should have a noconflict feature added, and it doesn't need a language change.) However, since my specific use case was the one PEP 422 was originally written to solve, and PEP 487 does not address that use case, it is not a suitable substitute for PEP 422. This is also not your fault; you didn't force Nick to withdraw it, after all. ;-) My main concern in this thread, however, is ensuring that either the use case behind PEP 422 doesn't get dropped, or that Nick is now okay with me implementing that feature by monkeypatching buildclass. Since he practically begged me not to do that in 2012, and IIRC specifically created PEP 422 to provide an alternative way for me to accomplish this specific use case, I wanted to see what his current take was. (That is, did he forget the history of the PEP, or does he no longer care about userspace code hooking buildclass? Is there some other proposal that would be a viable alternative? etc.)

It's actually both - I'd forgotten the origins of PEP 422, and I've come around to the view that this level of implicit behaviour is a bad idea because it's inherently confusing. The various incarnations of PEP 422 ended up remaining at least somewhat confusing for me and I wrote it. The "what happens when?" story in PEP 487 is much simpler.

That means I'm now OK with monkeypatching build_class being the only way to get dynamic hooking of the class currently being defined from the class body - folks that really want that behaviour can monkeypatch it in, while folks that think it's a bad idea don't need to worry about.

Now you want to be able to write decorators whose details are filled in at class creation time. Not "now"; it's been possible to do this in Python 2 for over a decade, and code that does so is in current use by other packages. The package providing this feature (DecoratorTools) was downloaded 145 times today, and 3274 times in the past month, so there is active, current use of it by other Python 2 packages. (Though I don't know how many of them depend directly or indirectly upon this particular feature.) Currently, however, it is not possible to port this feature of DecoratorTools (or any other package that uses that feature, recursively) to Python 3, due to the removal of metaclass and the lack of any suitable substitute hook.

Right, any post-namespace-execution modification of class definitions in Python 3 currently has to be declared in the class definition header, either as an explicit decorator, as a metaclass declaration, or through inheritance from a particular base class.

PEP 487 doesn't change that, it just adds a way for the last case to work without needing a custom metaclass in the picture.

PEP 422 did change that, by providing a way for the namespace itself to register a post-execution hook, akin to one of the way's metaclass could be used in Python 2.

What's changed since I first wrote PEP 422 is that I've come to view the requirement to be explicit in Python 3 as a gain for readability, rather than as a limitation to be eliminated - there will always be some hint in the class header that post-namespace-execution modifications may be happening, even if it's just having a declared base class other than object.

Your point is that you want to be able to use your decorators without having to ask users to also inherit a specific class. I personally don't think that's desirable. Many frameworks out there have such kind of decorators and mandatory base classes and that works fine. The intended use case is for generic method decorators that have nothing to do with the base class per se, so inheriting from a specific base-class is an anti-feature in this case.

If that's the use case, it would be preferable to explore enhancing the type based dispatch capabilities in functools as a follow-up to Guido's work on type hinting enhacements.

The only problem remains once you need to inherit more than one of those classes, as their metaclasses most likely clash. This is what PEP 487 fixes. No, it addresses the issue for certain specific metaclass use cases. It does not solve the problem of metaclass conflict in general; for that you need something like the sample noconflict code I posted, which works for Python 3.1+ and doesn't require a language change.

Neither PEP 422 nor 487 are designed to eliminate metaclass conflicts in general, they're primarily designed to let base classes run arbitrary code after the namespace has been executed in a subclass definition without needing a custom metaclass. They both introduce a new tier in the metaprogramming hierarchy between explicit class decorators and full custom metaclasses.

So my opinion is that it is not too hard a requirement to ask a user to inherit a specific mixin class for the sake of using a decorator. If this logic were applied to PEP 487 as it currently stands, the PEP should be rejected, since its use case is even more easily accomplished by inheriting from a specific mixin class. (Since the feature only works on subclasses anyway!)

No, you can't do it currently without risking a backwards incompatibility through the introduction of a custom metaclass.

Further, if the claim is that metaclass conflict potential makes PEP 487 worthy of a language change, then by the same logic method decorators are just as worthy of a language change, since any mixin required to use a method decorator would be just as susceptible to metaclass conflicts as SubclassInit.

There wouldn't be a custom metaclass involved in the native implementation of PEP 487, only in the backport.

Finally, I of course disagree with the conclusion that it's okay to require mixins in order for method decorators to access the containing class, since it is not a requirement in Python 2, due to the availability of the metaclass hook.

Right, that's the crux of the disagreement: is the fact that you can inject a postprocessor into a Python 2 class namespace while it is being executed a good thing, or does it inherently lead to surprising code where it isn't clear where the postprocessing injection happened?

When I first wrote PEP 422 I was of the view that "Python 2 allows class definition postprocessing injection, we should allow it in Python 3 as well". I've since changed my view to "Having to declare post-processing of a class definition up front as a decorator, base class or metaclass is a good thing for readability, as otherwise there's nothing obvious when reading a class definition that tells you whether or not postprocessing may happen, so you have to assume its possible for every class definition".

Further, PEP 422 was previously approved to fix this problem, and has a patch in progress, so I'm understandably upset by its sudden withdrawal and lack of suitable replacement.

PEP 422 has never been approved - it's only ever been Draft or Deferred (and now Withdrawn). If you would like to take it over and champion it as a competitor to PEP 487, I'd be fine with that, I just no longer think it's a good idea myself.

At this point, though, I mostly just want to get some kind of closure. After three years, I'd like to know if this is a yea or nay, so I can port the thing and move on, whether it's through a standardized mechanism or ugly monkeypatching. Honestly, the only reason I'm even discussing this in the first place is because 1) Nick pleaded with me three years ago not to hold off porting until a standardized way of doing this could be added to the language,

My apologies for leading you up the garden path - back then I genuinely thought bringing back the capability was a good idea, but the combination of the original deferral to process the previous lot of feedback, and the more recent discussions with Martin that led to the creation of PEP 487 and the withdrawal of PEP 422 changed my mind.

and 2) I've had some recent inquiries from users about porting PEAK-Rules (which uses this particular feature of DecoratorTools) to Python 3. So I went to check on PEP 422's status, and here we are.

Given my change of heart, I believe that at this point, if you were willing to champion a revived PEP 422 that implemented the behaviour you're after, that would be ideal, with monkeypatching the desired behaviour in as a fallback plan if the PEP is still ultimately rejected. Alternatively, you could go the monkeypatching path first, and then potentially seek standardisation later after you've had some practical experience with it - I now consider it an orthogonal capability to the feature in PEP 487, so the acceptance of the latter wouldn't necessarily preclude acceptance of a hook for class postprocessing injection.

Regards, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list