[Python-Dev] PEP 442 clarification for type hierarchies (original) (raw)

Stefan Behnel stefan_ml at behnel.de
Mon Aug 5 21:03:33 CEST 2013

Hi,

I was just continuing in my monologue when you replied, so I'll just drop my response below.

Antoine Pitrou, 05.08.2013 20:51:

On Sun, 04 Aug 2013 09:23:41 +0200 Stefan Behnel wrote:

I'm currently catching up on PEP 442, which managed to fly completely below my radar so far. It's a really helpful change that could end up fixing a major usability problem that Cython was suffering from: user provided deallocation code now has a safe execution environment (well, at least in Py3.4+). That makes Cython a prime candidate for testing this, and I've just started to migrate the implementation. That's great to hear. "Safe execution environment" for finalization code is exactly what the PEP is about.

One thing that I found to be missing from the PEP is inheritance handling. The current implementation doesn't seem to care about base types at all, so it appears to be the responsibility of the type to call its super type finalisation function. Is that really intended? Yes, it is intended that users have to call super().del() in their del implementation, if they want to call the upper-level finalizer. This is exactly the same as in init() and (most?) other special functions.

That's the Python side of things. However, if a subtype overwrites tp_finalize(), then there should be a protocol for making sure the super type's tp_finalize() is called, and that it's being called in the right kind of execution environment.

Another bit is the exception handling. According to the documentation, tpfinalize() is supposed to first save the current exception state, then do the cleanup, then call WriteUnraisable() if necessary, then restore the exception state.

http://docs.python.org/3.4/c-api/typeobj.html#PyTypeObject.tpfinalize Is there a reason why this is left to the user implementation, rather than doing it generically right in PyObjectCallFinalizer() ? That would also make it more efficient to call through the super type hierarchy, I guess. I don't see a need to repeat this exception state swapping at each level. I didn't give much thought to this detail. Originally I was simply copying this bit of semantics from tpdealloc and tpdel, but indeed we could do better. Do you want to open an issue about it?

I think the main problem I have with the PEP is this part:

""" The PEP doesn't change the semantics of:

Meaning, it was designed to explicitly ignore this use case. That's a mistake, IMHO. If we are to add a new finalisation protocol, why not make it work in the general case (i.e. fix the problem once and for all), instead of restricting it to a special case and leaving the rest to each user to figure out again?

Separating the finalisation from the deallocation is IMO a good idea. It fixes cyclic garbage collection, that's excellent. And it removes the differences between GC and normal refcounting cleanup by clarifying in what states finalisation and deallocation are executed (one safe, one not).

I think what's missing is the following.

My guess is that a recursive finalisation phase where subtype code calls into the supertype is generally more efficient, so I think I'd prefer that.

I think it's a mistake that the current implementation calls the finalisation from tp_dealloc(). Instead, both the finalisation and the deallocation should be called externally and independently from the cleanup mechanism behind Py_DECREF(). (There is no problem in CS that can't be solved by adding another level of indirection...)

An obvious open question is how to deal with exceptions during finalisation. Any break in the execution chain would mean that a part of the type wouldn't be finalised. One way to handle this could be to simply assume that the deallocation phase would still clean up anything that's left over. Or the protocol could dictate that each level must swallow its own exceptions and call the super type finaliser with a clean exception state. This might suggest that an external iterative call loop through the finaliser hierarchy has a usability advantage over recursive calls.

Just dropping my idea here.

Stefan

More information about the Python-Dev mailing list