[Python-Dev] bpo-36829: Add sys.unraisablehook() (original) (raw)

Victor Stinner vstinner at redhat.com
Wed May 15 21:23:55 EDT 2019

Hi,

I recently modified Python 3.8 to start logging I/O error in io destructors when using the development mode (-X dev) or when Python is built in debug mode. This change allows to debug bpo-18748 very strange crash in PyThread_exit_thread(), when a thread closes an arbitrary file descriptor by mistake.

The problem is that exceptions raised in a destructor cannot be passed to the current function call: the exception is logged into sys.stderr with a message like "Exception ignored in: ...". It's easy to miss such message in stderr.

Thomas Grainger opened 36829 to propose to add -X abortunraisable command line option: abort immediately the current process (by calling Py_FatalError()) at the first PyErr_WriteUnraisable() call. Thomas had like a way to easily detect these in CI. I'm not satisfied by his proposal, since it only gives little control to the user on how "unraisable exceptions" are handled: the process dies, that's all.

https://bugs.python.org/issue36829

I proposed a different approach: add a new sys.unraisablehook hook which is called to handle an "unraisable exception". To handle them differently, replace the hook. For example, I wrote a custom hook to log these exceptions into a file (the output on the Python test suite is interesting!). It also becomes trivial to reimplement Thomas's idea (kill the process):

import signal
def hook(unraisable): signal.raise_signal(signal.SIGABRT)
sys.unraisablehook = hook

I plan to merge my implementation soon, are you fine with that?

https://github.com/python/cpython/pull/13187

--

The first implementation of my API used sys.unraisablehook(exc_type, exc_value, exc_tb, obj). The problem is that Serhiy Storchaka asked me to add a new error message field which breaks the API: the API is not future-proof.

I modified my API to create an object to pack arguments. The new API becomes sys.unraisablehook(unraisable) where unraisable has 4 fields: exc_type, exc_value, exc_tb, obj. This API is now future-proof: adding a new field will not break existing custom hooks!

By the way, I like this idea of adding an optional error message, and I plan to implement it once sys.unraisablehook is merged ;-) I already implemented it previously, but I reverted my changes to make the PR simpler to review.

Extract of the documentation:

""" Add new :func:sys.unraisablehook function which can be overriden to control how "unraisable exceptions" are handled. It is called when an exception has occurred but there is no way for Python to handle it. For example, when a destructor raises an exception or during garbage collection (:func:gc.collect). """

My implementation has a limitation: if PyErr_WriteUnraisable() is called after the Python finalization cleared sys attributes (almost the last function call of Py_Finalize), the default hook is called instead of the custom hook. In this case, sys.stderr is None and so the default hook does nothing.

These late calls to PyErr_WriteUnraisable() cannot be catched with my approached, whereas Thomas Grainger's command line option "-X abortunraisable" allows that.

My concern with Thomas's approach is that if the problem is killed with SIGABRT by such late PyErr_WriteUnraisable(), a low-level debugger like gdb is needed to investigate the crash, since Python is already finalized and so cannot be used to investigate.

I prefer to allow arbitrary hook with the limitation, rather than always kill the process with SIGABRT at the first PyErr_WriteUnraisable() and require to use a low-level debugger.

Victor

Night gathers, and now my watch begins. It shall not end until my death.

More information about the Python-Dev mailing list