[Python-Dev] Threading, atexit, and logging (original) (raw)

Tim Peters tim.peters at gmail.com
Wed Dec 6 09:47:26 CET 2006

[Martin v. Löwis]

In bug #1566280 somebody reported that he gets an exception where the logging module tries to write to closed file descriptor.

Upon investigation, it turns out that the file descriptor is closed because the logging atexit handler is invoked. This is surprising, as the program is far from exiting at this point.

But the main thread is done, right? None of this appears to make sense unless we got into Py_Finalize(), and that doesn't happen until the main thread has nothing left to do.

Investigating further, I found that the logging atexit handler is registered after the threading atexit handler, so it gets invoked before the threading's atexit.

Ya, and that sucks. Can't recall details now, but it's not the first time the vagaries of atexit ordering bit a threaded program. IMO, threading shouldn't use atexit at all.

Now, threading's atexit is the one that keeps the application running, by waiting for all non-daemon threads to shut down. As this application does all its work in non-daemon threads, it keeps running for quite a while - except that the logging module gives errors.

The real problem here is that atexit handlers are invoked even though the program logically doesn't exit, yet (it's not just that the threading atexit is invoked after logging atexit - this could happen to any atexit handler that gets registered). I added a patch to this report which makes the MainThread _exitfunc a sys.exitfunc, chaining what is there already. This will work fine for atexit (as atexit is still imported explicitly to register its sys.exitfunc), but it might break if other applications still insist on installing a sys.exitfunc.

Well, that's been officially deprecated since 2.4, but who knows?

What do you think about this approach?

It's expedient :-) So was using atexit for this to begin with. Probably "good enough". I'd rather, e.g., that threading stuff an exit function into a module global, and change Py_Finalize() to look for that and run it (if present) before invoking call_sys_exitfunc(). That is, break all connection between the core's implementation of threading and the user-visible atexit machinery.

atexit is a hack specific to "don't care about order" finalization functions, and it gets increasingly obscure to try to force it to respect a specific ordering sometimes (e.g., now you have a patch to try to fix it by relying on an obscure deprecated feature and hoping users don't screw with that too -- probably "good enough", but still sucky).

More information about the Python-Dev mailing list