[Python-Dev] Fun with 2.3 shutdown (original) (raw)

Tim Peters tim at zope.com
Sun Sep 21 19:50:42 EDT 2003

[Armin Rigo]

The behavior of Python regarding whether threads should continue to run after the main one exited has never been too clear. Wouldn't it make sense to try to control this aspect more precisely ?

threading.py already tried to. Whether or not a native platform thread outlives the main thread varies across platforms. A Thread created via threading.py prevents the main thread from exiting on all platforms so long as the Thread is running, unless the user calls its setDaemon(True) method before starting the thread. Whether a daemon Thread can outlive the main thread remains platform-dependent. These platform dependencies exist because there's no cross-platform way for Python to forcibly end a thread's life from outside the thread, or to force a thread to continue running after the main thread exits. Python can wait for a Thread to end on its own in a cross-platform way, and threading.py keeps the main thread alive until all non-daemon Threads have stopped running via that portable method. So it does what it reasonably could do.

I have also experienced this kind of shutdown glitches.

You mean the uselessly empty "Exception in thread" messages at shutdown? I've only seen those when running Zope3 tests so far (I count non-empty "Exception in thread" shutdown msgs as a different glitch, though). I expect every long-time Python programmer has seen the ever-popular "object 'None' has no attribute so-and-so" kinds of shutdown glitches, and we fix more of those every release by purging del methods, and methods invoked by del methods, of references to module globals. The segfaults at shutdown I saw later while trying to debug the Zope3 symptom were new to me.

It definitely needs more thinking, but what about this one: Python would only shutdown after all threads are finished;

In the Zope3 case, there are about a dozen Threads still running, and they'll never finish. Nine are in an Event.wait() that will never trigger; the rest are in an unbounded polling loop with a sleep(3) each time around. They're all daemon Threads, though, so asked Python explicitly not to wait for them to finish.

but exiting the main thread would by default send a SystemExit exception to all current threads, if that can be done cleanly (oh well, this problem again).

I'm afraid that would be a major change in shutdown semantics for non-daemon Threads, yes? Right now the main thread will happily wait as long as it takes for Threads to decide to stop on their own, and I've seen programs rely on that (fire up a bunch of producer and consumer Threads, then let the main thread fall off the end of the script). It might help the Zope3 problem, though, if SystemExit got raised in daemon Threads when the main thread wanted to exit. But stuffing a pending exception on a Thread's todo list is asynchronous, and in the case of the Zope3 Threads sitting in their sleep(3) polling loops, the SystemExit won't be noticed until the sleep expires. Since 3 might actually be bigger than 3 in some apps , there's really no limit on how long the main thread might have to wait to be sure all the daemon Threads noticed their SystemExit.

Another question, what was the rationale for setting module globals to None instead of simply deleting them (and getting the expected AttributeErrors both at C and at Python level) ?

I suspect it's because there's no "efficient" way to iterate over a dict and simultaneously mutate it in a size-changing way; the C PyDict_Next() protocol allows the caller to change the value associated with existing keys (like setting them to None), but is unpredictable if the caller adds or deletes keys during PyDict_Next() iteration.

Well, due to quirks of the current implementation, I suspect it actually could delete all the keys without harm (the relevant quirk is that the current implementation never resizes a dict in response to a key deletion; it can resize only when a new key gets added; resizing can shuffle the keys around in a new order, which is what makes PyDict_Next unpredictable if the dict does get resized during iteration).

More information about the Python-Dev mailing list