[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2) (original) (raw)

Tim Peters tim.peters at gmail.com
Sat Dec 20 21:34:19 CET 2008

[M.-A. Lemburg]

These long exit times are usually caused by the garbage collection of objects. This can be a very time consuming task.

[Leif Walsh]

In that case, the question would be "why is the interpreter collecting garbage when it knows we're trying to exit anyway?".

Because user-defined destructors (like del methods and weakref callbacks) may be associated with garbage, and users presumably want those to execute. Doing so requires identifying identifying garbage and releasing it, same as if the interpreter didn't happen to be exiting.

BTW, the original poster should try this: use whatever tools the OS supplies to look at CPU and disk usage during the long exit. What I /expect/ is that almost no CPU time is being used, while the disk is grinding itself to dust. That's what happens when a large number of objects have been swapped out to disk, and exit processing has to page them all back into memory again (in order to decrement their refcounts). Python's cyclic gc (the gc module) has nothing to do with this -- it's typically the been-there-forever refcount-based non-cyclic gc that accounts for supernaturally long exit times.

If that is the case here, there's no evident general solution. If you have millions of objects still alive at exit, refcount-based reclamation has to visit all of them, and if they've been swapped out to disk it can take a very long time to swap them all back into memory again.

More information about the Python-Dev mailing list