[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2) (original) (raw)

Steve Holden steve at holdenweb.com
Sat Dec 20 23:44:28 CET 2008

Antoine Pitrou wrote:

Leif Walsh <leif.walsh gmail.com> writes:

It might be a semantic change that I'm looking for here, but it seems to me that if you turn off the garbage collector, you should be able to expect that either it also won't run on exit, or it should have a way of letting you tell it not to run on exit. [...] I'm skeptical that it's a garbage collector problem. The script creates one dict containing lots of strings and ints. The thing is, strings and ints aren't tracked by the GC as they are simple atomic objects. Therefore, the /only/ object created by the script which is tracked by the GC is the dict. Moreover, since there is no cycle created, the dict should be directly destroyed when its last reference dies (the "del" statement), not go through the garbage collection process. Given that the problem is reproduced on certain systems and not others, it can be related to an interaction between allocation patterns of the dict implementation, the Python memory allocator, and the implementation of the C malloc() / free() functions. I'm no expert enough to find out more on the subject. I believe the OP engendered a certain amount of confusion by describing object deallocation as being performed by the garbage collector. So he perhaps didn't understand that even decref'ing all the objects only referenced by the dict will take a huge amount of time unless there's enough real memory to hold it.

regards Steve

Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/

More information about the Python-Dev mailing list