[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2) (original) (raw)

Kristján Valur Jónsson kristjan at ccpgames.com
Sat Dec 20 11:02:38 CET 2008

Can you distill the program into something reproducible? Maybe with something slightly less than 45Gb but still exhibiting some degradation of exit performance? I can try to point our commercial profiling tools at it and see what it is doing. K

-----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Mike Coleman Sent: 19. desember 2008 23:30 To: python-dev at python.org Subject: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

I have a program that creates a huge (45GB) defaultdict. (The keys are short strings, the values are short lists of pairs (string, int).) Nothing but possibly the strings and ints is shared.

The program takes around 10 minutes to run, but longer than 20 minutes to exit (I gave up at that point). That is, after executing the final statement (a print), it is apparently spending a huge amount of time cleaning up before exiting. I haven't installed any exit handlers or anything like that, all files are already closed and stdout/stderr flushed, and there's nothing special going on. I have done 'gc.disable()' for performance (which is hideous without it)--I have no reason to think there are any loops.

Currently I am working around this by doing an os._exit(), which is immediate, but this seems like a bit of hack. Is this something that needs fixing, or that has already been fixed?

Mike

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com

More information about the Python-Dev mailing list