[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2) (original) (raw)

Mike Coleman tutufan at gmail.com
Tue Dec 23 00:28:48 CET 2008

On Mon, Dec 22, 2008 at 2:54 PM, Ivan Krstić <krstic at solarsail.hcs.harvard.edu> wrote:

It's still not clear to me, from reading the whole thread, precisely what you're seeing. A self-contained test case, preferably with generated random data, would be great, and save everyone a lot of investigation time.

I'm still working on a test case. The first couple of attempts, using a half-hearted attempt to model the application behavior wrt this dict didn't demonstrate bad behavior.

My impression is that no one's burning much time on this but me at the moment, aside from offering helpful advice. If you are, you might want to wait. I noticed just now that the original hardware was throwing some chipkills, so I'm retesting on something else.

In the meantime, can you 1) turn off all swap files and partitions, and 2) confirm positively that your CPU cycles are burning up in userland?

For (1), I don't have that much control over the machine. Plus, based on watching with top, I seriously doubt the process is using swap in any way. For (2), yes, 100% CPU usage.

(In general, unless you know exactly why your workload needs swap, and have written your program to take swapping into account, having any swap on a machine with 64GB RAM is lunacy. The machine will grind to a complete standstill long before filling up gigabytes of swap.)

The swap is not there to support my application per se. Clearly if you're swapping, generally you're crawling. This host is used by a reasonably large set of non- and novice programmers, who sometimes vacuum up VM without realizing it. If you have a nice, big swap space, you can 'kill -STOP' these offenders, and allow them to swap out while you have a leisurely discussion with the owner and possibly 'kill -CONT' later, as opposed to having to do a quick 'kill -KILL' to save the machine. That's my thinking, anyway.

Mike

More information about the Python-Dev mailing list