Where'd my memory go? (was Re: [Python-Dev] Dictionary sparseness) (original) (raw)

Skip Montanaro skip@pobox.com
Mon, 5 May 2003 14:35:23 -0500 

Alex> My only TRUE wish for tuning performance of Python applications is
Alex> to have SOME ways to measure memory footprints with sensible
Alex> guesses about where they come from

Here's a thought. Debug builds appear to now add a getobjects method to sys. Would it be possible to also add another method to sys (also only available on debug builds) which knows just enough about basic builtin object types to say a little about how much space it's consuming? For example, I could do something like this:

allocdict = {}
for o in sys.getobjects(0):
    allocsize = sys.get_object_allocation_size(o)
    # I'm not a fan of {}.setdefault()
    alloc = allocdict.get(type(o), [])
    alloc.append(allocsize) # or alloc.append((allocsize, o))
    allocdict[type(o)] = alloc

Once the list is traversed you can poke around in allocdict figuring out where your memory went (other than to allocdict itself!).

(I was tempted to suggest another method, but I fear that would just spread the mess around. That may also be a viable option though.)

Skip