[Python-bugs-list] [ python-Bugs-227699 ] Memory leaks using imp.load_module (original) (raw)

noreply@sourceforge.net noreply@sourceforge.net
Thu, 18 Oct 2001 16:28:23 -0700

Bugs item #227699, was opened at 2001-01-05 12:30 You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=105470&aid=227699&group_id=5470

Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Charles G Waldman (cgw) Assigned to: Barry Warsaw (bwarsaw) Summary: Memory leaks using imp.load_module

Initial Comment: #!/usr/bin/env python

import os import sys import imp

print os.getpid()

sys.path.append("/tmp")

count = 0

while (count<1000): modname = "testmod%s" % count count = count + 1

filename = '/tmp/'  + modname + '.py'
f  = open(filename, 'w')
for x in range(10000):
    f.write('x%s=%s\n'%(x,x))
f.close()

modinfo = imp.find_module(modname)
print 'loading', modname
m = apply(imp.load_module, ('testmod',) + modinfo)

run "watch -n 1 ps up

where is the pid printed out by this program

Comment By: Guido van Rossum (gvanrossum) Date: 2001-10-18 16:28

Message: Logged In: YES user_id=6380

Barry, note that he's cheating with the module name passed to load_module(): the module name is always set to 'testmod', so he's in fact overwriting the module on each load. (This acts the same way as reload(), so it's not creating new module objects -- it's loading the code in the same namespace over and over.)

I see the process size grow very quickly to 15M, probably on account of the huge parse tree for those 10,000 assignment statements (much less for the 10,000 actual variables created). After that it gradually grows -- after 100 loads it's up to 27 M.

Maybe Insure doesn't see this because the data is actually kept alive somewhere?

Comment By: Barry Warsaw (bwarsaw) Date: 2001-10-18 15:30

Message: Logged In: YES user_id=12800

I'm having a hard time tracking down the leak. I don't believe it's at the C layer since preliminary Insure runs don't indicate any leaked blocks (there are plenty of inuse blocks, but those generally don't count).

I will note that if you comment out the imp.load_module call the memory usage remains constant.

Loading each module though is going to naturally consume more memory, right? And those loaded modules aren't ever going to get freed, at least without mucking about in sys.modules. The memory consumption with ps is pretty meager, so I suspect we don't have a valid leak here, but just normal memory consumption due to importation of 10k modules.

I'm running a more detailed insure run right now just to be sure, but I'm skeptical that there's really a bug here.

Comment By: Barry Warsaw (bwarsaw) Date: 2001-02-23 11:53

Message: I can verify that memory grows with this process. I will run it under insure to get more information.

Comment By: Guido van Rossum (gvanrossum) Date: 2001-01-18 19:43

Message: Barry...?!?!

Comment By: Fred L. Drake, Jr. (fdrake) Date: 2001-01-07 15:32

Message: Assigned to Barry since he's the memory leak expert (& has the right tools).

You can respond by visiting: http://sourceforge.net/tracker/?func=detail&atid=105470&aid=227699&group_id=5470