[Python-Dev] refleak hunting season (original) (raw)

Michael Hudson mwh at python.net
Mon Aug 2 16:39:36 CEST 2004

• Previous message: [Python-Dev] Re: Can we limit the effects of module execution to sys.modules? (was Fix import errors to have data)
• Next message: [Python-Dev] refleak hunting season
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Last time round I found a bunch of reference counting bugs in Python just after a major release; I'm trying to do it all before 2.4.0 this time.

I've hacked regrtest again (patch attached). Here's my notes file, though as is the case with such things writing stuff down tends to cause thoughts which cause experiments which cause the notes to be out of date more or less immediately...

test_codeccallbacks leaked [2, 2, 2, 2] references registries

test_copy leaked [82, 82, 82, 82] references copy_reg.dispatch_table -- now taken care of in regrtest.py

test_descr leaked [2, 2, 2, 2] references resurrection, still

test_descrtut leaked [1, 1, 1, 1] references exec 'print x' in defaultdict() fixed, need to check in (bit subtle though)

test_distutils leaked [2, 2, 2, 2] references no idea

test_gc leaked [18, 18, 18, 18] references trashcan

test_hotshot leaked [111, 111, 111, 111] references there's been a patch on SF for this for about a year!

test_minidom leaked [1, 1, 1, 1] references testPickledDocument, don't know more

test_mutants leaked [74, -1123, 1092, -521] references randomized test

test_pkg leaked [3, 3, 3, 3] references something to do with execfile? or tempfile.mkstemp? the leak is on the dummy key from dictobject.c!! very odd. see bug #808596 for the explanation.

test_pkgimport leaked [6, 6, 6, 6] references no idea similar to the above? (but certainly not the same) I think this might be caused by interning like the above.

test_sax leaked [1899, 1899, 1899, 1899] references no idea -- scary numbers, though!

test_threadedtempfile leaked [0, 1, -21, 9] references randomized test?

test_threading_local leaked [9, 0, 7, 0] references not sure what's going on here; not sure it's a leak, but it's definitely odd

test_unicode leaked [7, 7, 7, 7] references the codec registry

test_urllib2 leaked [120, -80, -70, 120] references CacheFTPHandler

test_xrange leaked [1, 1, 1, 1] references fixed in CVS

I'd really appreciate someone taking a look at test_sax. The numbers for test_gc and test_descr are artifacts and if someone sees how to fix them, I'd be grateful. test_threading_local behaves very strangely. test_distutils I haven't even begun to look at.

Generally, we're in pretty good shape! (Or our tests really suck, your call ;-).

Cheers, mwh

-------------- next part -------------- Index: regrtest.py

RCS file: /cvsroot/python/python/dist/src/Lib/test/regrtest.py,v retrieving revision 1.156 diff -c -r1.156 regrtest.py *** regrtest.py 26 Jul 2004 12:09:13 -0000 1.156 --- regrtest.py 2 Aug 2004 14:31:41 -0000

*** 8,26 ****

Command line options:

! -v: verbose -- run tests in verbose mode with output to stdout ! -q: quiet -- don't print anything except if a test fails ! -g: generate -- write the output file for a test instead of comparing it ! -x: exclude -- arguments are tests to exclude ! -s: single -- run only a single test (see below) ! -r: random -- randomize test execution order ! -f: fromfile -- read names of tests to run from a file (see below) ! -l: findleaks -- if GC is available detect tests that leak memory ! -u: use -- specify which special resource intensive tests to run ! -h: help -- print this text and exit ! -t: threshold -- call gc.set_threshold(N) ! -T: coverage -- turn on code coverage using the trace module ! -L: runleaks -- run the leaks(1) command just before exit

If non-option arguments are present, they are names for tests to run, unless -x is given, in which case they are names for tests not to run. --- 8,27 ----

Command line options:

! -v: verbose -- run tests in verbose mode with output to stdout ! -q: quiet -- don't print anything except if a test fails ! -g: generate -- write the output file for a test instead of comparing it ! -x: exclude -- arguments are tests to exclude ! -s: single -- run only a single test (see below) ! -r: random -- randomize test execution order ! -f: fromfile -- read names of tests to run from a file (see below) ! -l: findleaks -- if GC is available detect tests that leak memory ! -u: use -- specify which special resource intensive tests to run ! -h: help -- print this text and exit ! -t: threshold -- call gc.set_threshold(N) ! -T: coverage -- turn on code coverage using the trace module ! -L: runleaks -- run the leaks(1) command just before exit ! -R: huntrleaks -- search for reference leaks (needs debug build, v. slow)

If non-option arguments are present, they are names for tests to run, unless -x is given, in which case they are names for tests not to run.

*** 84,89 **** --- 85,91 ---- import getopt import random import warnings

• import sre import cStringIO import traceback

*** 127,133 ****

def main(tests=None, testdir=None, verbose=0, quiet=False, generate=False, exclude=False, single=False, randomize=False, fromfile=None, ! findleaks=False, use_resources=None, trace=False, runleaks=False): """Execute a test suite.

  This also parses command-line options and modifies its behavior

--- 129,136 ----

def main(tests=None, testdir=None, verbose=0, quiet=False, generate=False, exclude=False, single=False, randomize=False, fromfile=None, ! findleaks=False, use_resources=None, trace=False, runleaks=False, ! huntrleaks=False): """Execute a test suite.

  This also parses command-line options and modifies its behavior

*** 152,162 ****

  test_support.record_original_stdout(sys.stdout)
  try:

! opts, args = getopt.getopt(sys.argv[1:], 'hvgqxsrf:lu:t:TL', ['help', 'verbose', 'quiet', 'generate', 'exclude', 'single', 'random', 'fromfile', 'findleaks', 'use=', 'threshold=', 'trace', ! 'runleaks' ]) except getopt.error, msg: usage(2, msg) --- 155,165 ----

  test_support.record_original_stdout(sys.stdout)
  try:

! opts, args = getopt.getopt(sys.argv[1:], 'hvgqxsrf:lu:t:TLR', ['help', 'verbose', 'quiet', 'generate', 'exclude', 'single', 'random', 'fromfile', 'findleaks', 'use=', 'threshold=', 'trace', ! 'runleaks', 'huntrleaks' ]) except getopt.error, msg: usage(2, msg)

*** 191,196 **** --- 194,201 ---- gc.set_threshold(int(a)) elif o in ('-T', '--coverage'): trace = True

•     elif o in ('-R', '--huntrleaks'):

•         huntrleaks = True
      elif o in ('-u', '--use'):
          u = [x.lower() for x in a.split(',')]
          for r in u:

*** 288,294 **** tracer.runctx('runtest(test, generate, verbose, quiet, testdir)', globals=globals(), locals=vars()) else: ! ok = runtest(test, generate, verbose, quiet, testdir) if ok > 0: good.append(test) elif ok == 0: --- 293,299 ---- tracer.runctx('runtest(test, generate, verbose, quiet, testdir)', globals=globals(), locals=vars()) else: ! ok = runtest(test, generate, verbose, quiet, testdir, huntrleaks) if ok > 0: good.append(test) elif ok == 0:

*** 397,403 **** tests.sort() return stdtests + tests

! def runtest(test, generate, verbose, quiet, testdir=None): """Run a single test. test -- the name of the test generate -- if true, generate output, instead of running the test --- 402,408 ---- tests.sort() return stdtests + tests

! def runtest(test, generate, verbose, quiet, testdir=None, huntrleaks=False): """Run a single test. test -- the name of the test generate -- if true, generate output, instead of running the test

*** 415,420 **** --- 420,427 ---- cfp = None else: cfp = cStringIO.StringIO()

• if huntrleaks:

•     refrep = open("reflog.txt", "a")
  try:
      save_stdout = sys.stdout
      try:

*** 435,440 **** --- 442,478 ---- indirect_test = getattr(the_module, "test_main", None) if indirect_test is not None: indirect_test()

•         if huntrleaks:

•             import copy_reg

•             fs = warnings.filters[:]

•             ps = copy_reg.dispatch_table.copy()

•             import gc

•             def cleanup():

•                 import _strptime, urlparse, warnings

•                 warnings.filters[:] = fs

•                 gc.collect()

•                 sre.purge()

•                 _strptime._regex_cache.clear()

•                 urlparse.clear_cache()

•                 copy_reg.dispatch_table.clear()

•                 copy_reg.dispatch_table.update(ps)

•             deltas = []

•             if indirect_test:

•                 for i in range(9):

•                     rc = sys.gettotalrefcount()

•                     indirect_test()

•                     cleanup()

•                     deltas.append(sys.gettotalrefcount() - rc - 2)

•             else:

•                 for i in range(9):

•                     rc = sys.gettotalrefcount()

•                     reload(the_module)

•                     cleanup()

•                     deltas.append(sys.gettotalrefcount() - rc - 2)

•             if max(map(abs, deltas[-4:])) > 0:

•                 print >>refrep, test, 'leaked', \

•                       deltas[-4:], 'references'

•                 #refrep.flush()
      finally:
          sys.stdout = save_stdout
  except test_support.ResourceDenied, msg:

*** 486,492 **** fp.close() else: expected = test + "\n" ! if output == expected: return 1 print "test", test, "produced unexpected output:" sys.stdout.flush() --- 524,530 ---- fp.close() else: expected = test + "\n" ! if output == expected or huntrleaks: return 1 print "test", test, "produced unexpected output:" sys.stdout.flush() -------------- next part --------------

-- I never disputed the Perl hacking skill of the Slashdot creators. My objections are to the editors' taste, the site's ugly visual design, and the Slashdot community's raging stupidity. -- http://www.cs.washington.edu/homes/klee/misc/slashdot.html#faq

• Previous message: [Python-Dev] Re: Can we limit the effects of module execution to sys.modules? (was Fix import errors to have data)
• Next message: [Python-Dev] refleak hunting season
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the Python-Dev mailing list