| msg204604 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2013-11-27 17:49 |
| Someone proposed to use the new tracemalloc module in test.regrtest, I don't remember who. Maybe Nick Coghlan? I tried an hack dash_R(), but the result was not reliable. I should try harder :-) |
|
|
| msg220343 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2014-06-12 12:48 |
| I tried different options to show the memory leaks found by tracemalloc, but I'm not satisfied by any option. I get a lot of noise, the output is almost useless. Randomly, between 1 and 1000 KB are allocated or released in random files: in unittest or linecache modules for example. It looks like the major issue is that the unittest leaks memory. For example, there are still 4 TestCase instances alive after the execution of test_sys. But later, these instances are deleted. It looks like it creates reference leaks and so memory is only released late. Calling gc.collect() doesn't help. I remember that I already saw something strange with the _Outcome class used in unittest.TestCase.run(). See the issue #19880 (changeset 09658ea0b93d). The asyncio module has a similar issue: it stores an exception which indirectly contains a reference cycle in the Exception.__traceback__ attribute. http://bugs.python.org/issue17911 https://code.google.com/p/tulip/issues/detail?id=155 |
|
|
| msg220344 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2014-06-12 12:50 |
| regrtest.patch is a work-in-progress patch. It shows the "top 10" when the -R option is used, ex: "python -m test -R 3:3 test_sys". |
|
|
| msg222697 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2014-07-10 20:28 |
| The problem is snapshots will by themselves create allocation noise: you cannot really use tracemalloc to detect leaks, only to diagnose the leaks you have detected. |
|
|
| msg222698 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2014-07-10 20:30 |
| > The problem is snapshots will by themselves create allocation noise: you cannot really use tracemalloc to detect leaks, only to diagnose the leaks you have detected. It's trivial to ignore allocations done in the tracemalloc module. My work-in-progress patch uses for example these filters: tracemalloc_filters = [ tracemalloc.Filter(False, '<frozen importlib._bootstrap>', all_frames=True), tracemalloc.Filter(False, tracemalloc.__file__, all_frames=True), ] |
|
|
| msg222701 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2014-07-10 20:55 |
| > My work-in-progress patch uses for example these filters If you use filters, you need to add filters for fnmatch and a couple other things :) |
|
|
| msg222702 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2014-07-10 21:15 |
| 2014-07-10 22:55 GMT+02:00 Antoine Pitrou <report@bugs.python.org>: > If you use filters, you need to add filters for fnmatch and a couple other things :) tracemalloc.Filter(False, tracemalloc.__file__, all_frames=True) ignores all memory allocated directly or indirectly by the tracemalloc module: notice the all_frames=True parameter. I'm saving at least 5 frames in tracemalloc when I work on this issue. |
|
|
| msg225266 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2014-08-13 07:49 |
| Some of the changes to the way regrtest handles test cases has shown a weird cycle in test_codecs - see issue #22166 for details. I'll try Victor's patch to see if it's more enlightening than my hacked together attempt. |
|
|
| msg225267 - (view) |
Author: Alyssa Coghlan (ncoghlan) * |
Date: 2014-08-13 07:55 |
| On second thoughts, what Victor is doing isn't that different from my existing hack - I'll lift the display code, but I'm happy with my data collection code. One thing I did that seemed to reduce the noise substantially was to leave tracing turned off most of the time, and have a start/stop pair around the actual execution of the test and the subsequent cache cleanup. |
|
|
| msg225277 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2014-08-13 13:25 |
| Slightly unrelated, but in numba I've rewritten parallel testing and refleak testing as regular unittest extensions (TestRunner / TestResult subclasses). It also means you have per-test method refleak results, instead of per-test file. See e.g. RefleakTestResult in https://github.com/numba/numba/blob/master/numba/tests/__init__.py |
|
|
| msg238421 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2015-03-18 12:44 |
| I'm no more interested to work on this issue. regrtest.patch is unstable, and I don't know how to make it more reliable. Antoine's allocation counter is enough right now. |
|
|