| msg10287 - (view) |
Author: Bernhard Herzog (bernhard) |
Date: 2002-04-12 18:48 |
| The following program leaks memory on Python 2.1.3: import inspect def leak(): frame = inspect.currentframe() while 1: leak() (On Linux at least the process size grows *very* quickly!) System: Python 2.1.3 (#1, Apr 12 2002, 20:09:56) [GCC 2.95.4 20011006 (Debian prerelease)] on linux2 It has no memory problems with Python 2.2. Workaround: del frame in leak. The docs at least should mention this. |
|
|
| msg10288 - (view) |
Author: Tim Peters (tim.peters) *  |
Date: 2002-04-12 19:16 |
| Logged In: YES user_id=31435 Yes, when you create a cyclic structure involving frames in 2.1, it plain leaks -- frames aren't looked at by the cyclic garbage detector in 2.1. Frames were added to cyclic gc collection in 2.2. However, it sure looks to me like this program *still* leaks in 2.2 (and current CVS), just a lot slower (I've watched it grow to over 30MB on Windows, with no sign of ever stopping). But if I call gc.collect() periodically in the "while 1:" (every 100th time) it doesn't leak at all (and stays at about 2MB). Assigning to Neal in the hopes that it will be instantly obvious to him why gc isn't happening without being forced -- or perhaps why gc is happening but is ineffective in this case. Maybe the frameobject free_list is interfering? |
|
|
| msg10289 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2002-04-12 22:00 |
| Logged In: YES user_id=31435 The attached fleak.py seems to show that, under 2.2 and current CVS, we're leaking one frame per top-level call. It prints gc: collecting generation 2... gc: objects in each generation: 0 0 21971 gc: done, 20000 unreachable, 0 uncollectable. 20000 [<frame object at 0x00755B90>] {<type 'frame'>: 20000} However, stranger, if you set VARIANT=1 at the top, the output changes to gc: collecting generation 2... gc: objects in each generation: 0 0 2070 gc: done, 99 unreachable, 0 uncollectable. 99 [<frame object at 0x0077C770>] {<type 'frame'>: 99} So (this will make sense if you look at the code), the *frequency* of gc determines how much stuff is considered unreachable in the end. Disabling the frameobject free_list didn't appear to make any difference to this. |
|
|
| msg10290 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2002-04-12 22:27 |
| Logged In: YES user_id=31435 Ack, I'm hallucinating -- forget fleak.py. What it actually shows is that we're *not* leaking frames(!). Something is still very strange, but I don't know what. Staring at the output from (the much simpler) fleak2.py shows that the 1st generation keeps growing over time, which is weird by itself. gc believes it reclaims everything unreachable each time it triggers too, yet the process grows about 50MB/minute on my box. |
|
|
| msg10291 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2002-04-12 22:39 |
| Logged In: YES user_id=31435 OK, it *is* the frameobject free_list that's screwing us here. Disable that mechanism, and the fleak2.py output is steady over time, and the process doesn't grow over time. gc is triggered by an excess of allocations over deallocations, by some delta N. After the first N frames in cycles are allocated, gc cleans them up, but they all go into the free_list. Then it takes N+N frames in cycles before gc is triggered again (the first N come out of the free_list so don't bump the allocation count at all), and all N+N are returned to the free_list. Then it takes N+N+N frames in cycles before gc is triggered again, etc. Over time, the number of frames in the free_list grows without bound. I'm thinking about putting a bound on it. That is, if frame_dealloc sees there are already K frames on free_list, free the memory instead of keeping it devoted to frames forever. |
|
|
| msg10292 - (view) |
Author: Neil Schemenauer (nascheme) * |
Date: 2002-04-12 22:54 |
| Logged In: YES user_id=35752 Sounds like a sensible solution. I think this is a problem with all GC collected objects that use a free list. I can't think of a nice general solution for it though (other than removing the other free lists :). |
|
|
| msg10293 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2002-04-12 23:05 |
| Logged In: YES user_id=31435 Reassigned to me. Bernhard, can you suggest where the docs could say something in 2.1 that would do you any good? And/or what they could say that would help? Documenting cases where cycles can cause leaks hasn't appeared to do much good over the years -- if someone doesn't already know that assigning the current frame to a local vrbl in that frame causes a cycle, or why such a cycle leaks before 2.2, they're not easy things to get across. |
|
|
| msg10294 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2002-04-13 00:03 |
| Logged In: YES user_id=31435 Neil, are there other gcable containers that use free lists? Offhand I could only think of tuples, but they already put a bound on the their free list size (MAXSAVEDTUPLES). Of course I agree it is a general problem. Another approach could expose part of the guts of _PyObject_GC_Malloc, and tell free-list addicts (FLAs) they have to call that on every "virtual" allocation; similarly for virtual deallocation. The "excess" approach remains, I think, a clever and solid one, and from that view "the problem" is that the FLAs simply don't participate in it. OTOH, FLAs are using a free-list because they're keen to make allocation as fast as possible, so making newing external calls is unattractive on that ground. |
|
|
| msg10295 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2002-04-13 05:25 |
| Logged In: YES user_id=31435 The slower leak in current CVS got plugged via bounding the frameobject free_list, in Objects/frameobject.c; new revision: 2.61 Also backported to the 2.2 branch. Reassigned to Fred for 2.1, in case he can figure out what kind of doc change might help. |
|
|
| msg10296 - (view) |
Author: Bernhard Herzog (bernhard) |
Date: 2002-04-13 14:26 |
| Logged In: YES user_id=2369 For the documentation, perhaps a note in section 3.11.4, "The interpreter stack", might be appropriate. Something like this: In Python 2.1 and earlier, stackframes stored directly or indirectly in local variables can easily cause referency cycles the garbage collector can't collect, leading to memory leaks. To avoid this, it's a good idea to explicitly remove the cycle in a finally: clause. For example: def handle_stackframe_without_leak(): frame = inspect.currentframe() try: # do something with the frame finally: del frame The situation here is very similar to sys.exc_info and the traceback in its return value. |
|
|
| msg10297 - (view) |
Author: Fred Drake (fdrake) |
Date: 2002-04-23 21:21 |
| Logged In: YES user_id=3066 Added text & example based on previous comment to Doc/lib/libinspect.tex revisions 1.3.2.1, 1.10.6.1, and 1.11. |
|
|