[Python-Dev] Does eval() leak? (original) (raw)

Guido van Rossum guido at python.org
Mon Feb 20 17:31:44 CET 2006

On 2/16/06, John Marshall <John.Marshall at ec.gc.ca> wrote:

Hi,

Should I expect the virtual memory allocation to go up if I do the following? ----- raw = open("data").read() while True: d = eval(raw) ----- I would have expected the memory allocated to the object referenced by d to be deallocated, garbage collected, and reallocated for the new eval(raw) results, assigned to d. The file contains a large, SIMPLE (no self refs; all native python types/objects) dictionary (>300K).

You're probably running into the problem that the concrete parse tree built up by the parser is rather large. While the memory used for that tree is freed to Python's malloc pool, thus making it available for other allocations by the same process, it is likely that the VM allocation for the process will permanently go up.

When I try something like this (*) I see the virtual memory size go up indefinitely with Python 2.3.5, but not with Python 2.4.1 or 2.5(head). Even so, the problem may be fragmentation instead of a memory leak; fragmentation problems are even harded to debug than leaks (since they depend on the heuristics applied by the platform's malloc implementation).

You can file a bug for 2.3 but unless you also provide a patch it's unlikely to be fixed; the memory allocation code was revamped significantly for 2.4 so there's no simple backport of the fix available.

(*) d = {} for i in range(100000): d[repr(i)] = i s = str(d) while 1: x = eval(s); print 'x'

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

More information about the Python-Dev mailing list