Issue 1525678: exec and eval allocate lots of memory and do not free it (original) (raw)

I'm not sure if this is a bug. The "bug" is that if I start a new Python session, create a dict or list called d which takes around 2 MB of memory, and then I set d = eval(repr(d)), the Python process now is using ~38 MB of memory more than where it started at. The high memory usage continues even after d is deleted.

Example 1:

% python

Memory use: 3216 KB

d = dict.fromkeys(range(50000))

Memory use: 5400 KB

d = eval('%r' % d)

Memory use: 41620 KB

del d

Memory use: 40080 KB

I am using Python 2.4.1 (#65, Mar 30 2005) on Windows XP SP2 with 512 MB RAM.

If we start with a larger initial dict -- say dict.fromkeys(range(1000**2)), then the line d = eval('%r' % d) can easily cause the process to start paging to disk, even though both the data structure and its string representation fit easily in memory.

Perhaps this behavior is due Python caching bytecodes. One peculiarity about this "bug" is that if Example 1 is repeated with a second variable such as "d2", which is set to the value dict.fromkeys(range(50000,100000)), then the memory usage ends up exactly at 40080 KB after the second "del" statement. If Python were caching the bytecodes, then one would expect the repetition of the example to put the memory usage at ~80000 KB.

Logged In: YES user_id=21627

This "bug" falls into the "won't fix" category, and also in the "already fixed" category.

Python does indeed free the memory; there is no caching going on. It just doesn't return the memory to the operating system. You can see that the memory is really freed by performing the same operating over and over again (say, a thousand times), and watch the memory consumption not grow.

Python obtains the memory not from the system, but from malloc, which obtains it from the system. Whether or not malloc will return memory to the system depends on the malloc implementation; this is out of our control (it's in the Microsoft C library).

However, Python does not return the memory to malloc, either. In the specific case, there are two allocators on top of malloc operating: the integer allocator, and the small objects allocator.

The integer allocator allocates a chunk from malloc and then subdivides it into integer objects. This memory is never returned to malloc; you are using this allocator within the range() function. When the integers are released, the memory becomes available for other integer objects, but not for objects of another kind.

The small objects allocator is likely used for the repr strings of the integers. Object sizes are rounded up to the next multiple of 8 (say, 24), and then a pool of 24-byte-sized blocks is maintained. When the string objects are released, they are released to the pool. In Python 2.4, pool memory is never returned to malloc.

In Python 2.5, this aspect is fixed: under certain circumstances (which are too involved to describe here), pool memory is returned to malloc, which then hopefully returns it to the system.

Closing as "won't fix".