[Python-Dev] Reduce memory footprint of Python (original) (raw)

Antoine Pitrou solipsis at pitrou.net
Sun Oct 6 17:45:41 CEST 2013

Hi,

On Sun, 6 Oct 2013 17:32:37 +0200 Victor Stinner <victor.stinner at gmail.com> wrote:

Slowly, I'm trying to see if it would be possible to reduce the memory footprint of Python using the tracemalloc module. First, I noticed that linecache can allocate more than 2 MB. What do you think of adding a registry of "clear cache" functions? For exemple, re.purge() and linecache.clearcache(). gc.collect() clears free lists. I don't know if gc.collect() should be related to this new registy (clear all caches) or not.

Rather than "clear" the cache, perhaps limit its size? AFAICT, linecache is used in the warnings and traceback modules.

The dictionary of interned Unicode strings can be large: up to 1.5 MB (with +30,000 strings). Just the dictionary, excluding size of strings. Is the size normal or not? Using tracemalloc, this dictionary is usually to largest memory block.

You don't tell us what the total memory occupation is. A 1.5MB overhead is not the same on a 10MB total or on a 50MB total. (also, how many modules were loaded? large third-party libraries?)

The interned strings dictionary could perhaps be replaced with a set?

unittest doesn't look to release memory (the TestCase class) after the execution of a test.

testimport.testmodulewithlargestack() creates a large Python module and import it, but it does not unload it.

That's not really relevant to memory use on production systems, though.

Regards

Antoine.

More information about the Python-Dev mailing list