[Python-Dev] Reduce memory footprint of Python (original) (raw)

Victor Stinner victor.stinner at gmail.com
Sun Oct 6 23:26:27 CEST 2013

2013/10/6 <martin at v.loewis.de>:

Quoting Antoine Pitrou <solipsis at pitrou.net>:

The linecache module is used implicitly by the traceback and warnings module, so perhaps quite a bit more than one would imagine :-)

I think limiting the linecache cache size would be good enough. So what specific limit would you suggest?

Before using tracemalloc, I didn't even know that lineache has a cache. And Antoine wrote, linecache is used indirectly by many modules. Even if you get a single line, the module does all the whole file content. I was surprised to see that the cache size can be up to 2 MB.

(To answer to your question, yes, our application on embedded device does format exceptions and tracebacks, so use linecache.)

The re cache is limited. I remember that I saw it once bigger than 700 KB, it was to build a huge range of Unicode characters... but I'm now unable to reproduce the case.

I'm quite sure that the cache has an impact on performances. For example, if you format a traceback where most frames come from the same file, you don't want to decode the file from Unicode for each line. I don't want to remove the cache, and I'm unable to configure its size.

I suggested a function to clear "all" caches, because if you care of memory, you would like to flush all caches (even if you don't know that linecache or re have caches), not anytime, but when your application is idle. I propose a registry because IMO the module should register itself to the registry, the registry should not have to know all modules. (It should not the module to clear its cache ;-))

I worked on a project spending weeks to try to reduce the memory footprint by 1 MB. And it was hard :-) Biggest objects were in the application but it would help to be able to free Python internal caches.

Victor

More information about the Python-Dev mailing list