[Python-Dev] Reduce memory footprint of Python (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Mon Oct 7 03:31:34 CEST 2013

On 7 Oct 2013 07:28, "Victor Stinner" <victor.stinner at gmail.com> wrote:

2013/10/6 <martin at v.loewis.de>: > Quoting Antoine Pitrou <solipsis at pitrou.net>: > >> The linecache module is used implicitly by the traceback and warnings >> module, so perhaps quite a bit more than one would imagine :-) >> >> I think limiting the linecache cache size would be good enough. > > So what specific limit would you suggest? Before using tracemalloc, I didn't even know that lineache has a cache. And Antoine wrote, linecache is used indirectly by many modules. Even if you get a single line, the module does all the whole file content. I was surprised to see that the cache size can be up to 2 MB. (To answer to your question, yes, our application on embedded device does format exceptions and tracebacks, so use linecache.) The re cache is limited. I remember that I saw it once bigger than 700 KB, it was to build a huge range of Unicode characters... but I'm now unable to reproduce the case. I'm quite sure that the cache has an impact on performances. For example, if you format a traceback where most frames come from the same file, you don't want to decode the file from Unicode for each line. I don't want to remove the cache, and I'm unable to configure its size. I suggested a function to clear "all" caches, because if you care of memory, you would like to flush all caches (even if you don't know that linecache or re have caches), not anytime, but when your application is idle. I propose a registry because IMO the module should register itself to the registry, the registry should not have to know all modules. (It should not the module to clear its cache ;-))

We kind of have such a manual registry already: the reference leak hunting in regrtest needs to know about a lot of these caches so it can clear them.

Something like a "gc.purge_caches" operation with a "gc.atpurge" registration function sounds like a reasonable operation to me (especially if it offered a way to get a list of which modules had registered cache purging functions).

Like any import time side effects, we'd need to look carefully at the impact it has on the lifecycle of the module globals. We'd also need to be sure that any caches we registered really were pure caches.

Cheers, Nick.

I worked on a project spending weeks to try to reduce the memory footprint by 1 MB. And it was hard :-) Biggest objects were in the application but it would help to be able to free Python internal caches. Victor

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20131007/7142dab8/attachment.html>

More information about the Python-Dev mailing list