[Python-Dev] Discussion related to memory leaks requested (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Thu Jan 14 23:19:25 EST 2016

On 15 January 2016 at 05:25, Matthew Paulson <paulson at busiq.com> wrote:

Hi All:

I've created a simple program to make sure I wasn't lying to you all ;-> Here it is: for (ii = 0; ii < 100; ii++) { PyInitialize(); if ((code = PyCompileString(p, "foo", Pyfileinput)) == NULL) printf("PyRunSimpleString() failed\n"); else { if (PyRunSimpleString(p) == -1) printf("PyRunSimpleString() failed\n"); PyCLEAR(code); } PyFinalize(); } This sequence causes about 10k growth per iteration and after many cycles, there's no indication that any pooling logic is helping. Our "useful" example is slightly more complex, and therefore may explain why I was seeing about 16k per iteration. Unless I've done something obviously wrong, I tend to believe Benjamin's claim that this issue is well known. Suggestion: I have had great success with similar problems in the past by using a pools implementation sitting on top of what I call a "block memory allocator". The bottom (block) allocator grabs large blocks from the heap and then doles them out to the pools layer, which in turn doles them out to the requester. When client memory is freed -- it is NOT -- rather it's added to the pool which contains like-sized blocks -- call it an "organized free list". This is a very, very fast way to handle high allocation frequency patterns. Finally, during shutdown, the pool simply vaporizes and the block allocator returns a the (fewer) large blocks back to the heap. This avoids thrashing the heap, forcing it to coalesce inefficiently and also avoids heap fragmentation, which can cause unwanted growth as well... Note that this would be a "hard-reset" of all allocated memory, and any global data in the text segment would also have to be cleared, but it would provide a fast, clean way to ensure that each invocation was 100% clean.

CPython does use an arena based allocator, but PyFinalize doesn't purge it (if it did, there'd be segfaults rather than memory growth when modules keep pointers across Initialize/Finalize cycles).

Building with PYMALLOC_DEBUG and setting PYTHONMALLOCSTATS in the environment will cause it to dump debugging info during Py_Finalize.

Building with Py_TRACE_REFS and setting PYTHONDUMPREFS also provides info on live Python objects during shutdown.

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20160115/0933176a/attachment.html>

More information about the Python-Dev mailing list