(original) (raw)

Hi All:

I've created a simple program to make sure I wasn't lying to you all ;->

Here it is:

for (ii = 0; ii < 100; ii++)
{
Py\_Initialize();

if ((code = Py\_CompileString(p, "foo", Py\_file\_input)) \== NULL)
printf("PyRun\_SimpleString() failed\\n");
else
{
if (PyRun\_SimpleString(p) == -1)
printf("PyRun\_SimpleString() failed\\n");

Py\_CLEAR(code);
}

Py\_Finalize();
}

This sequence causes about 10k growth per iteration and after many cycles, there's no indication that any pooling logic is helping. Our "useful" example is slightly more complex, and therefore may explain why I was seeing about 16k per iteration.

Unless I've done something obviously wrong, I tend to believe Benjamin's claim that this issue is well known.

Suggestion: I have had great success with similar problems in the past by using a pools implementation sitting on top of what I call a "block memory allocator". The bottom (block) allocator grabs large blocks from the heap and then doles them out to the pools layer, which in turn doles them out to the requester. When client memory is freed -- it is NOT -- rather it's added to the pool which contains like-sized blocks -- call it an "organized free list". This is a very, very fast way to handle high allocation frequency patterns. Finally, during shutdown, the pool simply vaporizes and the block allocator returns a the (fewer) large blocks back to the heap. This avoids thrashing the heap, forcing it to coalesce inefficiently and also avoids heap fragmentation, which can cause unwanted growth as well...

Note that this would be a "hard-reset" of all allocated memory, and any global data in the text segment would also have to be cleared, but it would provide a fast, clean way to ensure that each invocation was 100% clean.