[Python-Dev] Py_Finalize does not release all memory, not even closely (original) (raw)

Tim Peters tim.peters at gmail.com
Tue Apr 11 20:47:36 CEST 2006

[Martin v. Löwis]

Several places in the documentation claim that PyFinalize will release all memory:

http://docs.python.org/api/embedding.html says that literally,

It's wrong ;-).

and

http://docs.python.org/api/initialization.html#l2h-778 suggests it is a bug when memory is not freed in PyFinalize. This has left people to believe that this is a bug: <https://sourceforge.net/tracker/index.php?func=detail&aid=1445210&groupid=5470&atid=105470>

Well, there may well be a bug (or multiple bugs) underlying that one too. It's one thing for Py_Finalize() not to release all memory (it doesn't and probably never will), but it's not necessarily the same thing if running Py_Initialize() ... Py_Finalize() repeatedly keeps leaking more and more memory.

However, I don't see any chance to make this promise even remotely. Objects allocated in extension modules, and held in global variables (e.g. socketmodule.c:socketerror, socketherror, socketgaierror, sockettimeout) will never be released, right?

Not unless the module has a finalization function called by Py_Finalize() that frees such things (like PyString_Fini and PyInt_Fini).

Other globals allocated via a

static PyObject *someglobal = NULL;

...

if (someglobal == NULL)
    someglobal = allocate_an_object_somehow();

pattern shouldn't contribute to continuing leaks across Py_Initialize() ... Py_Finalize() loops.

And because of the small objects allocator, their pool will remain allocated, right? And, then, the arena.

Before Python 2.5, arenas are never freed, period. In Python 2.5, an arena will be freed if and only if it contains no allocated object by the time Py_Finalize ends.

There may also be trash cycles that aren't collected during Py_Finalize because I had to comment out Py_Finalize's second call to PyGC_Collect(); new-style class objects are among the trash thingies leaked (although if Py_Initialize() is called again, it's possible that they'll get cleaned up by cyclic gc after all).

See Misc/SpecialBuilds.txt, section Py_TRACE_REFS, entry PYTHONDUMPREFS, for a way to get a dump of all heap objects Py_Finalize leaves alive. I doubt that's been run in years; Guido and I used it in 2.3b1 to cure some unreasonably high finalization leakage at the time.

So ISTM that invoking PyFinalize after importing socket will yield atleast 256KiB garbage. Of course, that's not real garbage, because the next PyInitialize'd interpreter will continue to allocate from the arenas.

I'm not clear on whether, e.g., init_socket() may get called more than once if socket-slinging code appears in a Py_Initialize() ... Py_Finalize(). If it doesn't, then, e.g., the unconditional

socket_gaierror = PyErr_NewException("socket.gaierror", ...);

won't contribute to ongoing leaks. But if it does, then we'll systematically leak the exception object on each loop trip.

But still, the actual objects that the modules hold on to will not be reclaimed until the process terminates.

Without a _Fini() function called by Py_Finalize, that's correct.

Please correct me if I'm wrong.

If you ever are, I will ;-)

More information about the Python-Dev mailing list