[Python-Dev] Activating pymalloc (original) (raw)

Tim Peters tim.one@comcast.net
Fri, 15 Mar 2002 13:05:40 -0500

[Guido, on PyMem_XXX still calling malloc/free even when pymalloc is enabled]

Well, then you would have to provide yet another set of macros that don't require the GIL, for use by extensions that need to allocate memory while not holding the GIL. Of course, you could say, "then just use malloc/free", but the intention of pymem.h was that it would also be possible to provide a malloc/free substitute that was thread-safe -- unlike pymalloc. I don't know how realistic that is -- the only candidate would be GNU malloc but that would just replace the malloc/realloc/free entry points so wouldn't need any of the pymem.h-approved hackery.

That's an old sore point: the memory API is, in its many "hidden" layers, almost pure YAGNI. Vladimir found it useful when he was exploring the universe of all possible memory allocation algorithms, but I've seen no evidence that anyone else cares (or even understands it).

Rather than have PyMem_XXX not call malloc/free, I expect what I'd like more is a way for core code to say "use obmalloc.c here, even though I'm not allocating 'an object' at this point". This could be used for, e.g., list guts and especially for Unicode string guts. But despite all the layers of macros here, there doesn't appear to be a blessed way to spell that! The malloc substitute in obmalloc.c is

_THIS_MALLOC(size_t nbytes)

and _THIS_MALLOC is a macro exapnding to PyCore_OBJECT_MALLOC_FUNC, and PyCore_OBJECT_MALLOC_FUNC is a macro expanding to _PyCore_ObjectMalloc when pymalloc is enabled. _PyCore_ObjectMalloc is not a macro, it's the actual ultimate name of the malloc substitute (ignoring the possibilities for somone to break into the macro chain). All these macro names are supposed to be "invisible" (internal to the memory implementation).

In the other direction, e.g.,

PyObject_New -> _PyObject_New -> PyObject_MALLOC -> PyCore_OBJECT_MALLOC -> PyCore_OBJECT_MALLOC_FUNC, and we're back in the macro chain traced above

There's just no sane way to say "give me obmalloc.c's malloc". I'd rather we named obmalloc.c's entry points, e.g., PyMalloc_Malloc directly ...