[Python-Dev] Activating pymalloc (original) (raw)

Tim Peters tim.one@comcast.net
Sun, 24 Mar 2002 18:38:19 -0500

Suppose we were to take the idea of "enforcing the rules" seriously. That means we'd first have to know what the rules are. I count 40 primary (just "get", "resize", and "free") memory APIs now:

malloc, realloc, free PyMem_{Malloc, Realloc, Free} PyMem_{MALLOC, REALLOC, FREE} PyMem_{New, Resize, Del} PyMem_{NEW, RESIZE, DEL} PyObject_{Malloc, Realloc, Free} PyObject_{MALLOC, REALLOC, FREE} PyObject_{New, NewVar, Del} PyObject_{NEW, NEWVAR, DEL} PyObject_GC_{New, NewVar, Resize, Del} PyMalloc_{New, NewVar, Del} PyMalloc{Malloc, Realloc, Free} PyMalloc{MALLOC, REALLOC, FREE}

The last 6 were recently added, and made part of the private API to avoid complicating the public API even more.

I don't understand what "API family" means, and none of the prose I've seen made the notion clearer. The only way it can mean something intelligible to me is via a table like the following, spelling out exactly which "get memory" spellings are legal with which "release memory" spellings (counting realloc/resize as a form of release). Catering to Martin's belief that functions and macros are necessarily in distinct families, yet catering also to the real-life examples Neil found, this is the most restrictive set of families I think we have a chance of enforcing without massive breakage:

Memory released by Must have been originally allocated by one of these one of these

realloc, malloc free

PyObject_GC_Del PyObject_GC_New, PyObject_GC_NewVar

PyObject_GC_Resize PyObject_GC_NewVar

PyMalloc_Del PyMalloc_New, PyMalloc_NewVar

_PyMalloc_Realloc, _PyMalloc_Malloc _PyMalloc_Free

_PyMalloc_REALLOC, _PyMalloc_MALLOC _PyMalloc_FREE

PyObject_Del PyObject_New, PyObject_NewVar

PyObject_DEL PyObject_NEW, PyObject_NEWVAR

PyObject_Realloc, PyObject_Malloc PyObject_Free

PyObject_REALLOC, PyObject_MALLOC PyObject_FREE

PyMem_Del, PyMem_New PyMem_Resize

PyMem_RESIZE PyMem_NEW

PyMem_Realloc PyMem_Malloc

PyMem_FREE, PyMem_MALLOC PyMem_REALLOC

PyMem_DEL, PyMem_NEW, PyMem_Free PyObject_NEW

The last block caters to the frequenct mixes of PyObject_NEW with PyMem_DEL and PyMem_Free Neil found in real life.

I always found the distinction between "free" and "del" spellings to be pointlessly pedantic (there's no difference under the covers, and never was -- and backward compatibility ensures there never will be).

The distincion between Mem and Object has also become pointlessly pedantic.

The distinction between macro and function spellings also doesn't make much sense to me, despite Martin's defense (you shouldn't use a macro spelling in an extension simply because you're in an extension, and macros can change across releases; it's not because we're reserving the right for the function and macro spellings to do incompatible things in a single release).

The distinctions between raw and typed memory, and between var and non-var objects, still make good sense.

Putting all those together could cut the # of groupings in half:

Memory released by Must have been originally allocated by one of these one of these

realloc, malloc free

PyObject_GC_Del PyObject_GC_New, PyObject_GC_NewVar

PyObject_GC_Resize PyObject_GC_NewVar

PyMalloc_Del PyMalloc_New, PyMalloc_NewVar

PyMem_Del, PyMem_New, PyMem_DEL, PyMem_NEW, PyMem_Free, PyMem_Malloc, PyMem_FREE, PyMem_MALLOC, PyObject_Del, PyObject_New, PyObject_DEL, PyObject_NEW, PyObject_Free PyObject_Malloc, PyObject_FREE PyObject_MALLOC PyObject_NewVar, PyObject_NEWVAR

PyMem_Realloc, PyMem_Malloc, PyMem_REALLOC, PyMem_MALLOC, PyObject_Realloc, PyObject_Malloc, PyObject_REALLOC PyObject_MALLOC

PyMem_Resize, PyMem_New, PyMem_RESIZE PyMem_NEW

_PyMalloc_Realloc, _PyMalloc_Malloc _PyMalloc_Free, _PyMalloc_MALLOC _PyMalloc_REALLOC, _PyMalloc_FREE

I don't yet know whether we can actually enforce anything here, but we have to explain the rules regardless. What kind of table would you make up? I expect the last table above matches what most extension authors think, although some probably lump free/malloc into the block with the other 8 historical ways to spell free.