[Python-Dev] Suggested memory API rules for 2.3 (original) (raw)

Tim Peters tim.one@comcast.net
Wed, 03 Apr 2002 17:59:46 -0500

"Programs" does not include the Python core. We can do anything we need to do in the core, based on our complete knowledge of each release's internal implementation details.

[Guido]

However, since many people use the core as an example, we may have to be careful here. Certainly xxmodule.c, xxobject.c and xxsubtype.c should do the recommended thing;

I'll make sure that they do.

but if too many of the built-in objects use the macros despite those being in danger of deprecation, that might still perpetuate use of those macros.

It's hard to imagine that the macro names will often be cheaper than the function names, given the new requirement that they have identical I/O behavior. So, yes, I expect a lot of mindless source edits to replace core use of macro names with function names (and I'm good at massive mindless source edits ). There may be still be cases where the core wants to exploit that, e.g., it knows a malloc argument is > 0, and so skip the _PyMem_EXTRA business; if so, I expect we'll grow an internal API spelling for that.

After memory has been obtained via one of these functions, it should be resized and freed only by a function from the same line, except that PyMemFree may be used freely in place of PyMemDel, and PyObjectFree in place of PyObjectDel.

I actually had a slightly different rule in mind:

I did too at first.

mixing and matching between two adjacent lines that only differ by case is also permitted. Except of course that the macro lines will be deprecated, so maybe it doesn't matter to spell this out.

Exactly so. Since programs should not use macro names at all anymore, text devoted to drawing fine distinctions among macro name use is pretty much a waste. Note there is no "shall" or "shall not" text forbidding mixing corresponding macro and function names: the text above is only "should be ... from the same line". The later blurbs of the form

Memory obtained by PyMem_{Malloc, MALLOC} shall be resized only
by PyMem_{Realloc, REALLOC}.

are not intended not to be read as forbidding mixing PyMem_Malloc with PyMem_REALLOC. Perhaps that would be clearer said

Memory obtained by either of PyMem_{Malloc, MALLOC} shall be
resized only by either of PyMem_{Realloc, REALLOC}.

The exhaustive tables I posted earlier are much clearer for stuff like this, but, again, I'm decreasingly keen to spell out elaborate details for spellings that are discouraged.

[on whether private-API spellings for raw malloc etc should be introduced]

IMO, PyMemMalloc() and PyMemFree() should be good enough.

I was expecting this would be an objective matter resolved by exhaustive study of the code base when renaming the macro uses. I'm happy to settle it arbitrarily, though .

Relationship to Platform Allocator

All names in lines A, B, C and D ultimately invoke the platform C malloc, realloc, or free. However, programs shall not mix any of these names with direct calls to the platform malloc, calloc, realloc, or free referencing the same base memory addresses, as Python may need to perform bookkeeping of its own around its calls to the platform allocator.

Really? Why not just say these are wrappers around malloc and free?

I'm specifically trying to leave the door open for the PyMalloc_DebugXYZ routines to capture all uses of memory API functions in PYMALLOC_DEBUG mode. It also reads better as-is than to say, well, ya, OK, I suppose you can mix raw calls to libc with these particular memory API families, but the PyObject_GC_xyz family is entirely different in this respect. I don't want to draw distinctions just because the current implementation makes it possible to draw them; users shall consider malloc/realloc/free as a distinct API.

(On platforms where it matters, they will be guaranteed to use the same heap as the core uses -- this is apparently an issue with Windows DLLs.)

But that's another reason (if it's real ) to say "no mixing" here: if this is a use they care about, then, e.g., they're dead meat if they mix PyMem_Malloc() with raw free() (the former would use the malloc in the Python DLL, the latter the free in the user's DLL).