[Python-Dev] Moving forward on the object memory API (original) (raw)

Tim Peters tim.one@comcast.net
Sun, 31 Mar 2002 15:41:50 -0500

[Neil Schemenauer]

It looks like Tim managed to work some magic and make pymalloc's free handle memory allocated by either the system allocator or pymalloc itself.

Yes, I believe that's so now. We'll know better after we try it <wink -- I did some light contrived testing of mixed uses>. One more: provided that an address was obtained via the system allocator (malloc or realloc), it should be safe to free it via the pymalloc free even when the GIL is not held.

That means we can make PyMemDEL use the pymalloc's 'free' and make PyObjectNEW use pymalloc's 'malloc'.

It means we have a lot more freedom in mixing and matching now. Unfortunately, this comes down to spelling out what "the rules" actually are, and there's been no progress on that. The only feedback I've gotten on the two exhaustive tables I posted earlier was along the lines of "geez, this really is a mess -- I'm not sure what to do".

Here's my plan for making this happen:

#define PyMemMALLOC(n) malloc(n) #define PyMemREALLOC(p, n) realloc((void *)(p), (n)) #define PyMemFREE(p) free((void *)(p))

Is there a reason not to make these the simpler

#define PyMem_MALLOC malloc

etc?

I think making PyMemFREE call free() is safe. I haven't seen any extension modules allocate with PyObjectNEW and free with PyMemFREE.

What about PyMem_Free()? You reported that MySQL-python-0.3.5 and crng-1.1 mix PyObject_NEW with PyMem_Free.

Note that Guido recently reported:

The memory thus allocated is returned by PyOS_Readline() (which calls
either function through a hook pointer), and both its callers (the
tokenizer and raw_input()) free the result using PyMem_DEL or
PyMem_FREE (these two seem to be used synonymically).

If that's so, PyMem_DEL and PyMem_FREE should be the same (but that grates on me too, FWIW).

We deprecate the PyMem* functions.

This may hit two snags: the docs currently say (and have for some time) that all of

PyMem_{MALLOC, REALLOC, FREE, NEW, RESIZE, DEL}

are "deprecated in extension modules". The docs encourage using the PyMem function spellings instead (or perhaps the docs insist on that -- it depends on what the docs mean by "deprecated"; IMO deprecation is without consequence in any context where we can't deliver warning msgs for at least a release cycle -- else we'll never actually get rid of "deprecated" stuff). The rationale has to do with cross-release binary compatibility: if an extension spells a thing PyMem_Malloc, they have "a right" to expect that their extension won't need to be recompiled across releases just because Python changes what PyMem_MALLOC does.

There's no need for them, IMHO:

#define PyMemMalloc PyMemMALLOC #define PyMemNew PyMemNEW #define PyMemResize PyMemRESIZE #define PyMemFree PyMemDEL #define PyMemDel PyMemDEL /* See comment near MALLOCZERORETURNSNULL in pyport.h. */ _#define PyMemRealloc(p, n) _ _do { _ _sizet n = n; _ _PyMemREALLOC(p, n ? n : 1); _ } while (0)

Note that PyMem_Realloc needs to return a result (else, e.g., the user can't know whether it failed). In any case, I predict Guido will say that the function spellings must indeed be functions, and are still the recommended way, and that the macro spellings are still deprecated for use in extensions (but not in the core), and that "deprecated" in this context will continue to mean nothing stronger than "should, but there's no actual consequence if you don't, apart from opening yourself up to possible x-release binary incompatibility".

Next, we base PyObject{MALLOC,REALLOC,FREE} on pymalloc.

No problem there: if we don't, I wasted my last few days.

Basically:

#ifdef WITHPYMALLOC #define PyObjectMALLOC(n) PyMallocMalloc(n) #define PyObjectREALLOC(op, n) PyMallocRealloc((void *)(op), (n)) #define PyObjectFREE(op) PyMallocFree((void *)(op)) #else #define PyObjectMALLOC(n) PyMemMALLOC(n) #define PyObjectREALLOC(op, n) PyMemREALLOC((void *)(op), (n)) #define PyObjectFREE(op) PyMemFREE((void *)(op)) #endif

Again I wonder why these aren't simply name substitutions ("#define PyObject_MALLOC _PyMalloc_Malloc", etc).

PyMemDEL and PyMemFree need to call pymalloc:

#define PyMemDEL(op) PyObjectFREE(op)

At least those two, yes (see above for questions about whether PyMem_FREE also needs to play along; but I'd sure like some way for an extension author to say "give me the same damn platform free() Python uses, without any damn wrappers").

We go through the source and replace all the PyMem* function calls with the equivalent PyMem* macro calls and replace PyMemDEL with PyMemFREE.

As above, I expect Guido will resist at least part of this part.

The recommended API for extension modules writers would be PyMem{MALLOC,REALLOC,NEW,RESIZE,FREE},

Ditto, but more so, since it's a 180 degree reversal of what the docs have (at least) recommended.

PyObject{New,NewVar,Del}, and PyObjectGC{New,NewVar,Del}.

This part should be non-controversial so far as it goes. But the docs don't currently give any cautions about using the macro versions of PyObject_XXX, and in addition to those we're also missing PyObject_{Malloc, MALLOC, Realloc, REALLOC, Free, FREE} and PyObject_GC_Resize in this account. Straightening all that out again requires agreeing on what the rules really are. I think ultimately has to come from Guido, but the best way to get that to happen is to provoke him with schemes he doesn't like .