[Python-Dev] Moving forward on the object memory API (original) (raw)

Tim Peters tim.one@comcast.net
Sun, 31 Mar 2002 18:00:16 -0500

[Tim]

Is there a reason not to make these the simpler

#define PyMemMALLOC malloc

[Neil Schemenauer]

It adds a type cast in some cases. We should keep is simple were it doesn't matter.

Is it ever necesssary to add a type cast? For example, under ANSI C, free() is declared to take a void* argument, and C will automatically cast whatever is passed to it to void*. Adding our own (void*) cast is redundant. We can assume all extensions include Python.h, which in turn includes stdlib.h, so the proper system prototypes should always be in scope in extensions.

A difficulty with doing more than plain name substitution is that, e.g.,

void *p = (void*)&PyMem_FREE;

doesn't work right if PyMem_FREE doesn't expand to (just) a function name. But so far the docs are silent on whether such uses are supported.

PyMemFREE would call free() by PyMemFree would call PyMallocFree. PyMemFREE is relatively new. 1.5.2 only had PyMemDEL and that was the recommended way to free object memory. PyMemFree is new too but based on my survey, some people use it for freeing object memory.

Ya, and I don't know how to resolve this.

[on PyMem_{xxx, XXX}]

I guess we could deprecate the uppercase versions and make the lowercase ones macros.

The docs already deprecated the uppercase names. Whether the lowercase names can be macros depends on what they expand to (binary compatibility again).

As far as warnings go, I think the alternative spellings should be allowed for a long time to come. We can't just keep changing our tune and expect the extension writers dance. :-)

I don't expect they'll ever go away, which is why I've got little enthusiasm for arguing about "deprecation".

... I thought we decided that making PyMemMALLOC do anything but malloc() was hopeless? The only reason the PyMem* layer is there is that so adventurous people can replace the system malloc() with something like a conservative GC malloc. In that case the extensions would have to be recompiled.

Actually, PyMem_MALLOC probably should make the same guarantee as PyMem_Malloc about what happens if you pass it 0. And we can never be sure that "an important" platform won't sprout a bug that we need to patch over with a more-or-less trivial wrapper. As Guido just covered in another msg, PyMem_Malloc is also needed to ensure that an extension gets its memory from the same malloc the Python DLL is using on Windows. This can be important if it's possible for the corresponding free() to be done by the Python DLL. Or so I've been told.

I was just trying to cut down the number of functions and macros.

Which I appreciate! Guido will want to do that too, I'm just warning that after he thinks about it , he'll retract at least one of his +1s and swing to the opposite pole.

... Extensions can use PyObject{Malloc,Realloc,Free} but the type based allocators are preferred.

They presumably exist for extensions that want to use the same allocator as the object allocator uses. For example, someone who has tons of raw little C memory blobs that aren't objects, but that could benefit from exploiting the pymalloc malloc.

Extensions should not use PyObject{MALLOC,REALLOC,FREE}.

This would be consistent with what the docs say about PyMem_UPPERCASE. I think Guido likes to use lowercase/UPPERCASE to distinguish solely between whether a thing is implemented via a function or a macro, and I think that's where some of the historical confusion comes from. As a user, I don't care at all how a thing is implemented, and much prefer UPPERCASE/lowercase to distinguish whether binary compatibility is guaranteed. And that view appears to be shared by people who changed the memory API after Guido. So what lowercase/UPPERCASE is intended to convey has become another murky mess. That's another argument that's yet to begin.

I forget about PyObjectResize and PyObjectGCResize. They should be part of the preferred API.

Curiously, there is no PyObject_Resize: PyObject_GC_Resize is our only general object resize function.