[Python-Dev] Activating pymalloc (original) (raw)

Guido van Rossum guido@python.org
Fri, 15 Mar 2002 11:01:31 -0500

[Guido] > Some extensions will break because they don't use the right > alloc/dealloc macros (we just fixed a few of these in Zope)

[Tim]

Are "the right" alloc/dealloc macros documented anywhere? I can't find it, if they are.

There's a long comment block at the start of objimpl.h documenting PyObject_New, PyObject_NewVar, PyObject_Del, and PyObject_Init. These are the only ones that 3rd party extensions should use for allocating storage for objects. In particular, a type's tp_dealloc function should use PyObject_Del when the type's constructor uses PyObject_New or a variant. It ends with:

Unless you have specific memory management requirements, it is
recommended to use PyObject_{New, NewVar, Del}. */

While that's pretty hidden, it's also pretty unambiguous on what "the right" thing is.

Unfortunately, core object types are allowed to use other macros as well, and many 3rd party extensions copy examples from the core rather than reading comments in obscure header files.

There's also the question of how to allocate non-object memory (like the hash table of a dict or the array of pointers for a list). According to a comment in pymem.h, for allocating non-object memory, you should always use the PyMem_* family of functions and macros. In addition, it is also quite clear on the need for extension modules to always use the functions, not the macros.

Much of a pymalloc fan as I am, I'm -1 on introducing it if we can't give extension authors a clear, brief, prominent and self-contained account of the memory rules they're supposed to follow (I don't really know what they are -- I always reverse-engineer it as needed, because I never found a place in the docs that offered sufficient guidance).

I think the problem is not that there are no clear unambiguous rules. It's that they have been undocumented and violated in code used as an example for too long.

We should at least try to do something in 2.3 that will help extension authors do the right thing in subsequent versions. Documenting the rules is one thing we should definitely do. Maybe we can also provide a configuration option that complains loudly when an extension is compiled that uses the wrong macros. Maybe that configuration option could even be on by default (it should only affect compiling extensions, not linking with them).

--Guido van Rossum (home page: http://www.python.org/~guido/)