[Python-Dev] Activating pymalloc (original) (raw)

Tim Peters tim.one@comcast.net
Fri, 15 Mar 2002 04:52:59 -0500

[Tim]

Are "the right" alloc/dealloc macros documented anywhere? I can't find it, if they are.

[Martin]

http://www.python.org/doc/current/api/memoryInterface.html

is as precise and comprehensive as anything I would have written myself.

Then no wonder extension authors can't get this right <0.9 wink>.

The Examples even give explicit examples what cases to avoid.

Heh -- I never noticed the examples before. It's curious that they claim

PyMem_Del(buf3);  /* Wrong -- should be PyMem_Free() */

when the docs just finished saying

void PyMem_Del(void *p)
    Same as PyMem_Free().

Either the example or the doc's "same as" is incomprehensible.

It's also curious that the examples say

free(buf1);       /* Fatal -- should be PyMem_Del()  */

when the docs said no such thing (and this may be the #1 error in practice). The Examples page (but not the docs) says

Indeed, it is required to use the same memory API family for a given
memory block,

but the crucial phrase "the same memory API family" is left undefined. At a minimum, it needs to identify "the memory API families" by explicit, exhaustive enumeration. The mess in practice is a fact, and proves people don't understand what the docs are trying to say here. I'm not sure myself.

Like, are the functions and macros interchangeable? For example, if you know something was allocated via PyMem_Malloc, is it OK to release it via PyMem_FREE? I simply can't guess from what the docs say, again because what constitutes "a family" is left undefined.

For PyObject, the equivalent documentation is at

http://www.python.org/doc/current/api/allocating-objects.html

I'll be charitable and guess this defines a different family .

Note that users also have to know about

http://www.python.org/doc/current/api/supporting-cycle-detection.html

in order to get at

PyObject_GC_New / PyObject_GC_NewVar / PyObject_GC_Del

So far we've covered more than two dozen spellings (even without plain "malloc" and "free"), spread over at least 4 manual pages. One compact table listing all of them in "legal" pairs would be an enormous help.

... Do you think you would understand the documentation that is currently there?

Not well enough to use with confidence, no. I've seen the docs before, although I did miss the Examples section previously. I have a fancy editor with an integrated symbol database, and in practice I chase down the macros and functions until I see what they do at the leaves. The endless layers of indirection macros make this quite a pain (and I see you resorted to looking at post-preprocessor listings).

BTW, I found the documentation by grepping for PyMem in all .tex files, because I could not be bothered to read the online version, either.

They're easy to find via the index in the C API manual.

It leaves out a good many details (i.e. additional API), but I think this is deliberate - you are not supposed to ever use this other API, anyway.

Which API?

I wonder whether one could design a script that analyses Python code for asymmetric usage of memory management functions, e.g. trying to match

expression->FIELD = ALLOCATOR; DEALLOCATOR(otherexpression->FIELD); This would complain if there is a FIELD that has no symmetric usage, or if a certain allocator has no counterpart at all in a source file. I then wonder if that script would have found the errors you cite.

Well, to a first approximation, just searching for "free(" is valuable! In the Python source, the thread implementations use malloc and free on purpose, but it also turns up a curious

        free((char *)Py_FileSystemDefaultEncoding);

in _localmodule.c. If not an outright bug, that's at least begging for a comment, as the only obvious definition is in bltinmodule.c:

#if defined(MS_WIN32) && defined(HAVE_USABLE_WCHAR_T) const char *Py_FileSystemDefaultEncoding = "mbcs"; #else const char Py_FileSystemDefaultEncoding = NULL; / use default */ #endif

Does that mean that, on Windows, we may free() a static char*?!