[Python-Dev] RE: Program very slow to finish (original) (raw)

Tim Peters tim.one@home.com
Tue, 6 Nov 2001 17:30:57 -0500

[Neil Schemenauer]

The problem is with extension modules. We can make sure code in CVS always has the big lock held when calling PyMalloc. We can't be sure that extension modules are safe.

So can that ever be solved? Some part of the puzzle went unaddressed here.

The other, more serious problem is that many extension modules allocate memory with PyObjectNew() and free it with free() or PyMemDEL() instead of PyObjectDel() or PyObjectDEL().

Umm, OK, PyObject_DEL() ... is a macro ... which redirects to PyObject_FREE() ... which is a macro ... which redirects to PyCore_OBJECT_FREE() ... which is a macro ... which redirects to PyCore_OBJECT_FREE_FUNC() ... which is a macro ... which redirects to _PyCore_ObjectFree (WITH_PYMALLOC) or PyCore_FREE_FUNC (without). PyCore_FREE_FUNC is a macro ... which redirects to free(). And _PyCore_ObjectFree ... doesn't exist. I must have missed an #undef in there somewhere ... ah, OK, in the WITH_PYMALLOC case, PyCore_OBJECT_FREE_FUNC(== _PyCore_ObjectFree) appears in obmalloc.c's

#define _THIS_FREE PyCore_OBJECT_FREE_FUNC

so that _THIS_FREE is actually _PyCore_ObjectFree; then obmalloc.c's

void _THIS_FREE(void *p) {

expands to

void PyCore_OBJECT_FREE_FUNC(void *p) {

which expands again to

void _PyCore_ObjectFree(void *p) {

and we're done. Couldn't be simpler . Then PyMem_DEL ... oh, forget it. Now I remember why I gave up on this last time I looked at it -- it's Preprocessor Hell.

If pymalloc is enabled then memory allocated by a PyObject* function must be freed by a PyObject* funciton.

Damn, you're good.

mxDateTime is the first module I ran into that does this but there are many others I'm sure. I think almost all of the C modules I have written do it. The code in xxmodule.c used to do it as well.

no-solutions-only-more-problems-hey-i'm-sick-today-too-ly y'rs Neil

Big Hammer? Change every one of the existing guys to resolve to malloc() and free(). Then declare them all obsolete, define a much smaller new set of names, edit the core to use the new guys, and non-core modules get stuck with malloc/free until they're rewritten too. It sucks, but we'd be in better shape today if that had been done the last time this got reworked.