[Python-Dev] head crashing (was: Fwd: [Python-checkins] buildbot warnings in x86 mvlgcc trunk) (original) (raw)

Tim Peters tim.peters at gmail.com
Tue May 1 23:29:00 CEST 2007

[Neal Norwitz]

In rev 54982 (the first time this crash was seen), I see something which might create a problem. In python/trunk/Modules/posixmodule.c (near line 6300):

+ PyMemFREE(mode); PyENDALLOWTHREADS

Shouldn't do that.

[Brett Cannon]

The PyMemMALLOC call that creates 'mode' is also called without explicitly holding the GIL.

Or that ;-)

Can you call PyMemFREE() without the GIL held? I couldn't find it documented either way.

I believe the GIL does not need to be held, but obviously Tim or someone with more memory experience should step in to say definitively.

The GIL should be held. The relevant docs are in the Python/C API manual, section "8.1 Thread State and the Global Interpreter Lock":

Therefore, the rule exists that only the thread that has acquired the global
interpreter lock may operate on Python objects or call Python/C

API functions.

PyMem_XYZ is certainly a "Python/C API function". There are functions you can call without holding the GIL, and section 8.1 intends to give an exhaustive list of those. These are functions that can't rely on the GIL, like PyEval_InitThreads() (which /creates/ the GIL), and various functions that create and destroy thread and interpreter state.

If you look at Include/pymem.h, PyMemFREE gets defined as PyObjectFREE in a debug build. PyObjectFree is defined at PyObjectDebugFree. That function checks that the memory has not been written with the debug bit pattern and then calls PyObjectFree. That call just sticks the memory back into pymalloc's memory pool which is implemented without using any Python objects.

But pymalloc's pools have a complex internal structure of their own, and cannot be mucked with safely by multiple threads simultaneously.

In other words no Python objects are used in pymalloc (to my knowledge) and thus is safe to use without the GIL.

Nope. For example, if two threads simultaneously try to free objects in the same obmalloc size class, there are a number of potential thread-race disasters in linking the objects into the same size-class chain.

In a release build this doesn't matter, since PyMem_XYZ map directly to the platform malloc/realloc/free, and so inherit the thread safety (or lack thereof) of the platform C implementations.

If it's necessary to do malloc/free kinds of things without holding the GIL, then the platform malloc/free must be called directly. Perhaps that's what posixmodule.c wants to do in this case.

More information about the Python-Dev mailing list