[Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4 (original) (raw)

Victor Stinner victor.stinner at gmail.com
Sat Jun 15 01:03:42 CEST 2013

I commited the new API (little bit different than my last patch on issue #3329): http://hg.python.org/cpython/rev/6661a8154eb3

The documentation will be available in a few minutes at: http://docs.python.org/3/c-api/memory.html

2013/6/14 Kristján Valur Jónsson <kristjan at ccpgames.com>:

Removing the GIL restriction would help to replace direct calls to malloc() with PyMemMalloc(). Using PyMemSetAllocators(), an application would be able to replace memory allocators, and these allocators would be used "everywhere". => see http://bugs.python.org/issue18203 To keep this interesting, I have a somewhat different opinion to Victor :) have put comments in the original defect, but would like to repeat them here. IMHO, keeping the GIL restriction on PyMemMALLOC is useful. 1) It allows it to be replaced with PyObjectMALLOC(). Or anything else. In particular, an implementer is free to add memory profiling support and other things without worrying about implementation details. Requiring it to be GIL free severely limits what it can do. For example, it would be forbidden to delegate to PyObjectMALLOC when debugging.

For my own pytracemalloc tool, holding the GIL while calling PyMem_Malloc() is required to be able to retrieve the Python filename and line number of the caller. So you convinced me :-) I am also worried by the backward compatibility, even if I expect that only a very few developers replaced Python memory allocators. A custom memory allocator may not be thread-safe, so the GIL can also be convinient.

I added new functions in the "final" API: PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(). These functions are just wrapper for malloc(), realloc() and free(). The GIL does not need to be hold. No process is done before/after at all. Behaviour of PyMem_RawMalloc(0) is platform depend for example. "size > PY_SSIZE_T_MAX" check is not done neither, but it may be interesting to add this check for security reasons (it is already in place for PyMem_Malloc and PyObject_Malloc).

Using these new functions instead of malloc/realloc/free is interesting because the internal functions can be replaced with PyMem_SetRawAllocators() and many checks are added in debug mode (ex: check for buffer under- and overflow).

PyObject_Malloc() was not documented, so I did not document PyObject_SetAllocators().

In the final API, I added a new PyMemAllocators structure to simplify the API. I also made _PyObject_SetArenaAllocators() private because I don't like its API (it is not homogenous with PyMem_SetAllocators) and it is concerned by less use cases. I prefer to wait a little before making this API public.

I didn't use "#ifndef Py_LIMITED_API", so all new functions are part of the stable API. Is it correct?

Victor

More information about the Python-Dev mailing list