[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators (original) (raw)

Victor Stinner victor.stinner at gmail.com
Sat Jun 15 03:54:50 CEST 2013

2013/6/15 Antoine Pitrou <solipsis at pitrou.net>:

http://hg.python.org/cpython/rev/6661a8154eb3 ... Issue #3329: Add new APIs to customize memory allocators

* Add a new PyMemAllocators structure * New functions: - PyMemRawMalloc(), PyMemRawRealloc(), PyMemRawFree(): GIL-free memory allocator functions - PyMemGetRawAllocators(), PyMemSetRawAllocators() - PyMemGetAllocators(), PyMemSetAllocators() - PyMemSetupDebugHooks() - PyObjectGetArenaAllocators(), PyObjectSetArenaAllocators() My two cents, but I would prefer if this whole changeset was reverted. I think it adds too much complexity in the memory allocation APIs, for a pretty specialized benefit. IMHO, we should be able to get by with less allocation APIs (why the new Raw APIs) and less hook-setting functions.

Ok, I reverted my commit.

I posted my initial patch 3 months ago on the bug tracker. I got some reviews and discussed with Kristján Valur Jónsson who heavily modified Python for his game at CCP. I started two threads on python-dev this week (ok, only two days ago). I thaugh that the last known issues were fixed with the addition of PyMem_SetupDebugHooks() (to avoid an environment variable, as asked by Nick) and PyMem_RawMalloc() (have a GIL-free allocator).

I will work on a PEP to explain all these new functions and their use cases.

**

The addition of PyMem_RawMalloc() is motivated by the issue #18203 (Replace calls to malloc() with PyMem_Malloc()). The goal is to be able to setup a custom allocator for all allocation made by Python, so malloc() should not be called directly. PyMem_RawMalloc() is required in places where the GIL is not held (ex: in os.getcwd() on Windows).

PyMem_Malloc() is misused (called without the GIL held) in different places. Examples: the readline modules and functions called at Python startup, including main(). Replacing PyMem_Malloc() with malloc() would not allow to use the custom allocator everywhere, so PyMem_RawMalloc() is also required here.

The last point is an extension of the issue #18203: some external libraries like zlib or OpenSSL are also calling malloc() directly. But Python can configure these libraries to use a custom memory allocator. I plan to configure external libraries to use PyMem_GetRawAllocators() if PyMem_SetRawAllocators() was called (if PyMem_RawMalloc is not simply malloc) and if setting a custom allocator only affect a function and not the whole library.

Victor

More information about the Python-Dev mailing list