[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance (original) (raw)

Victor Stinner victor.stinner at gmail.com
Thu Feb 4 07:29:42 EST 2016

Hi,

2016-02-04 11:17 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:

Do you see any drawback of using pymalloc for PyMemMalloc()? Yes: You cannot free memory allocated using pymalloc with the standard C lib free().

That's not completly new.

If Python is compiled in debug mode, you get a fatal error with a huge error message if you free the memory allocated by PyMem_Malloc() using PyObject_Free() or PyMem_RawFree().

But yes, technically it's possible to use free() when Python is not compiled in debug mode.

It would be better to go through the list of PyMem*() calls in Python and replace them with PyObject*() calls, where possible.

There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc() and PyMem_Free().

I would prefer to modify a single place having to replace 536 calls :-/

Does anyone recall the rationale to have two families to memory allocators? The PyMem*() APIs were needed to have a cross-platform malloc() implementation which returns standard C lib free()able memory, but also behaves well when passing 0 as size.

Yeah, PyMem_Malloc() & PyMem_Free() help to have a portable behaviour. But, why not PyObject_Malloc() & PObject_Free() were not used in the first place?

An explanation can be that PyMem_Malloc() can be called without the GIL held. But it wasn't true before Python 3.4, since PyMem_Malloc() called (indirectly) PyObject_Malloc() when Python was compiled in debug mode, and PyObject_Malloc() requires the GIL to be held.

When I wrote the PEP 445, there was a discussion about the GIL. It was proposed to allow to call PyMem_xxx() without the GIL: https://www.python.org/dev/peps/pep-0445/#gil-free-pymem-malloc

This option was rejected.

Victor

More information about the Python-Dev mailing list