[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance (original) (raw)

M.-A. Lemburg mal at egenix.com
Thu Feb 4 07:54:54 EST 2016

On 04.02.2016 13:29, Victor Stinner wrote:

Hi,

2016-02-04 11:17 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:

Do you see any drawback of using pymalloc for PyMemMalloc()?

Yes: You cannot free memory allocated using pymalloc with the standard C lib free(). That's not completly new. If Python is compiled in debug mode, you get a fatal error with a huge error message if you free the memory allocated by PyMemMalloc() using PyObjectFree() or PyMemRawFree(). But yes, technically it's possible to use free() when Python is not compiled in debug mode.

Debug mode is a completely different beast ;-)

It would be better to go through the list of PyMem*() calls in Python and replace them with PyObject*() calls, where possible. There are 536 calls to the functions PyMemMalloc(), PyMemRealloc() and PyMemFree(). I would prefer to modify a single place having to replace 536 calls :-/

You have a point there, but I don't think it'll work out that easily, since we are using such calls to e.g. pass dynamically allocated buffers to code in extensions (which then have to free the buffers again).

Does anyone recall the rationale to have two families to memory allocators?

The PyMem*() APIs were needed to have a cross-platform malloc() implementation which returns standard C lib free()able memory, but also behaves well when passing 0 as size. Yeah, PyMemMalloc() & PyMemFree() help to have a portable behaviour. But, why not PyObjectMalloc() & PObjectFree() were not used in the first place?

Good question. I guess developers simply thought of PyObject_Malloc() being for PyObjects, not arbitrary memory buffers, most likely because pymalloc was advertised as allocator for Python objects, not random chunks of memory.

Also: PyObject_*() APIs were first introduced with pymalloc, and no one really was interested in going through all the calls to PyMem_*() APIs and convert those to use the new pymalloc at the time.

All this happened between Python 1.5.2 and 2.0.

One of the reasons probably also was that pymalloc originally did not return memory back to the system malloc(). This was changed only some years ago.

An explanation can be that PyMemMalloc() can be called without the GIL held. But it wasn't true before Python 3.4, since PyMemMalloc() called (indirectly) PyObjectMalloc() when Python was compiled in debug mode, and PyObjectMalloc() requires the GIL to be held.

When I wrote the PEP 445, there was a discussion about the GIL. It was proposed to allow to call PyMemxxx() without the GIL: https://www.python.org/dev/peps/pep-0445/#gil-free-pymem-malloc This option was rejected.

AFAIR, the GIL was not really part of the consideration at the time. We used pymalloc for PyObject allocation, that's all.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Feb 04 2016)

Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/

More information about the Python-Dev mailing list