[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance (original) (raw)

M.-A. Lemburg mal at egenix.com
Thu Feb 4 09:05:41 EST 2016

On 04.02.2016 14:25, Victor Stinner wrote:

Thanks for your feedback, you are asking good questions :-)

2016-02-04 13:54 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:

There are 536 calls to the functions PyMemMalloc(), PyMemRealloc() and PyMemFree().

I would prefer to modify a single place having to replace 536 calls :-/ You have a point there, but I don't think it'll work out that easily, since we are using such calls to e.g. pass dynamically allocated buffers to code in extensions (which then have to free the buffers again). Ah, interesting. But I'm not sure that we delegate the responsability of freeing the memory to external libraries. Usually, it's more the opposite: a library gives us an allocated memory block, and we have to free it. No?

Sometimes, yes, but we also do allocations for e.g. parsing values in Python argument tuples (e.g. using "es" or "et"):

https://docs.python.org/3.6/c-api/arg.html

We do document to use PyMem_Free() on those; not sure whether everyone does this though.

I checked if we call directly malloc() to pass the buffer to a library, but I failed to find such case.

Again, in debug mode, calling free() on a memory block allocated by PyMemMalloc() will likely crash. Since we run the Python test suite with a Python compiled in debug mode, we would already have detected such bug, no?

The Python test suite doesn't test Python C extensions, so it's not surprising that it passes :-)

See also my old issue http://bugs.python.org/issue18203 which replaced almost all direct calls to malloc() with PyMemMalloc() or PyMemRawMalloc().

Good question. I guess developers simply thought of PyObjectMalloc() being for PyObjects, Yeah, I also understood that, but in practice, it looks like PyMemMalloc() is slower than so using it makes the code less efficient than it can be. Instead of teaching developers that well, in fact, PyObjectMalloc() is unrelated to object programming, I think that it's simpler to modify PyMemMalloc() to reuse pymalloc ;-)

Perhaps if you add some guards somewhere :-)

Seriously, this may work if C extensions use the APIs consistently, but in order to tell, we'd need to check few. I know that I switched over all mx Extensions to use PyObject_*() instead of PyMem_*() or native malloc() several years ago and have not run into any issues.

I guess the main question then is whether pymalloc is good enough for general memory allocation needs; and the answer may well be "yes".

BTW: Tuning pymalloc for commonly used object sizes is another area where Python could gain better performance, i.e. reserve more / pre-allocate space for often used block sizes. pymalloc will also only work well for small blocks (up to 512 bytes). Everything else is routed to the system malloc().

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Feb 04 2016)

Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/

More information about the Python-Dev mailing list