[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance (original) (raw)

Victor Stinner victor.stinner at gmail.com
Fri Mar 25 08:11:31 EDT 2016

So what do you think? Is it worth to change PyMem_Malloc() allocator to pymalloc for a small speedup?

Should we do something else before doing that?

Or do you expect that too many applications use PyMem_Malloc() without holding the GIL and will not run try to run their application with PYTHONMALLOC=debug?

Victor

2016-03-15 0:19 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:

2016-02-12 14:31 GMT+01:00 M.-A. Lemburg <mal at egenix.com>:

If your program has bugs, you can use a debug build of Python 3.5 to detect misusage of the API. Yes, but people don't necessarily do this, e.g. I have for a very long time ignored debug builds completely and when I started to try them, I found that some of the things I had been doing with e.g. free list implementations did not work in debug builds. I just added support for debug hooks on Python memory allocators on Python compiled in release mode. Set the environment variable PYTHONMALLOC to debug to try with Python 3.6. I added a check on PyObjectMalloc() debug hook to ensure that the function is called with the GIL held. I opened an issue to add a similar check on PyMemMalloc(): https://bugs.python.org/issue26563 Yes, but those are part of the stdlib. You'd need to check a few C extensions which are not tested as part of the stdlib, e.g. numpy, scipy, lxml, pillow, etc. (esp. ones which implement custom types in C since these will often need the memory management APIs). It may also be a good idea to check wrapper generators such as cython, swig, cffi, etc. I ran the test suite of numpy, lxml, Pillow and cryptography (used cffi). I found a bug in numpy. numpy calls PyMemMalloc() without holding the GIL: https://github.com/numpy/numpy/pull/7404 Except of this bug, all other tests pass with PyMemMalloc() using pymalloc and all debug checks. Victor

More information about the Python-Dev mailing list