[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance (original) (raw)

Brett Cannon brett at python.org
Wed Mar 9 12:54:18 EST 2016

On Wed, 9 Mar 2016 at 06:57 Victor Stinner <victor.stinner at gmail.com> wrote:

2016-02-08 15:18 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>: >> Perhaps if you add some guards somewhere :-) > > We have runtime checks but only implemented in debug mode for efficiency. > > By the way, I proposed once to add an environment variable to allow to > enable these checks without having to recompile Python. Since the PEP > 445, it became easy to implement this. What do you think? > https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable

Ok, I wrote a patch to implement a new PYTHONMALLOC environment variable: http://bugs.python.org/issue26516 PYTHONMALLOC=debug installs debug hooks to: * detect API violations, ex: PyObjectFree() called on a buffer allocated by PyMemMalloc() * detect write before the start of the buffer (buffer underflow) * detect write after the end of the buffer (buffer overflow) https://docs.python.org/dev/c-api/memory.html#c.PyMemSetupDebugHooks The main advantage of this variable is that you don't have to recompile Python in debug mode to benefit of these checks.

I just wanted to say this all sounds awesome! Thanks for all the hard work on making our memory management story easier to work with, Victor.

-Brett

Recompiling Python in debug mode requires to recompile all extensions modules since the debug ABI is incompatible. When I played with tracemalloc on Python 2 ( http://pytracemalloc.readthedocs.org/ ), I had such issues, it was very annoying with non-trivial extension modules like PyQt or PyGTK. With PYTHONMALLOC, you don't have to recompile extension modules anymore!

With tracemalloc and PYTHONMALLOC=debug, we will have a complete tool suite to "debug memory"! My motivation for PYTHONMALLOC=debug is to detect API violations to prepare my change on PyMemMalloc() allocator ( http://bugs.python.org/issue26249 ), but also to help users to detect bugs. It's common that users report a bug: "Python crashed", but have no idea of the responsible of the crash. I hope that detection of buffer underflow & overflow will help them to detect bugs in their own extension modules. Moreover, I added PYTHONMALLOC=malloc to ease the use of external memory debugger on Python. By default, Python uses pymalloc allocator for PyObjectMalloc() which raises a lot of false positive in Valgrind. We even have a configuration (--with-valgrind) and a Valgrind suppressino file to be able to skip these false alarms in Valgrind. IMHO PYTHONMALLOC=malloc is a simpler option to use Valgrind (or other tools). Victor

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20160309/61eb5cdf/attachment.html>

More information about the Python-Dev mailing list