(original) (raw)
On Wed, 9 Mar 2016 at 06:57 Victor Stinner <victor.stinner@gmail.com> wrote:
2016-02-08 15:18 GMT+01:00 Victor Stinner <victor.stinner@gmail.com>:
\>> Perhaps if you add some guards somewhere :-)
\>
\> We have runtime checks but only implemented in debug mode for efficiency.
\>
\> By the way, I proposed once to add an environment variable to allow to
\> enable these checks without having to recompile Python. Since the PEP
\> 445, it became easy to implement this. What do you think?
\> https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable
Ok, I wrote a patch to implement a new PYTHONMALLOC environment variable:
http://bugs.python.org/issue26516
PYTHONMALLOC=debug installs debug hooks to:
\* detect API violations, ex: PyObject\_Free() called on a buffer
allocated by PyMem\_Malloc()
\* detect write before the start of the buffer (buffer underflow)
\* detect write after the end of the buffer (buffer overflow)
https://docs.python.org/dev/c-api/memory.html#c.PyMem\_SetupDebugHooks
The main advantage of this variable is that you don't have to
recompile Python in debug mode to benefit of these checks.
I just wanted to say this all sounds awesome! Thanks for all the hard work on making our memory management story easier to work with, Victor.
-Brett
Recompiling Python in debug mode requires to recompile \*all\*
extensions modules since the debug ABI is incompatible. When I played
with tracemalloc on Python 2 ( http://pytracemalloc.readthedocs.org/
), I had such issues, it was very annoying with non-trivial extension
modules like PyQt or PyGTK. With PYTHONMALLOC, you don't have to
recompile extension modules anymore!
With tracemalloc and PYTHONMALLOC=debug, we will have a complete tool
suite to "debug memory"!
My motivation for PYTHONMALLOC=debug is to detect API violations to
prepare my change on PyMem\_Malloc() allocator (
http://bugs.python.org/issue26249 ), but also to help users to detect
bugs.
It's common that users report a bug: "Python crashed", but have no
idea of the responsible of the crash. I hope that detection of buffer
underflow & overflow will help them to detect bugs in their own
extension modules.
Moreover, I added PYTHONMALLOC=malloc to ease the use of external
memory debugger on Python. By default, Python uses pymalloc allocator
for PyObject\_Malloc() which raises a lot of false positive in
Valgrind. We even have a configuration (--with-valgrind) and a
Valgrind suppressino file to be able to skip these false alarms in
Valgrind. IMHO PYTHONMALLOC=malloc is a simpler option to use Valgrind
(or other tools).
Victor
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org