[Python-Dev] Enhancement of Python memory allocators (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Thu Jun 13 01:21:42 CEST 2013

On 13 Jun 2013 09:09, "Victor Stinner" <victor.stinner at gmail.com> wrote:

Hi, I would like to improve memory allocators of Python. My two use cases are replacing memory allocators with custom allocators in embedded system and hooking allocators to track usage of memory. I wrote a patch for this, I'm going to commit it if nobody complains: http://bugs.python.org/issue3329 Using this patch, detecting memory corruptions (buffer underflow and overflow) can be done without recompilation. We may add an environment variable to enable Python debug functions at runtime, example: PYDEBUGMALLOC=1. There is just a restriction: the environment variable would not be ignored with -E command line option, because command line options are parsed after the first memory allocation. What do you think?

The rest of it sounds fine, but please don't add the runtime switching support to our existing main function. Interpreter startup is a mess already. If you were interested in helping directly with PEP 432, though, that would be good - I haven't been able to spend much time on it lately.

Cheers, Nick.

***** The patch adds the following functions: void PyMemGetAllocators( void **ctxp, void* (**mallocp) (void *ctx, sizet size), void* (**reallocp) (void *ctx, void *ptr, sizet size), void (**freep) (void *ctx, void *ptr)); void PyMemSetAllocators( void *ctx, void* (*malloc) (void *ctx, sizet size), void* (*realloc) (void *ctx, void *ptr, sizet size), void (*free) (void *ctx, void *ptr)); It adds 4 similar functions (get/set) for PyObjectMalloc() and allocators of pymalloc arenas. ***** For the "track usage of memory" use case, see the following project which hooks memory allocators using PyMemSetAllocators() and PyObjectSetAllocators() to get allocated bytes per filename and line number. https://pypi.python.org/pypi/pytracemalloc ***** Another issue proposes to use VirtualAlloc() and VirtualFree() for pymalloc arenas, see: http://bugs.python.org/issue13483 I don't know if it would be interesting, but it would now possible to choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc, ...) at runtime, with an environment variable for example. Victor

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130613/76c7f867/attachment.html>

More information about the Python-Dev mailing list