[Python-Dev] Enhancement of Python memory allocators (original) (raw)

Victor Stinner victor.stinner at gmail.com
Thu Jun 13 01:06:30 CEST 2013

Hi,

I would like to improve memory allocators of Python. My two use cases are replacing memory allocators with custom allocators in embedded system and hooking allocators to track usage of memory.

I wrote a patch for this, I'm going to commit it if nobody complains: http://bugs.python.org/issue3329

Using this patch, detecting memory corruptions (buffer underflow and overflow) can be done without recompilation. We may add an environment variable to enable Python debug functions at runtime, example: PYDEBUGMALLOC=1. There is just a restriction: the environment variable would not be ignored with -E command line option, because command line options are parsed after the first memory allocation. What do you think?

The patch adds the following functions:

void PyMem_GetAllocators( void *ctx_p, void (**malloc_p) (void ctx, size_t size), void (**realloc_p) (void *ctx, void *ptr, size_t size), void (**free_p) (void *ctx, void *ptr));

void PyMem_SetAllocators( void ctx, void (*malloc) (void ctx, size_t size), void (*realloc) (void *ctx, void *ptr, size_t size), void (*free) (void *ctx, void *ptr));

It adds 4 similar functions (get/set) for PyObject_Malloc() and allocators of pymalloc arenas.

For the "track usage of memory" use case, see the following project which hooks memory allocators using PyMem_SetAllocators() and PyObject_SetAllocators() to get allocated bytes per filename and line number. https://pypi.python.org/pypi/pytracemalloc

Another issue proposes to use VirtualAlloc() and VirtualFree() for pymalloc arenas, see: http://bugs.python.org/issue13483

I don't know if it would be interesting, but it would now possible to choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc, ...) at runtime, with an environment variable for example.

Victor

More information about the Python-Dev mailing list