[Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Wed Jun 19 05:32:01 CEST 2013

On 19 June 2013 09:23, Scott Dial <scott+python-dev at scottdial.com> wrote:

On 6/18/2013 4:40 PM, Victor Stinner wrote:

No context argument -------------------

Simplify the signature of allocator functions, remove the context argument: * void* malloc(sizet size) * void* realloc(void *ptr, sizet newsize) * void free(void *ptr) It is likely for an allocator hook to be reused for PyMemSetAllocator() and PyObjectSetAllocator(), or even PyMemSetRawAllocator(), but the hook must call a different function depending on the allocator. The context is a convenient way to reuse the same custom allocator or hook for different Python allocators. I think there is a lack of justification for the extra argument, and the extra argument is not free. The typical use-case for doing this continuation-passing style is when the set of contexts is either unknown, arbitrarily large, or infinite. In other words, when it would be either impossible or impractical to enumerate all of the contexts. However, in this case, we have only 3.

Note that the context is part of the BlockAllocator structure, NOT predefined by Python.

Your proposal already puts forward having 3 pairs of Get/Set functions, so there is no distinct advantage in having a single typedef instance that you pass in to all 3 of them. And, having all 3 pairs use the same typedef is a bit of an attractive nuisance, in that one could pass the wrong allocators to the wrong setter. With that, I could argue that there should be 3 typedefs to prevent coding errors.

I'm not sure we should be restricting this to the CPython internal domains indefinitely. If we use a domain based model from the start, then that will allow us in the future to let extension modules declare additional domains rather than having to employ library specific logic in either the CPython core or in embedding applications.

Nevertheless, the ctx argument buys the implementer nothing if they have to begin their alloc function with "if(ctx == X)". In other words, there is nothing simpler about:

""" void *alloc(void *ctx, sizet size) { if(ctx == PYALLOCPYMEM) return allocpymem(size); else if(ctx == PYALLOCPYMEMRAW) return allocpymemraw(size); else if(ctx == PYALLOCPYOBJECT) return allocpyobject(size); else abort(); } PyMemBlockAllocator pymemallocator = {.ctx=PYALLOCPYMEM, .alloc=&alloc, .free=&free}; PyMemBlockAllocator pymemrawallocator = {.ctx=PYALLOCPYMEMRAW, .alloc=&alloc, .free=&free}; PyMemBlockAllocator pyobjectallocator = {.ctx=PYALLOCPYOBJECT, .alloc=&alloc, .free=&free}; """

Why would anyone do that? The context is so embedding applications can distinguish the CPython runtime from their other domains that use the same allocator functions. If you wanted to use completely different allocators for each domain, you would just do that and ignore the context argument entirely.

Agreed more of that rationale needs to be moved from the issue tracker into the PEP, though.

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list