peps: c6199202522a (original) (raw)

Mercurial > peps

changeset 4959:c6199202522a

PEP 445: add "Redesign Debug Checks on Memory Allocators as Hooks" section

line wrap: on

line diff

--- a/pep-0445.txt +++ b/pep-0445.txt @@ -40,8 +40,8 @@ Use cases: Proposal ======== -New functions and new structure -------------------------------- +New Functions and Structures +----------------------------

• Add a new GIL-free (no need to hold the GIL) memory allocator: @@ -109,11 +109,12 @@ New functions and new structure
  • void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
  • void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator) -* Add a new function to setup the builtin Python debug hooks when a

• memory allocator is replaced: +* Add a new function to setup the debug checks on memory allocators when

• a memory allocator is replaced:
  • void PyMem_SetupDebugHooks(void)

• • the function does nothing is Python is not compiled in debug mode

• • Install the debug hook on all memory block allocators.
• • The function does nothing is Python is not compiled in debug mode

• Memory allocators always returns NULL if size is greater than PY_SSIZE_T_MAX. The check is done before calling the @@ -133,8 +134,12 @@ Default allocators:
• pymalloc arena allocator: mmap(), munmap() (and ctx is NULL), or malloc() and free() if mmap() is not available -The builtin Python debug hooks were introduced in Python 2.3 and -implement the following checks: + +Redesign Debug Checks on Memory Allocators as Hooks +---------------------------------------------------- + +Since Python 2.3, Python implements different checks on memory +allocators in debug mode:
• Newly allocated memory is filled with the byte 0xCB, freed memory is filled with the byte 0xDB. @@ -143,14 +148,36 @@ implement the following checks:
• Detect write before the start of the buffer (buffer underflow)
• Detect write after the end of the buffer (buffer overflow) +In Python 3.3, the checks are installed by replacing +PYMEM_DOMAIN_MEM and PYMEM_DOMAIN_OBJ allocators, the previous +allocator is no more called. The new allocator is the same for both +domains: PyMem_Malloc() and PyMem_Realloc() call indirectly +PyObject_Malloc() and PyObject_Realloc(). + +This PEP redesigns the debug checks as hooks on the existing allocators +in debug mode. Examples of call traces without the hooks: + +* PyMem_Malloc() => _PyMem_RawMalloc() => malloc() +* PyObject_Free() => _PyObject_Free() + +Call traces when the hooks are installed (debug mode): + +* PyMem_Malloc() => _PyMem_DebugMalloc() =>

• _PyMem_RawMalloc() => malloc() +* PyObject_Free() => _PyMem_DebugFree() => _PyObject_Free() + +As a result, PyMem_Malloc() and PyMem_Realloc() now always call +malloc() and realloc(), instead of calling PyObject_Malloc() +and PyObject_Realloc() in debug mode. + +When at least one memory allocator is replaced with +PyMem_SetAllocator(), the PyMem_SetupDebugHooks() function must +be called to install the debug hooks on top on the new allocator. + Don't call malloc() directly anymore ------------------------------------ -PyMem_Malloc() and PyMem_Realloc() always call malloc() and -realloc(), instead of calling PyObject_Malloc() and -PyObject_Realloc() in debug mode. - PyObject_Malloc() falls back on PyMem_Malloc() instead of malloc() if size is greater or equal than 512 bytes, and PyObject_Realloc() falls back on PyMem_Realloc() instead of