peps: a17ebebe52ca (original) (raw)

--- a/pep-0445.txt +++ b/pep-0445.txt @@ -40,18 +40,20 @@ Use cases: Proposal ======== -API changes ------------ +New functions and new structure +------------------------------- -* Add new GIL-free (no need to hold the GIL) memory allocator functions: +* Add a new GIL-free (no need to hold the GIL) memory allocator:

void* PyMem_RawMalloc(size_t size)
void* PyMem_RawRealloc(void *ptr, size_t new_size)
void PyMem_RawFree(void *ptr)
- the behaviour of requesting zero bytes is not defined: return NULL
or a distinct non-NULL pointer depending on the platform.

- The newly allocated memory will not have been initialized in any
way.
- Requesting zero bytes returns a distinct non-NULL pointer if
possible, as if PyMem_Malloc(1) had been called instead.

-* Add a new PyMemBlockAllocator structure:: +* Add a new PyMemAllocator structure:: typedef struct { /* user context passed as the first argument @@ -66,69 +68,70 @@ API changes /* release a memory block */ void (*free) (void *ctx, void *ptr);

} PyMemBlockAllocator;

} PyMemAllocator;

-* Add new functions to get and set internal functions of

PyMem_RawMalloc(), PyMem_RawRealloc() and PyMem_RawFree(): +* Add a new PyMemAllocatorDomain enum to choose the Python

allocator domain. Domains:

- void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)
- void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)
- default allocator: malloc(), realloc(), free()

- PYALLOC_PYMEM_RAW: PyMem_RawMalloc(), PyMem_RawRealloc()
and PyMem_RawRealloc()

-* Add new functions to get and set internal functions of

PyMem_Malloc(), PyMem_Realloc() and PyMem_Free():

- PYALLOC_PYMEM: PyMem_Malloc(), PyMem_Realloc() and
PyMem_Realloc()

- void PyMem_GetAllocator(PyMemBlockAllocator *allocator)
- void PyMem_SetAllocator(PyMemBlockAllocator *allocator)
- malloc(ctx, 0) and realloc(ctx, ptr, 0) must not return
NULL: it would be treated as an error.
- default allocator: malloc(), realloc(), free();
PyMem_Malloc(0) calls malloc(1)
and PyMem_Realloc(NULL, 0) calls realloc(NULL, 1)

- PYALLOC_PYOBJECT: PyObject_Malloc(), PyObject_Realloc()
and PyObject_Realloc()

+ +* Add new functions to get and set memory allocators: -* Add new functions to get and set internal functions of

PyObject_Malloc(), PyObject_Realloc() and
PyObject_Free():

- void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
- void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
- The new allocator must return a distinct non-NULL pointer when
requesting zero bytes

- void PyObject_GetAllocator(PyMemBlockAllocator *allocator)
- void PyObject_SetAllocator(PyMemBlockAllocator *allocator)
- malloc(ctx, 0) and realloc(ctx, ptr, 0) must not return
NULL: it would be treated as an error.
- default allocator: the pymalloc allocator - -* Add a new PyMemMappingAllocator structure:: +* Add a new PyObjectArenaAllocator structure:: typedef struct { /* user context passed as the first argument to the 2 functions */ void *ctx;

   /* allocate a memory mapping */[](#l1.87)

   /* allocate an arena */[](#l1.88)
   void* (*alloc) (void *ctx, size_t size);[](#l1.89)

   /* release a memory mapping */[](#l1.91)

   /* release an arena */[](#l1.92)
   void (*free) (void *ctx, void *ptr, size_t size);[](#l1.93)

} PyMemMappingAllocator;

- -* Add a new function to get and set the memory mapping allocator:

} PyObjectArenaAllocator;

- void PyMem_GetMappingAllocator(PyMemMappingAllocator *allocator)
- void PyMem_SetMappingAllocator(PyMemMappingAllocator *allocator)
- Currently, this allocator is only used internally by pymalloc to
allocate arenas.

+* Add new functions to get and set the arena allocator used by

pymalloc: -* Add a new function to setup the builtin Python debug hooks when memory

allocators are replaced:

- void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
- void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator) + +* Add a new function to setup the builtin Python debug hooks when a
memory allocator is replaced:
- void PyMem_SetupDebugHooks(void)

- the function does nothing is Python is compiled not compiled in
debug mode

- the function does nothing is Python is not compiled in debug mode + +* Memory allocators always returns NULL if size is greater than
PY_SSIZE_T_MAX. The check is done before calling the
inner function. + +The pymalloc allocator is optimized for objects smaller than 512 bytes +with a short lifetime. It uses memory mappings with a fixed size of 256 +KB called "arenas". -* The following memory allocators always returns NULL if size is

greater than PY_SSIZE_T_MAX (check before calling the internal
function): PyMem_RawMalloc(), PyMem_RawRealloc(),
PyMem_Malloc(), PyMem_Realloc(), PyObject_Malloc(),
PyObject_Realloc(). +Default allocators: + +* PYALLOC_PYMEM_RAW, PYALLOC_PYMEM: malloc(),

realloc(), free() (and ctx is NULL); call malloc(1) when
requesting zero bytes +* PYALLOC_PYOBJECT: pymalloc allocator which fall backs on
PyMem_Malloc() for allocations larger than 512 bytes +* pymalloc arena allocator: mmap(), munmap() (and ctx is
NULL), or malloc() and free() if mmap() is not available The builtin Python debug hooks were introduced in Python 2.3 and implement the following checks: @@ -141,23 +144,34 @@ implement the following checks:

Detect write after the end of the buffer (buffer overflow) -Other changes -------------- +Don't call malloc() directly anymore +------------------------------------ -* PyMem_Malloc() and PyMem_Realloc() always call malloc()

and realloc(), instead of calling PyObject_Malloc() and
PyObject_Realloc() in debug mode +PyMem_Malloc() and PyMem_Realloc() always call malloc() and +realloc(), instead of calling PyObject_Malloc() and +PyObject_Realloc() in debug mode. + +PyObject_Malloc() falls back on PyMem_Malloc() instead of +malloc() if size is greater or equal than 512 bytes, and +PyObject_Realloc() falls back on PyMem_Realloc() instead of +realloc() -* PyObject_Malloc() falls back on PyMem_Malloc() instead of
malloc() if size is greater or equal than
SMALL_REQUEST_THRESHOLD (512 bytes), and PyObject_Realloc()
falls back on PyMem_Realloc() instead of realloc() +Replace direct calls to malloc() with PyMem_Malloc(), or +PyMem_RawMalloc() if the GIL is not held. + +Configure external libraries like zlib or OpenSSL to allocate memory +using PyMem_Malloc() or PyMem_RawMalloc(). If the allocator of a +library can only be replaced globally, the allocator is not replaced if +Python is embedded in an application. -* Replace direct calls to malloc() with PyMem_Malloc(), or
PyMem_RawMalloc() if the GIL is not held +For the "track memory usage" use case, it is important to track memory +allocated in external libraries to have accurate reports, because these +allocations may be large. -* Configure external libraries like zlib or OpenSSL to allocate memory
using PyMem_RawMalloc() +If an hook is used to the track memory usage, the memory allocated by +malloc() will not be tracked. Remaining malloc() in external +libraries like OpenSSL or bz2 may allocate large memory blocks and so +would be missed in memory usage reports. Examples @@ -171,8 +185,8 @@ and 10 bytes per memory mapping:: #include <stdlib.h>

int block_padding = 2;
int mapping_padding = 10;

int alloc_padding = 2;
int arena_padding = 10;

void* my_malloc(void *ctx, size_t size) { @@ -191,49 +205,49 @@ and 10 bytes per memory mapping:: free(ptr); }

void* my_alloc_mapping(void *ctx, size_t size)

void* my_alloc_arena(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); }

void my_free_mapping(void *ctx, void *ptr, size_t size)

void my_free_arena(void *ctx, void *ptr, size_t size) { free(ptr); } void setup_custom_allocator(void) {

   PyMemBlockAllocator block;[](#l1.222)

   PyMemMappingAllocator mapping;[](#l1.223)

```
   PyMemAllocator alloc;[](#l1.224)
```

   PyObjectArenaAllocator arena;[](#l1.225)

   block.ctx = &block_padding;[](#l1.227)

```
   block.malloc = my_malloc;[](#l1.228)
```

   block.realloc = my_realloc;[](#l1.229)

```
   block.free = my_free;[](#l1.230)
```

   alloc.ctx = &alloc_padding;[](#l1.231)

```
   alloc.malloc = my_malloc;[](#l1.232)
```

   alloc.realloc = my_realloc;[](#l1.233)

```
   alloc.free = my_free;[](#l1.234)
```

   PyMem_SetRawAllocator(&block);[](#l1.236)

   PyMem_SetAllocator(&block);[](#l1.237)

   PyMem_SetAllocator(PYALLOC_PYMEM_RAW, &alloc);[](#l1.238)

   PyMem_SetAllocator(PYALLOC_PYMEM, &alloc);[](#l1.239)

   mapping.ctx = &mapping_padding;[](#l1.241)

   mapping.alloc = my_alloc_mapping;[](#l1.242)

   mapping.free = my_free_mapping;[](#l1.243)

   PyMem_SetMappingAllocator(mapping);[](#l1.244)

   arena.ctx = &arena_padding;[](#l1.245)

   arena.alloc = my_alloc_arena;[](#l1.246)

   arena.free = my_free_arena;[](#l1.247)

   PyObject_SetArenaAllocator(&arena);[](#l1.248)

PyMem_SetupDebugHooks(); } .. warning::

Remove the call PyMem_SetRawAllocator(&alloc) if the new
allocator are not thread-safe.

Remove the call PyMem_SetAllocator(PYALLOC_PYMEM_RAW, &alloc) if
the new allocator is not thread-safe.

Use case 2: Replace Memory Allocator, override pymalloc -------------------------------------------------------- -If your allocator is optimized for allocation of small objects (less -than 512 bytes) with a short lifetime, pymalloc can be overriden -(replace PyObject_Malloc()). +If your allocator is optimized for allocations of objects smaller than +512 bytes with a short lifetime, pymalloc can be overriden (replace +PyObject_Malloc()). Dummy example wasting 2 bytes per memory block:: @@ -260,22 +274,22 @@ Dummy example wasting 2 bytes per memory void setup_custom_allocator(void) {

   PyMemBlockAllocator alloc;[](#l1.276)

   PyMemAllocator alloc;[](#l1.277)
   alloc.ctx = &padding;[](#l1.278)
   alloc.malloc = my_malloc;[](#l1.279)
   alloc.realloc = my_realloc;[](#l1.280)
   alloc.free = my_free;[](#l1.281)

   PyMem_SetRawAllocator(&alloc);[](#l1.283)

   PyMem_SetAllocator(&alloc);[](#l1.284)

   PyObject_SetAllocator(&alloc);[](#l1.285)

   PyMem_SetAllocator(PYALLOC_PYMEM_RAW, &alloc);[](#l1.286)

   PyMem_SetAllocator(PYALLOC_PYMEM, &alloc);[](#l1.287)

   PyMem_SetAllocator(PYALLOC_PYOBJECT, &alloc);[](#l1.288)

PyMem_SetupDebugHooks(); } .. warning::

Remove the call PyMem_SetRawAllocator(&alloc) if the new
allocator are not thread-safe.

Remove the call PyMem_SetAllocator(PYALLOC_PYMEM_RAW, &alloc) if
the new allocator is not thread-safe.

@@ -285,15 +299,15 @@ Use case 3: Setup Allocator Hooks Example to setup hooks on all memory allocators:: struct {

```
   PyMemBlockAllocator raw;[](#l1.305)
```
```
   PyMemBlockAllocator mem;[](#l1.306)
```
```
   PyMemBlockAllocator obj;[](#l1.307)
```

```
   PyMemAllocator raw;[](#l1.308)
```
```
   PyMemAllocator mem;[](#l1.309)
```

   PyMemAllocator obj;[](#l1.310)
   /* ... */[](#l1.311)

} hook; static void* hook_malloc(void *ctx, size_t size) {

   PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx;[](#l1.316)

   PyMemAllocator *alloc = (PyMemAllocator *)ctx;[](#l1.317)
   /* ... */[](#l1.318)
   ptr = alloc->malloc(alloc->ctx, size);[](#l1.319)
   /* ... */[](#l1.320)

@@ -302,7 +316,7 @@ Example to setup hooks on all memory all static void* hook_realloc(void *ctx, void *ptr, size_t new_size) {

   PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx;[](#l1.325)

   PyMemAllocator *alloc = (PyMemAllocator *)ctx;[](#l1.326)
   void *ptr2;[](#l1.327)
   /* ... */[](#l1.328)
   ptr2 = alloc->realloc(alloc->ctx, ptr, new_size);[](#l1.329)

@@ -312,7 +326,7 @@ Example to setup hooks on all memory all static void hook_free(void *ctx, void *ptr) {

   PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx;[](#l1.334)

   PyMemAllocator *alloc = (PyMemAllocator *)ctx;[](#l1.335)
   /* ... */[](#l1.336)
   alloc->free(alloc->ctx, ptr);[](#l1.337)
   /* ... */[](#l1.338)

@@ -320,7 +334,7 @@ Example to setup hooks on all memory all void setup_hooks(void) {

   PyMemBlockAllocator alloc;[](#l1.343)

   PyMemAllocator alloc;[](#l1.344)
   static int installed = 0;[](#l1.345)

if (installed) @@ -330,27 +344,28 @@ Example to setup hooks on all memory all alloc.malloc = hook_malloc; alloc.realloc = hook_realloc; alloc.free = hook_free; -

   PyMem_GetRawAllocator(&hook.raw);[](#l1.353)

```
   alloc.ctx = &hook.raw;[](#l1.354)
```

   PyMem_SetRawAllocator(&alloc);[](#l1.355)

   PyMem_GetAllocator(PYALLOC_PYMEM_RAW, &hook.raw);[](#l1.356)

   PyMem_GetAllocator(PYALLOC_PYMEM, &hook.mem);[](#l1.357)

   PyMem_GetAllocator(PYALLOC_PYOBJECT, &hook.obj);[](#l1.358)

   PyMem_GetAllocator(&hook.mem);[](#l1.360)

```
   alloc.ctx = &hook.mem;[](#l1.361)
```

   PyMem_SetAllocator(&alloc);[](#l1.362)

```
   alloc.ctx = &hook.raw;[](#l1.363)
```

   PyMem_SetAllocator(PYALLOC_PYMEM_RAW, &alloc);[](#l1.364)

   PyObject_GetAllocator(&hook.obj);[](#l1.366)

```
   alloc.ctx = &hook.mem;[](#l1.367)
```

   PyMem_SetAllocator(PYALLOC_PYMEM, &alloc);[](#l1.368)

+ alloc.ctx = &hook.obj;

   PyObject_SetAllocator(&alloc);[](#l1.371)

   PyMem_SetAllocator(PYALLOC_PYOBJECT, &alloc);[](#l1.372)

} .. warning::

Remove the call PyMem_SetRawAllocator(&alloc) if hooks are not
thread-safe.

Remove the call PyMem_SetAllocator(PYALLOC_PYMEM_RAW, &alloc) if
hooks are not thread-safe.

.. note::

PyMem_SetupDebugHooks() does not need to be called: Python debug
hooks are installed automatically at startup.

PyMem_SetupDebugHooks() does not need to be called because the
allocator is not replaced: Python debug hooks are installed
automatically at startup.

Performances @@ -369,32 +384,22 @@ The full reports are attached to the iss Alternatives ============ -Only one get/set function for block allocators ----------------------------------------------- - -Replace the 6 functions: +More specific functions to get/set memory allocators +---------------------------------------------------- -* void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator) -* void PyMem_GetAllocator(PyMemBlockAllocator *allocator) -* void PyObject_GetAllocator(PyMemBlockAllocator *allocator) -* void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator) -* void PyMem_SetAllocator(PyMemBlockAllocator *allocator) -* void PyObject_SetAllocator(PyMemBlockAllocator *allocator) +Replace the 2 functions: + +* void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) +* void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) -with 2 functions with an additional domain argument: - -* int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator *allocator) -* int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator *allocator) - -These functions return 0 on success, or -1 if the domain is unknown. +with: -where domain is one of these values: - -* PYALLOC_PYMEM -* PYALLOC_PYMEM_RAW -* PYALLOC_PYOBJECT - -Drawback: the caller has to check if the result is 0, or handle the error. +* void PyMem_GetRawAllocator(PyMemAllocator *allocator) +* void PyMem_GetAllocator(PyMemAllocator *allocator) +* void PyObject_GetAllocator(PyMemAllocator *allocator) +* void PyMem_SetRawAllocator(PyMemAllocator *allocator) +* void PyMem_SetAllocator(PyMemAllocator *allocator) +* void PyObject_SetAllocator(PyMemAllocator *allocator) Make PyMem_Malloc() reuse PyMem_RawMalloc() by default @@ -404,12 +409,6 @@ Make PyMem_Malloc() reuse PyMem_RawMallo calling PyMem_SetRawAllocator() would also also patch PyMem_Malloc() indirectly. -.. note:: -

In the implementation of this PEP (issue #3329),
PyMem_RawMalloc(0) calls malloc(0),
whereas PyMem_Malloc(0) calls malloc(1). -

Add a new PYDEBUGMALLOC environment variable -------------------------------------------- @@ -445,7 +444,7 @@ Define allocator functions as macros usi to get the C filename and line number of a memory allocation. Example of PyMem_Malloc macro with the modified -PyMemBlockAllocator structure:: +PyMemAllocator structure:: typedef struct { /* user context passed as the first argument @@ -463,7 +462,7 @@ Example of PyMem_Malloc macro with t /* release a memory block */ void (*free) (void *ctx, const char *filename, int lineno, void *ptr);

} PyMemBlockAllocator;

} PyMemAllocator;

void* _PyMem_MallocTrace(const char *filename, int lineno, size_t size); @@ -485,12 +484,12 @@ changes add too much complexity for a li GIL-free PyMem_Malloc() ----------------------- -When Python is compiled in debug mode, PyMem_Malloc() calls -indirectly PyObject_Malloc() which requires the GIL to be held. -That's why PyMem_Malloc() must be called with the GIL held. +In Python 3.3, when Python is compiled in debug mode, PyMem_Malloc() +calls indirectly PyObject_Malloc() which requires the GIL to be +held. That's why PyMem_Malloc() must be called with the GIL held. -This PEP proposes to "fix" PyMem_Malloc() to make it always call -malloc(). So the "GIL must be held" restriction may be removed from +This PEP proposes changes PyMem_Malloc(): it now always call +malloc(). The "GIL must be held" restriction can be removed from PyMem_Malloc(). Allowing to call PyMem_Malloc() without holding the GIL might break @@ -516,9 +515,10 @@ Python call PyMem_Malloc() whereas t case, PyMem_Malloc() should be replaced with malloc() (or PyMem_RawMalloc()). -If an hook is used to the track memory usage, the malloc() memory -will not be seen. Remaining malloc() may allocate a lot of memory -and so would be missed in reports. +If an hook is used to the track memory usage, the memory allocated by +direct calls to malloc() will not be tracked. External libraries +like OpenSSL or bz2 should not call malloc() directly, so large +allocated will be included in memory usage reports. Use existing debug tools to analyze the memory @@ -545,8 +545,7 @@ GIL held allow to collect a lot of usefu Add msize() ----------- -Add another field to PyMemBlockAllocator and -PyMemMappingAllocator:: +Add another field to PyMemAllocator and PyObjectArenaAllocator:: size_t msize(void *ptr); @@ -574,6 +573,8 @@ It is likely for an allocator hook to be depending on the allocator. The context is a convenient way to reuse the same custom allocator or hook for different Python allocators. +In C++, the context can be used to pass this. + External libraries ================== @@ -589,6 +590,15 @@ Libraries used by Python:

expat: parserCreate()[](#l1.519) <http://hg.python.org/cpython/file/cc27d50bd91a/Modules/expat/xmlparse.c#l724> has a per-instance memory handler +* zlib: zlib 1.2.8 Manual <http://www.zlib.net/manual.html#Usage>,

pass an opaque pointer +* bz2: `bzip2 and libbzip2, version 1.0.5
http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html`_,
pass an opaque pointer +* lzma: `LZMA SDK - How to Use
http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html`_,
pass an opaque pointer +* lipmpdec doesn't have this extra ctx parameter Other libraries: @@ -596,6 +606,9 @@ Other libraries: http://developer.gnome.org/glib/unstable/glib-Memory-Allocation.html#g-mem-set-vtable`_

libxml2: xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>_, global +* Oracle's OCI: `Oracle Call Interface Programmer's Guide,

Release 2 (9.2)
http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm_[](#l1.540) [](#l1.541) See also the GNU libc: Memory Allocation Hooks http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html`_.