[Python-Dev] pyparallel and new memory API discussions... (original) (raw)

Victor Stinner victor.stinner at gmail.com
Wed Jun 19 18:20:15 CEST 2013

""" So, Victor, I'm interested to hear how the new API you're proposing will affect this solution I've come up with for pyparallel; I'm going to be absolutely dependent upon the ability to lock main thread pages as read-only in one fell-swoop -- am I still going to be able to do that with your new API in place? """

2013/6/19 Trent Nelson <trent at snakebite.org>:

On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:

> 1. All memory allocated in a parallel context is localized to a > private heap.

How do you allocate memory in this "private" heap? Did you add new functions to allocate memory? Yup: PyHeapMalloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365. All memory operations (PyObjectNew/Malloc etc) get intercepted during parallel thread execution and redirected to PyHeapMalloc(), which is a very simple slab allocator. (No need for convoluted buckets because we never free individual objects during parallel execution; instead, we just blow everything away at the end.)

Ok, so I don't think that the PEP 445 would change anything for you.

The following change might have an impact: If _PyHeap_Malloc is not thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the GIL is not held would avoid bugs in your code.

If you want to choose dynamically the allocator at runtime, you can replace PyObject_Malloc allocator using: -------------------------- 8< ----------------- static void * _PxMem_AllocMalloc(void *ctx, size_t size) { PyMemBlockAllocator *ctx; if (Py_PXCTX) return _PxMem_Malloc(size)) else return alloc->malloc(alloc->ctx, size); }

...

PyMemBlockAllocator pyparallel_pyobject;

static void * setup_pyparallel_allocator(void) { PyMemBlockAllocator alloc; PyObject_GetAllocator(&pyparallel_pyobject); alloc.ctx = &pyparallel_pyobject; alloc.malloc = _PxMem_AllocMalloc; ... PyObject_SetAllocator(&alloc); } -------------------------- 8< -----------------

But I don't know if you want pyparallel to be an "optional" feature chosen at runtime...

Victor

More information about the Python-Dev mailing list