[Python-Dev] An updated extended buffer PEP (original) (raw)

Carl Banks pythondev at aerojockey.com
Wed Mar 28 15:24:29 CEST 2007

Carl Banks wrote:

Here's a concrete example of where it would be useful: consider a ByteBufferSlice object. Basically, the object represents a shared-memory slice of a 1-D array of bytes (for example, Python 3000 bytes object, or an mmap object).

Now, if the ByteBufferSlice object could tell the consumer that someone else is managing the buffer, then it wouldn't have to keep track of views, thus simplifying things. P.S. In thinking about this, it occurred to me that there should be a way to lock the buffer without requesting details. ByteBufferSlice would already know the details of the buffer, but it would need to increment the original buffer's lock count. Thus, I propose new fuction: typedef int (lockbufferproc)(PyObject self);

And, because real examples are better than philosophical speculations, here's a skeleton implementation of the ByteBufferSlice array, sans boilerplate and error checking, and with some educated guessing about future details:

typedef struct { PyObject_HEAD PyObject* releaser; unsigned char* buf; Py_ssize_t length; } ByteBufferSliceObject;

PyObject* ByteBufferSlice_new(PyObject* bufobj, Py_ssize_t start, Py_ssize_t end) { ByteBufferSliceObject* self; BufferInfoObject* bufinfo;

self = (ByteBufferSliceObject*)type->tp_alloc(type, 0); bufinfo = PyObject_GetBuffer(bufobj);

self->releaser = bufinfo->releaser; self->buf = bufinfo->buf + start; self->length = end-start;

/* look how soon we're done with this information */ Py_DECREF(bufinfo);

return self; }

PyObject* ByteBufferSlice_dealloc(PyObject* self) { PyObject_ReleaseBuffer(self->releaser); self->ob_type->tp_free((PyObject*)self); }

PyObject* ByteBufferSlice_getbuffer(PyObject* self, int flags) { BufferInfoObject* bufinfo; static Py_ssize_t stridesarray[] = { 1 };

bufinfo = BufferInfo_New(); bufinfo->releaser = self->releaser; bufinfo->writable = 1; bufinfo->buf = self->buf; bufinfo->length = self->length; bufinfo->ndims = 1; bufinfo->strides = stridesarray; bufinfo->size = &self->length; bufinfo->subbufoffsets = NULL;

/* Before we go, increase the original buffer's lock count */ PyObject_LockBuffer(self->releaser);

return bufinfo; }

/* don't define releasebuffer or lockbuffer / / only objects that manage buffers themselves would define these */

/* Now look how easy this is / / Everything works out if ByteBufferSlice reexports the buffer */

PyObject* ByteBufferSlice_getslice(PyObject* self, Py_ssize_t start, Py_ssize_t end) { return ByteBufferSlice_new(self,start,end); }

The implementation of this is very straightforward, and it's easy to see why and how "bufinfo->release" works, and why it'd be useful.

It's almost like there's two protocols here: a buffer exporter protocol (getbuffer) and a buffer manager protocol (lockbuffer and releasebuffer). Some objects would support only exporter protocol; others both.

Carl Banks

More information about the Python-Dev mailing list