[Python-Dev] Understanding the buffer API (original) (raw)

Stefan Krah stefan at bytereef.org
Sat Aug 4 17:25:49 CEST 2012

Jeff Allen <ja...py at farowl.co.uk> wrote:

I still don't see why, if the consumer says "I'm assuming 1-D unsigned bytes", and that's what the data is, memoryviewgetbuf could not provide a shape and strides that agree with the data.

In most cases it won't matter. However, a consumer is entitled to rely on shape==NULL in response to a PyBUF_SIMPLE request. Perhaps there is code that tests for shape==NULL to determine C-contiguity.

This is an example that might occur in C. You hinted at the fact that not all of this may be relevant for Java, but on that I can't comment.

When you say such-and-such is disallowed, but the PEP or the data structures seem to provide for it, you mean memoryviewgetbuf() disallows it, since you've concluded it is not sensible?

The particular request of PyBUF_SIMPLE|PyBUF_FORMAT, when applied to any array that is not one-dimensional with format 'B' would lead to a contradiction: PyBUF_SIMPLE implies 'B', but format would be set to something else.

It is also a useless combination, since a plain PyBUF_SIMPLE suffices.

>I think the protocol would benefit from changing the getbuffer rules to: > > a) The buffer gets a 'flags' field that can store properties like > PyBUFSIMPLE, PyBUFCCONTIGUOUS etc. > > b) The exporter must always provide full information. > > c) If a buffer can be exported as unsigned bytes but has a different > layout, the exporter must perform a full cast so that the above > mentioned invariants are kept. > Just like PyManagedBuffer mbuf and its sister view in memoryview? I've thought the same things, but the tricky part is to do it compatibly.

a) I think I can achieve this. As I have interfaces and polymorphism on my side, and a commitment only to logical equivalence to CPython, I can have the preserved flags stashed away inside to affect behaviour. But it's not as simple as saving the consumer's request, and I'm still trying to work it out what to do, e.g. when the consumer didn't ask for C-contiguity, but in this case it happens to be true. In the same way, functions you have in abstract.c etc. can be methods that, rather than work out by inspection of a struct how to navigate the data on this call, already know what kind of buffer they are in. So SimpleBuffer.isContiguous(char order) can simply return true.

Avoiding repeated calls to PyBuffer_IsContiguous() was in fact the main reason for storing flags in the new MemoryViewObject.

It would be handy to have these flags in the Py_buffer structure, but that can only be considered for a future version of Python, perhaps no earlier than 4.0. The same applies of course to all three points that I made above.

Stefan Krah

More information about the Python-Dev mailing list