[Python-Dev] PEP 3118: Extended buffer protocol (new version) (original) (raw)

Carl Banks pythondev at aerojockey.com
Thu Apr 12 04:17:35 CEST 2007

Travis Oliphant wrote:

Carl Banks wrote:

Travis Oliphant wrote: > PyBUFREADONLY > The returned buffer must be readonly and the underlying object should make > its memory readonly if that is possible. I don't like the "if possible" thing. If it makes no guarantees, it pretty much useless over PyBUFSIMPLE. O.K. Let's make it raise an error if it can't set it read-only.

The thing that bothers me about this whole flags setup is that different flags can do opposite things.

Some of the flags RESTRICT the kind of buffers that can be exported (Py_BUF_WRITABLE); other flags EXPAND the kind of buffers that can be exported (Py_BUF_INDIRECT). That is highly confusing and I'm -1 on any proposal that includes both behaviors. (Mutually exclusive sets of flags are a minor exception: they can be thought of as either RESTICTING or EXPANDING, so they could be mixed with either.)

I originally suggested a small set of flags that expand the set of allowed buffers. Here's a little Venn diagram of buffers to illustrate what I was thinking:

http://www.aerojockey.com/temp/venn.png

With no flags, the only buffers allowed to be returned are in the "All" circle but no others. Add Py_BUF_WRITABLE and now you can export writable buffers as well. Add Py_BUF_STRIDED and the strided circle is opened to you, and so on.

My recommendation is, any flag should turn on some circle in the Venn diagram (it could be a circle I didn't draw--shaped arrays, for example--but it should be some circle).

PyBUFFORMAT The consumer will be using the format string information so make sure that member is filled correctly.

Is the idea to throw an exception if there's some other data format besides "b", and this flag isn't set? It seems superfluous otherwise. The idea is that a consumer may not care about the format and the exporter may want to know that to simplify the interface. In other words the flag is a way for the consumer to communicate that it wants format information (or not).

I'm -1 on using the flags for this. It's completely out of character compared to the rest of the flags. All other flags are there for the benefit of the consumer; this flag is useless to the consumer.

More concretely, all the rest of the flags are there to tell the exporter what kind of buffer they're prepared to accept. This flag, alone, does not do that.

Even the benefits to the exporter are dubious. This flag can't reduce code complexity, since all buffer objects have to be prepared to furnish type information. At best, this flag is a rare optimization. In fact, most buffers are going to point format to a constant string, regardless of whether this flag was passed or not:

bufinfo->format = "b";

If the exporter wants to raise an exception if the format is not requested is up to the exporter.

That seems like a bad idea. Suppose I have a contiguous numpy array of floats and I want to view it as a sequence of bytes. If the exporter's allowed to raise an exception for this, any consumer that wanted a data-neutral view of the data would still have to pass Py_BUF_FORMAT to guard against this. Wouldn't that be ironic?

PyBUFSHAPE The consumer can (and might) make use of using the ndims and shape members of the structure so make sure they are filled in correctly. PyBUFSTRIDES (implies SHAPE) The consumer can (and might) make use of the strides member of the structure (as well as ndims and shape)

Is there any reasonable benefit for allowing PyBUFSHAPE without PyBUFSTRIDES? Would the array be C- or Fortran-like? Yes, I could see a consumer not being able to handle simple striding but could handle shape information. Many users of NumPy arrays like to think of the array as an N-d array but want to ignore striding.

Ok, but is the indexing row-major or column-major? That has to be decided.

Carl Banks

More information about the Python-Dev mailing list