[Numpy-discussion] Proposal to accept NEP-18, array_function protocol (original) (raw)

Stephan Hoyer shoyer at gmail.com
Thu Aug 23 12:37:03 EDT 2018

RE: the types argument

On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith <njs at pobox.com> wrote:

This is much more of a detail as compared to the rest of the discussion, so I don't want to quibble too much about it. (Especially since if we keep things really-provisional, we can change our mind about the argument later :-).) Mostly I'm just confused, because there are lots of dunder functions in Python (and NumPy), and none of them take a special 'types' argument... so what's special about arrayfunction that makes it necessary/worthwhile?

What's special about array_function is that it's a hook that lets you override an entire API through a single interface. Unlike protocols like add, implementers of array_function don't know exactly which arguments could have implemented the operation.

Any implementation of, say, concatenate-via-arrayfunction is going to involve iterating through all the arguments and looking at each of them to figure out what kind of object it is and how to handle it, right? That's true whether or not they've done a "pre-check" using the types set, so in theory it's just as easy to return NotImplemented at that point. But I guess your point in the last paragraph is that this means there will be lots of chances to mess up the NotImplemented-returning code in particular, especially since it's less likely to be tested than the happy path, which seems plausible. So basically the point of the types set is to let people factor out that little bit of lots of functions into one common place?

It's also a pragmatic choice: libraries like dask.array and autograd.numpy have already implemented NumPy's API without overrides. These projects follow the current numpy convention: non-native array objects are coerced into native arrays (i.e., dask or autograd arrays). They don't do any type checking.

I doubt there would be much appetite for writing alternative versions of these APIs that return NotImplemented instead -- especially while this feature remains experimental.

I guess some careful devs might be unhappy with paying extra so that other lazier devs can get away with being lazy, but maybe it's a good tradeoff for us (esp. since as numpy devs, we'll be getting the bug reports regardless :-)).

The only extra amount we pay extra is the price of converting these types into a Python data structure and passing them into the array_function method call. We already had to collect them for array_function itself to identify unique types to call -- so this is a pretty minimal extra cost.

If that's the goal, then it does make me wonder if there might be a more direct way to accomplish it -- like, should we let classes define an arrayfunctiontypes attribute that numpy would check before even trying to dispatch to arrayfunction?

This could potentially work, but now the array_function protocol itself becomes more complex and out of sync with array_ufunc. It's a much smaller amount of additional complexity to add an additional passed argument. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180823/947508cc/attachment.html>

More information about the NumPy-Discussion mailing list