[Python-Dev] Python-versus-CPython question for mul dispatch (original) (raw)

Nathaniel Smith njs at pobox.com
Sun May 17 23:38:30 CEST 2015

On Sat, May 16, 2015 at 1:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

On 16 May 2015 at 07:35, Nathaniel Smith <njs at pobox.com> wrote:

On Thu, May 14, 2015 at 11:53 PM, Nathaniel Smith <njs at pobox.com> wrote:

On Thu, May 14, 2015 at 9:29 PM, Guido van Rossum <guido at python.org> wrote:

I expect you can make something that behaves like list by defining mul and rmul and returning NotImplemented.

Hmm, it's fairly tricky, and part of the trick is that you can never return NotImplemented (because you have to pretty much take over and entirely replace the normal dispatch rules inside mul and rmul), but see attached for something I think should work. So I guess this is just how Python's list, tuple, etc. work, and PyPy and friends need to match... For the record, it looks like PyPy does already have a hack to implement this -- they do it by having a hidden flag on the built-in sequence types which the implementations of '*' and '+' check for, and if it's found it triggers a different rule for dispatching to the op methods: https://bitbucket.org/pypy/pypy/src/a1a494787f4112e42f50c6583e0fea18db3fb4fa/pypy/objspace/descroperation.py?at=default#cl-692 Oh, that's rather annoying that the PyPy team implemented bug-for-bug compatibility there, and didn't follow up on the operand precedence bug report to say that they had done so. We also hadn't previously been made aware that NumPy is relying on this operand precedence bug to implement publicly documented API behaviour, so fixing it would break end user code :(

I don't think any of us were aware of it either :-).

It is a fairly obscure case -- it only comes up specifically if you have a single-element integer array that you are trying to multiply by a list that you expect to be auto-coerced to an array. If Python semantics were such that this became impossible to handle correctly then we would survive. (We've certainly survived worse, e.g. arr[array_of_indices] += 1 silently gives the wrong/unexpected result when array_of_indices has duplicate entries, and this bites people constantly. Unfortunately I can't see any reasonable way to fix this within Python's semantics, so... oh well.)

But yeah, given that we're at a point where list dispatch actually has worked this way forever and across multiple interpreter implementations, I think it's de facto going to end up part of the language specification unless someone does something pretty quick...

I guess that means someone in the numeric community will need to write a PEP to make this "try the other operand first" "feature" part of the language specification, so that other interpreters can implement it up front, rather than all having to come up with their own independent custom hacks just to make NumPy work.

I'll make a note...

P.S. It would also be nice if someone could take on the PEP for a Python level buffer API for 3.6: http://bugs.python.org/issue13797

At a guess, if you want to find people who have this itch strong enough to try scratching it, then probably numpy users are actually not your best bet, b/c if you have numpy then you already have workarounds. In particular, numpy still supports a legacy Python level buffer export API:

http://docs.scipy.org/doc/numpy/reference/arrays.interface.html#python-side

So if all you want is to hand a buffer to numpy (rather than to an arbitrary PEP 3118 consumer) then this works fine, and if you do need an arbitrary PEP 3118 consumer then you can use numpy as an adaptor (use array_interface to convert your object to ndarray -> ndarray supports the PEP 3118 API).

-n

-- Nathaniel J. Smith -- http://vorpus.org

More information about the Python-Dev mailing list