[Python-Dev] The role of NotImplemented: What is it for and when should it be used? (original) (raw)

Ethan Furman ethan at stoneleaf.us
Mon Nov 3 11:30:53 CET 2014

Just to be clear, this is about NotImplemented, not NotImplementedError.

tl;dr When a binary operation fails, should an exception be raised or NotImplemented returned?

When a binary operation in Python is attempted, there are two possibilities:

The main reason [1] that it can't work is that the two operands are of different types, and the first type does not know how to deal with the second type.

The question then becomes: how does the first type tell Python that it cannot perform the requested operation? The most obvious answer is to raise an exception, and TypeError is a good candidate. The problem with the exception raising approach is that once an exception is raised, Python doesn't try anything else to make the operation work.

What's wrong with that? Well, the second type might know how to perform the operation, and in fact that is why we have the reflected special methods, such as radd and rmod -- but if the first type raises an exception the rxxx methods will not be tried.

Okay, how can the first type tell Python that it cannot do what is requested, but to go ahead and check with the second type to see if it does? That is where NotImplemented comes in -- if a special method (and only a special method) returns NotImplemented then Python will check to see if there is anything else it can do to make the operation succeed; if all attempts return NotImplemented, then Python itself will raise an appropriate exception [2].

In an effort to see how often NotImplemented is currently being returned I crafted a test script [3] to test the types bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, deque, and OrderedDict with the operations for add, and, floordiv, iadd, iand, ifloordiv, ilshift, imod, imul, ior, ipow, irshift, isub, itruediv, ixor, lshift, mod, mul, or, pow, rshift, sub, truediv, and xor.

Here are the results of the 275 tests:

testing control...

ipow -- Exception <unsupported operand type(s) for ** or pow(): 'Control' and 'subtype'> raised errors in Control -- misunderstanding or bug?

testing types against a foreign class

iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'> raised instead of TypeError iand(Counter()) -- NotImplemented not returned, TypeError not raised ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'> raised instead of TypeError isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'> raised instead of TypeError

testing types against a subclass

mod(str()) -- NotImplemented not returned, TypeError not raised

iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised (should have worked) iand(Counter()) -- NotImplemented not returned, TypeError not raised ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised (should have worked) isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised (should have worked)

Two observations:

Back to the main point...

So, if my understanding is correct:

Is my understanding correct? Is this already in the docs somewhere, and I just missed it?

-- Ethan

[1] at least, it's the main reason in my code [2] usually a TypeError, stating either that the operation is not supported, or the types are unorderable [3] test script at the end [4] https://bugs.python.org/issue22766 [returning NotImplemented was rejected]

-- 8< ---------------------------------------------------------------------------- from collections import Counter, defaultdict, deque, OrderedDict from fractions import Fraction from decimal import Decimal from enum import Enum import operator import sys

py_ver = sys.version_info[:2]

types = ( bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, deque, OrderedDict, ) numeric_types = int, float, Decimal, Fraction

operators = ( 'add', 'and', 'floordiv', 'iadd', 'iand', 'ifloordiv', 'ilshift', 'imod', 'imul', 'ior', 'ipow', 'irshift', 'isub', 'itruediv', 'ixor', 'lshift', 'mod', 'mul', 'or', 'pow', 'rshift', 'sub', 'truediv', 'xor', )

if py_ver >= (3, 0): operators += ('gt', 'ge', 'le','lt')

ordered_reflections = { 'le': 'ge', 'lt': 'gt', 'ge': 'le', 'gt': 'lt', }

helpers

class SomeOtherClass: """" used to test behavior when a different type is passed in to the special methods """ def repr(self): return 'SomeOtherClass' some_other_class = SomeOtherClass()

class MainClassHandled(Exception): """ called by base class if both operands are of type base class """

class SubClassCalled(Exception): """ called by reflected operations for testing """

def create_control(test_op): def _any(self, other): if not type(other) is self.class: return NotImplemented raise MainClassHandled class Control: "returns NotImplemented when other object is not supported" _any.name = op setattr(Control, test_op, _any) return Control()

def create_subtype(test_op, base_class=object): def _any(*a): global subclass_called subclass_called = True raise SubClassCalled class subtype(base_class): add = sub = mul = truediv = floordiv = _any mod = divmod = pow = lshift = rshift = _any and = xor = or = _any radd = rsub = rmul = rtruediv = rfloordiv = _any rmod = rdivmod = rpow = rlshift = rrshift = _any rand = rxor = ror = _any le = lt = gt = ge = _any if issubclass(subtype, (bytes, bytearray)): value = b'hello' elif issubclass(subtype, str): value = 'goodbye' elif issubclass(subtype, (list, tuple)): value = (1, 2, 3) elif issubclass(subtype, (int, float, Decimal, Fraction)): value = 42 else: # ignore value return subtype() return subtype(value)

test exceptions

control against some other class

print('testing control...\n') errors = False for op in operators: control = create_control(op) op = getattr(operator, op) try: op(control, some_other_class) except TypeError: # the end result of no method existing, or each method called returning # NotImplemented because it does not know how to perform the requested # operation between the two types pass except Exception as exc: errors = True print('%s(%s()) -- Exception <%s> raised instead of TypeError' % (op.name, test_type.name, exc)) else: errors = True print('Control -- TypeError not raised for op %r' % op) if errors: print('errors in Control -- misunderstanding or bug?\n')

control against a subclass

errors = False for op in operators: subclass_called = False control = create_control(op) subtype = create_subtype(op, control.class) op = getattr(operator, op) try: op(control, subtype) except SubClassCalled: # if the control class properly signals that it doesn't know how to # perform the operation, of if Python notices that a reflected # operation exists, we get here (which is good) pass except MainClassHandled: errors = True print('Control did not yield to subclass for op %r' % op) except Exception as exc: if subclass_called: # exception was subverted to something more appropriate (like # unorderable types) pass errors = True print('%s -- Exception <%s> raised' % (op.name, exc)) else: errors = True print('Control -- op %r appears to have succeeded (it should not have)' % op) if errors: print('errors in Control -- misunderstanding or bug?\n')

tests

print('testing types against a foreign class\n') for test_type in types + numeric_types: errors = False for op in operators: op = getattr(operator, op) try: op(test_type(), some_other_class) except TypeError: pass except Exception as exc: errors = True print('%s(%s()) -- Exception <%s> raised instead of TypeError' % (op.name, test_type.name, exc)) else: print('%s(%s()) -- NotImplemented not returned, TypeError not raised' % (op.name, test_type.name)) if errors: print()

print()

test subclasses

print('testing types against a subclass\n') for test_type in types: errors = False for op in operators: subclass_called = False if not test_type.dict.get(op): continue subclass = create_subtype(op, test_type) op = getattr(operator, op) try: if test_type is str: op('%s', subtype) else: op(test_type(), subtype) except SubClassCalled: # expected, ignore pass except Exception as exc: if subclass_called: # exception raised by subclass was changed pass errors = True print('%s(%s()) -- Exception <%s> raised (should have worked)' % (op.name, test_type.name, exc)) else: errors = True print('%s(%s()) -- NotImplemented not returned, TypeError not raised' % (op.name, test_type.name)) if errors: print() for test_type in numeric_types: errors = False for op in operators: subclass_called = False if not test_type.dict.get(op): continue subtype = create_subtype(op, test_type) op = getattr(operator, op) try: op(test_type(), subtype) except SubClassCalled: # expected, ignore pass except Exception as exc: if subclass_called: # exception raised by subclass was changed pass errors = True print('%s(%s()) -- Exception <%s> raised (should have worked)' % (op.name, test_type.name, exc)) else: errors = True print('%s(%s)) -- NotImplemented not returned' % (op.name, test_type.name)) if errors: print() -- 8< ----------------------------------------------------------------------------

More information about the Python-Dev mailing list