[Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore(). (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Thu Oct 17 17:26:16 CEST 2013

On 17 October 2013 01:24, Barry Warsaw <barry at python.org> wrote:

On Oct 16, 2013, at 08:31 AM, Eric Snow wrote:

When a module's maintainer makes a decision on a relatively insignificant addition to the module, I'd expect little resistance or even comment (the original commit was months ago). That's why I'm surprised by the reaction to this change. It just seems like the whole thing is being blown way out of proportion to the detriment of other interesting problems. Sure, things could have been done differently. But in this case it's not that big a deal. Any project as big and diverse as Python needs a hierarchical structure of trust and responsibility. I see it roughly as core dev < module maintainer < release manager < bdfl delegate < bdfl. However, it's imperative to remain vigilantly transparent so that everyone understands the rationale and motivation behind a change, even if they disagree with it. Trust is extended upwards when this transparency is extended downloads. "'Cause I said so" only works at the top of the chain. ;)

Right, and that's why I never opted out of the thread entirely, despite being seriously tempted at times :)

It's also caused me to reflect on a few things over the past few days, including why the bar for contextlib is (at this point in time) comparatively lower than the one for functools, collections and itertools. I was also asked a very good question off-list as to why TransformDict escalated to needing a PEP, while I still believe a couple of issues is adequate for keeping this minor addition to contextlib (for the record: the patch Zero posted to http://bugs.python.org/issue19266 has been committed, so the pattern is now called "suppress").

An interesting lightning talk at BrisPy last night also included a key point: the speaker spent a lot of time creating custom internal tools for process engineers to use as an alternative to getting expensive Matlab licenses, and his experience was that object-oriented programming was just beyond what most process engineers could cope with in terms of writing their own code.

If you look at the history of various constructs in Python, it takes years for idioms and patterns to build up around them, even after the core syntax additions is made to the language. Once those idioms are in place, though, you get to a point where you can first teach people how to use them effectively and only later teach them how to build their own. Because the patterns have been enshrined in things with names, you also don't need to be able to recognise them on sight - you have something to look up the first you set, rather than having to interpret a particular shape of code as indicating a particular idiom.

This pattern recurs again and again:

Over time, this progressively lowers the barrier to entry for Python programming, as the intent is to enable users to do more without necessarily needing to learn (much) more, as well as allowing code authors to more clearly communicate their intent to code readers by using named patterns rather than leaving them implicit in the code structure.

Exception handling, however, is a notable exception(!) to that general pattern. Appropriately handling exceptions is actually harder than raising them:

raise Exception

vs:

try:
    <code that may raise an exception>
except Exception:
    <do something about it>

The introduction of context managers several years ago basically provides an opportunity to change that over time: by standardising various common try/except idioms, it becomes feasible to postpone the explanations of the underlying details.

Compared to the extensive toolkits we already have for functional abstractions, iterator based abstractions and data containers, the general purpose context manipulation idioms are still quite weak. contextlib.closing was part of the initial toolkit in Python 2.5, while contextlib.nested was later removed as fundamentally broken in the presence of resources that acquire their resource in init (like files).

contextlib.contextmanager is a tool for writing context managers not using them, as is contextlib.ContextDecorator.

contextlib.ExitStack is similarly on the "power tool" side of the fence, being the context management equivalent of the iter() and next() builtins (letting you cleanly manipulate the individual steps of a with statement, just as those two builtins let you manipulate the individual steps of a for loop).

By contrast, suppress() and redirect_stdout() are the first general purpose context managers added to contextlib since its incarnation in Python 2.5 (although there have been many various domain specific context manager additions elsewhere in the standard library).

It's worth unpacking the "simple" six line definition of contextlib.suppress, to see how many Python concepts it actually requires you to understand:

@contextmanager
def suppress(*exceptions):
    try:
        yield
    except exceptions:
        pass

By my count, it's at least:

  1. defining functions
  2. applying decorators
  3. accepting arbitrary positional arguments
  4. generators
  5. implementing context managers with generators
  6. exception handling
  7. using dynamic tuples for exception handling
  8. the interaction between generators, exception handling and context management

This function is short because of the information density provided by the various constructs it uses, but it's far from being simple.

We then turn to the motivating example for the addition, taking an eight (or five) line construct and turning it into a four (or even two!) line construct:

First, the eight line version:

try:
    os.remove("somefile.tmp")
except FileNotFoundError:
    pass

try:
    os.remove("someotherfile.tmp")
except FileNotFoundError:
    pass

And the five line version (which only applies in this particular example because the two commands are almost identical, and carries the cost of conveying less information in the traceback purely through line position):

for name in ("somefile.tmp", "someotherfile.tmp"):
    try:
        os.remove(name)
    except FileNotFoundError:
        pass

Now, the four line version:

with suppress(FileNotFoundError):
   os.remove("somefile.tmp")

with suppress(FileNotFoundError):
   os.remove("someotherfile.tmp")

And even a two line version:

with suppress(FileNotFoundError): os.remove("somefile.tmp")
with suppress(FileNotFoundError): os.remove("someotherfile.tmp")

With the try/except version, it's much harder to tell at a glance what the code is doing, because significant parts of the intent (the specific exception being caught and the "pass" keyword that indicates it is being suppressed) are hidden down and to the right.

Now, the other point raised in the thread is that there's a more general idiom that's potentially of interest, which is to keep track of whether or not an exception was thrown. I actually agree with that, and it's something I now plan to explore in contextlib2 and then (if I decide I actually like the construct) a PEP for Python 3.5. Specifically, it would be a replacement for the existing search loop idiom that was based on a context manager in order to generalise to more than one level of loop nesting. However, the potential ramifications of offering such a feature are vastly greater than those of the far more limited contextlib.suppress context manager (which only covers one very specific exception handling pattern), so it would need to go through the PEP process.

it would be something along the lines of the following (note: I'm not proposing this for 3.4. There's no way I'd propose it or anything like it for the standard library without trialling it in contextlib2 first, and letting it bake there for at least few months. It may even make sense to tie the failed/missing idiom to suppress() as was suggested in this thread rather than to the new construct):

# Arbitrarily nested search loop
with exit_label() as found:
    for i in range(x):
        for j in range(y):
            if matches(i, j):
                found.exit((i, j))
if found:
    print(found.value)

# Delayed exception handling
with exit_label(OSError) as failed:
    os.remove("somefile.tmp")
# Do other things...
if failed:
    print(failed.exc)

with exit_label(ValueError) as missing:
    location = data.find(substr)

if missing:

# Implementation sketch
def exit_label(exception):
    return _ExitLabel(exception)

class _ExitLabel:
    def __init__(exception):
        if not exception:
            class ExitToLabel(Exception):
                pass
            exception = ExitToLabel
        self._exception = exception
        self._sentinel = sentinel = object()
        self._value = sentinel
        self._exc = sentinel
        self._entered = False

    def __enter__(self):
        if self._entered:
            raise RuntimeError("Cannot reuse exit label")
        self._entered = True
        return self

    def __exit__(self, exc_type, exc_value, exc_tb):
       self._exc = exc_value
       if isinstance(exc_value, self._exceptions):
            traceback.clear_frames(exc_value.__traceback__)
            return True # Suppress the exception
        return False # Propagate the exception

    def __bool__(self):
       return self._exc and self._exc is not self._sentinel

     def exit(self, value=None):
         self._value = value
         raise self._exception

    @property
    def value(self):
        if self._value is self._sentinel:
            raise RuntimeError("Label value not yet set")
        return self._value

    @property
    def exc(self):
        if self._exc is self._exc:
            raise RuntimeError("Label exception result not yet set")
        return self._exc

Regards, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list