[Python-ideas] Yielding through context managers (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Sun Jan 6 10:06:31 CET 2013

On Sun, Jan 6, 2013 at 5:23 AM, Guido van Rossum <guido at python.org> wrote:

Possibly (though it will have to be a separate PEP -- PEP 3156 needs to be able to run on unchanged Python 3.3). Does anyone on this thread have enough understanding of the implementation of context managers and generators to be able to figure out how this could be specified and implemented (or to explain why it is a bad idea, or impossible)?

There aren't any syntax changes needed to implement asynchronous locks, since they're unlikely to experience high latency in exit. For that and similar cases, it's enough to use an asynchronous operation to retrieve the CM in the first place (i.e. acquire in iter rather than enter) or else have enter produce a Future that acquires the lock in iter (see http://python-notes.boredomandlaziness.org/en/latest/pep_ideas/async_programming.html#asynchronous-context-managers)

The real challenge is in handling something like an asynchronous database transaction, which will need to yield on exit as it commits or rolls back the database transaction. At the moment, the only solutions for that are to switch to a synchronous-to-asynchronous adapter like gevent or else write out the try/except block and avoid using the with statement.

It's not an impossible problem, just a tricky one to solve in a readable fashion. Some possible constraints on the problem space:

For example:

# Synchronous
for x in y:   # Invokes _iter = iter(y) and _iter.__next__()
    print(x)
#Asynchronous:
for x in yielding y:   # Invokes _iter = yield from iter(y) and

yield from _iter.next() print(x)

# Synchronous
with x as y:   # Invokes _cm = x, y = _cm.__enter__() and

_cm.exit(*args) print(y) #Asynchronous: with yielding x as y: # Invokes _cm = x, y = yield from _cm.enter() and yield from _cm.exit(*args) print(y)

A new keyword like "yielding" would make it explicit that what is going on differs from a (yield x) or (yield from x) in the corresponding expression slot.

Approaches with function level granularity may also be of interest - PEP 3152 is largely an exploration of that idea (but would need adjustments in light of PEP 3156)

Somewhat related, there's also a case to be made that "yield from x" should fall back to being equivalent to "x()" if x implements call but not iter. That way, async ready code can be written using "yield from", but passing in a pre-canned result via lambda or functools.partial would no longer require a separate operation that just adapts the asynchronous call API (i.e. iter) to the synchronous call one (i.e. call):

def async_call(f):
    @functools.wraps(f)
    def _sync(*args, **kwds):
        return f(*args, **kwds)
        yield # Force this to be a generator
    return _iterable_call

The argument against, of course, is the ease with which this can lead to a "wrong answer" problem where the exception gets thrown a long way from the erroneous code which left out the parens for the function call.

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-ideas mailing list