[Python-Dev] PEP 492: async/await in Python; v3 (original) (raw)

Yury Selivanov yselivanov.ml at gmail.com
Wed Apr 29 07:00:28 CEST 2015

Guido,

I found a solution how to disable 'yield from', iter()/tuple() and 'for..in' on native coroutines with 100% backwards compatibility.

The idea is to add one more code object flag: CO_NATIVE_COROUTINE, which will be applied, along with CO_COROUTINE to all 'async def' functions.

This way:

  1. old generator-based coroutines from asyncio are awaitable, because of CO_COROUTINE flag (that asyncio.coroutine decorator will set with 'types.coroutine').

  2. new 'async def' functions are awaitable because of CO_COROUTINE flag.

  3. GenObject iter and next raise error only if it has CO_NATIVE_COROUTINE flag. So iter(), next(), for..in aren't supported only for 'async def' functions (but will work ok on asyncio generator-based coroutines)

  4. 'yield from' only raises an error if it yields a coroutine with a CO_NATIVE_COROUTINE from a regular generator.

Thanks, Yury

On 2015-04-28 7:26 PM, Yury Selivanov wrote:

Hi Guido,

Thank you for a very detailed review. Comments below: On 2015-04-28 5:49 PM, Guido van Rossum wrote: Inline comments below...

On Mon, Apr 27, 2015 at 8:07 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:

Hi python-dev,

Another round of updates. Reference implementation has been updated: https://github.com/1st1/cpython/tree/await (includes all things from the below summary of updates + tests).

Summary: 1. "PyTypeObject.tpawait" slot. Replaces "tpreserved". This is to enable implementation of Futures with C API. Must return an iterator if implemented. That's fine (though I didn't follow this closely). My main question here is it OK to reuse 'tpreserved' (former tpcompare)? I had to remove this check: https://github.com/1st1/cpython/commit/4be6d0a77688b63b917ad88f09d446ac3b7e2ce9#diff-c3cf251f16d5a03a9e7d4639f2d6f998L4906 On the other hand I think that it's a slightly better solution than adding a new slot. 2. New grammar for "await" expressions, see 'Syntax of "await" expression' section I like it. Great! The current grammar requires parentheses for consequent await expressions: await (await coro()) I can change this (in theory), but I kind of like the parens in this case -- better readability. And it'll be a very rare case. 3. inspect.iscoroutine() and inspect.iscoroutineobjects() functions. What's the use case for these? I wonder if it makes more sense to have a check for a generalized awaitable rather than specifically a coroutine. It's important to at least have 'iscoroutine' -- to check that the object is a coroutine function. A typical use-case would be a web framework that lets you to bind coroutines to specific http methods/paths: @http.get('/spam') async def handlespam(request): ... 'http.get' decorator will need a way to raise an error if it's applied to a regular function (while the code is being imported, not in runtime). The idea here is to cover all kinds of python objects in inspect module, it's Python's reflection API. The other thing is that it's easy to implement this function for CPython: just check for COCOROUTINE flag. For other Python implementations it might be a different story. (More arguments for isawaitable() below) 4. Full separation of coroutines and generators. This is a big one; let's discuss. a) Coroutine objects raise TypeError (is NotImplementedError better?) in their iter and next. Therefore it's not not possible to pass them to iter(), tuple(), next() and other similar functions that work with iterables. I think it should be TypeError -- what you really want is not to define these methods at all but given the implementation tactic for coroutines that may not be possible, so the nearest approximation is TypeError. (Also, NotImplementedError is typically to indicate that a subclass should implement it.) Agree. b) Because of (a), for..in iteration also does not work on coroutines anymore. Sounds good. c) 'yield from' only accept coroutine objects from generators decorated with 'types.coroutine'. That means that existing asyncio generator-based coroutines will happily yield from both coroutines and generators. But every generator-based coroutine must be decorated with asyncio.coroutine(). This is potentially a backwards incompatible change. See below. I worry about backward compatibility. A lot. Are you saying that asycio-based code that doesn't use @coroutine will break in 3.5? I'll experiment with replacing (c) with a warning. We can disable iter and next for coroutines, but allow to use 'yield from' on them. Would it be a better approach? d) inspect.isgenerator() and inspect.isgeneratorfunction() return False for coroutine objects & coroutine functions. Makes sense. (d) can also break something (hypothetically). I'm not sure why would someone use isgenerator() and isgeneratorfunction() on generator-based coroutines in code based on asyncio, but there is a chance that someone did (it should be trivial to fix the code). Same for iter() and next(). The chance is slim, but we may break some obscure code. Are you OK with this? e) Should we add a coroutine ABC (for cython etc)? I, personally, think this is highly necessary. First, separation of coroutines from generators is extremely important. One day there won't be generator-based coroutines, and we want to avoid any kind of confusion. Second, we only can do this in 3.5. This kind of semantics change won't be ever possible. Sounds like Stefan agrees. Are you aware of http://bugs.python.org/issue24018 (Generator ABC)? Yes, I saw the issue. I'll review it in more detail before thinking about Coroutine ABC for the next PEP update. asyncio recommends using @coroutine decorator, and most projects that I've seen do use it. Also there is no reason for people to use iter() and next() functions on coroutines when writing asyncio code. I doubt that this will cause serious backwards compatibility problems (asyncio also has provisional status). I wouldn't count too much on asyncio's provisional status. What are the consequences for code that is written to work with asyncio but doesn't use @coroutine? Such code will work with 3.4 and (despite the provisional status and the recommendation to use @coroutine) I don't want that code to break in 3.5 (though maybe a warning would be fine). I also hope that if someone has their own (renamed) copy of asyncio that works with 3.4, it will all still work with 3.5. Even if asyncio itself is provisional, none of the primitives (e.g. yield from) that it is built upon are provisional, so there should be no reason for it to break in 3.5. I agree. I'll try warnings for yield-fromming coroutines from regular generators (so that we can disable it in 3.7/3.6). If that doesn't work, I think we need a compromise (not ideal, but breaking things is worse): - yield from would always accept coroutine-objects - iter(), next(), tuple(), etc won't work on coroutine-objects - for..in won't work on coroutine-objects

Thank you, Yury Some more inline comments directly on the PEP below. PEP: 492 Title: Coroutines with async and await syntax Version: RevisionRevisionRevision Last-Modified: DateDateDate Author: Yury Selivanov <yselivanov at sprymix.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Apr-2015 Python-Version: 3.5 Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015 Abstract ======== This PEP introduces new syntax for coroutines, asynchronous with statements and for loops. The main motivation behind this proposal is to streamline writing and maintaining asynchronous code, as well as to simplify previously hard to implement code patterns. Rationale and Goals =================== Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the yield from syntax introduced in PEP 380. This approach has a number of shortcomings: * it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. @asyncio.coroutine [1]); * it is not possible to natively define a coroutine which has no yield or yield from statements, again requiring the use of decorators to fix potential refactoring issues; (I have to agree with Mark that this point is pretty weak. :-) * support for asynchronous calls is limited to expressions where yield is allowed syntactically, limiting the usefulness of syntactic features, such as with and for statements. This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new ``async with`` statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new async for statement makes it possible to perform asynchronous calls in iterators. I wonder if you could add some adaptation of the explanation I have posted (a few times now, I feel) for the reason why I prefer to suspend only at syntactically recognizable points (yield [from] in the past, await and async for/with in this PEP). Unless you already have it in the rationale (though it seems Mark didn't think it was enough :-). I'll see what I can do. Specification ============= This proposal introduces new syntax and semantics to enhance coroutine support in Python, it does not change the internal implementation of coroutines, which are still based on generators. It's actually a separate issue whether any implementation changes. Implementation changes don't need to go through the PEP process, unless they're really also interface changes. It is strongly suggested that the reader understands how coroutines are implemented in Python (PEP 342 and PEP 380). It is also recommended to read PEP 3156 (asyncio framework) and PEP 3152 (Cofunctions). From this point in this document we use the word coroutine to refer to functions declared using the new syntax. *generator-based coroutine* is used where necessary to refer to coroutines that are based on generator syntax. Despite reading this I still get confused when reading the PEP (probably because asyncio uses "coroutine" in the latter sense). Maybe it would make sense to write "native coroutine" for the new concept, to distinguish the two concepts more clearly? (You could even change "awaitable" to "coroutine". Though I like "awaitable" too.) "awaitable" is a more generic term... It can be a future, or it can be a coroutine. Mixing them in one may create more confusion. Also, "awaitable" is more of an interface, or a trait, which means that the object won't be rejected by the 'await' expression. I like your 'native coroutine' suggestion. I'll update the PEP. New Coroutine Declaration Syntax -------------------------------- The following new syntax is used to declare a coroutine:: async def readdata(db): pass Key properties of coroutines: * async def functions are always coroutines, even if they do not contain await expressions. * It is a SyntaxError to have yield or yield from expressions in an async function. For Mark's benefit, might add that this is similar to how return and yield are disallowed syntactically outside functions (as are the syntactic constraints on await` and async for|def)._ _OK_ _* Internally, a new code object flag - COCOROUTINE- is introduced_ _to enable runtime detection of coroutines (and migrating existing_ _code). All coroutines have bothCOCOROUTINE and_ _COGENERATOR_ _flags set._ _* Regular generators, when called, return a *generator object*;_ _similarly, coroutines return a *coroutine object*._ _* StopIterationexceptions are not propagated out of coroutines,_ _and are replaced with aRuntimeError. For regular generators_ _such behavior requires a future import (see PEP 479)._ _types.coroutine()_ _-----------------_ _A new function coroutine(gen)is added to thetypesmodule. It_ _appliesCOCOROUTINEflag to the passed generator-function's code_ _object, making it to return a *coroutine object* when called._ _Clarify that this is a decorator that modifies the function object in_ _place._ _Good catch._ _This feature enables an easy upgrade path for existing libraries._ _Await Expression_ _----------------_ _The following newawait expression is used to obtain a result of_ _coroutine execution::_ _async def readdata(db):_ _data = await db.fetch('SELECT ...')_ _..._ _await, similarly to yield from, suspends execution of_ _readdatacoroutine untildb.fetch*awaitable* completes and_ _returns the result data._ _It uses theyield fromimplementation with an extra step of_ _validating its argument. awaitonly accepts an *awaitable*, which_ _can be one of:_ _* A *coroutine object* returned from a *coroutine* or a generator_ _decorated withtypes.coroutine()._ _* An object with an _await_method returning an iterator._ _Anyyield fromchain of calls ends with ayield. This is a_ _fundamental mechanism of how *Futures* are implemented. Since,_ _internally, coroutines are a special kind of generators, every_ _awaitis suspended by ayield somewhere down the chain of_ _await calls (please refer to PEP 3156 for a detailed_ _explanation.)_ _To enable this behavior for coroutines, a new magic method called_ __await_is added. In asyncio, for instance, to enable Future_ _objects inawait statements, the only change is to add_ __await_ = _iter_line toasyncio.Futureclass._ _Objects with_await_method are called *Future-like*_ _objects in_ _the rest of this PEP._ _Also, please note that_aiter_method (see its definition_ _below) cannot be used for this purpose. It is a different protocol,_ _and would be like using_iter_instead of_call_for_ _regular callables._ _It is aTypeErrorif_await_returns anything but an_ _iterator._ _* Objects defined with CPython C API with atpawaitfunction,_ _returning an iterator (similar to_await_method)._ _It is aSyntaxErrorto useawaitoutside of a coroutine._ _It is aTypeErrorto pass anything other than an *awaitable* object_ _to anawait expression._ _Syntax of "await" expression_ _''''''''''''''''''''''''''''_ _awaitkeyword is defined differently fromyieldandyield_ _from. The main difference is that *await expressions* do not require_ _parentheses around them most of the times._ _Examples::_ _================================== ==================================_ _Expression Will be parsed as_ _================================== ==================================_ _if await fut: pass if (await fut): pass_ _if await fut + 1: pass if (await fut) + 1: pass_ _pair = await fut, 'spam' pair = (await fut), 'spam'_ _with await fut, open(): pass with (await fut), open(): pass_ _await foo()['spam'].baz()() await ( foo()['spam'].baz()() )_ _return await coro() return ( await coro() )_ _res = await coro() ** 2 res = (await coro()) ** 2_ _func(a1=await coro(), a2=0) func(a1=(await coro()), a2=0)_ _================================== ==================================_ _See `Grammar Updates` section for details._ _Asynchronous Context Managers and "async with"_ _----------------------------------------------_ _An *asynchronous context manager* is a context manager that is able to_ _suspend execution in its *enter* and *exit* methods._ _To make this possible, a new protocol for asynchronous context managers_ _is proposed. Two new magic methods are added: _aenter_ and_ __aexit_. Both must return an *awaitable*._ _An example of an asynchronous context manager::_ _class AsyncContextManager:_ _async def _aenter_(self):_ _await log('entering context')_ _async def _aexit_(self, exctype, exc, tb):_ _await log('exiting context')_ _New Syntax_ _''''''''''_ _A new statement for asynchronous context managers is proposed::_ _async with EXPR as VAR:_ _BLOCK_ _which is semantically equivalent to::_ _mgr = (EXPR)_ _aexit = type(mgr)._aexit__ _aenter = type(mgr)._aenter_(mgr)_ _exc = True_ _try:_ _try:_ _VAR = await aenter_ _BLOCK_ _except:_ _exc = False_ _exitres = await aexit(mgr, *sys.excinfo())_ _if not exitres:_ _raise_ _finally:_ _if exc:_ _await aexit(mgr, None, None, None)_ _I realize you copied this from PEP 343, but I wonder, do we really need_ _two nested try statements and the 'exc' flag? Why can't the finally_ _step be_ _placed in an 'else' clause on the inner try? (There may well be a reason_ _but I can't figure what it is, and PEP 343 doesn't seem to explain it.)_ _Also, it's a shame we're perpetuating the sys.excinfo() triple in_ _the API_ _here, but I agree making _exit_ and _aexit_ different also isn't a_ _great idea. :-(_ _PS. With the new tighter syntax for await you don't need the_ _exitresvariable any more._ _Yes, this can be simplified. It was indeed copied from PEP 343._ _As with regularwithstatements, it is possible to specify multiple_ _context managers in a singleasync withstatement._ _It is an error to pass a regular context manager without_aenter__ _and _aexit_methods toasync with. It is a SyntaxError_ _to use async withoutside of a coroutine._ _Example_ _'''''''_ _With asynchronous context managers it is easy to implement proper_ _database transaction managers for coroutines::_ _async def commit(session, data):_ _..._ _async with session.transaction():_ _..._ _await session.update(data)_ _..._ _Code that needs locking also looks lighter::_ _async with lock:_ _..._ _instead of::_ _with (yield from lock):_ _..._ _(Also, the implementation of the latter is problematic -- check_ _asyncio/locks.py and notice that _enter_ is empty...)_ _Asynchronous Iterators and "async for"_ _--------------------------------------_ _An *asynchronous iterable* is able to call asynchronous code in its_ _*iter* implementation, and *asynchronous iterator* can call_ _asynchronous code in its *next* method. To support asynchronous_ _iteration:_ _1. An object must implement an _aiter_method returning an_ _*awaitable* resulting in an *asynchronous iterator object*._ _Have you considered making _aiter_ not an awaitable? It's not strictly_ _necessary I think, one could do all the awaiting in _anext_. Though_ _perhaps there are use cases that are more naturally expressed by_ _awaiting_ _in _aiter_? (Your examples all useasync def _aiter_(self): return_ _selfsuggesting this would be no great loss.)_ _There is a section in Design Considerations about this._ _I should add a reference to it._ _2. An *asynchronous iterator object* must implement an_anext__ _method returning an *awaitable*._ _3. To stop iteration _anext_must raise aStopAsyncIteration_ _exception._ _An example of asynchronous iterable::_ _class AsyncIterable:_ _async def _aiter_(self):_ _return self_ _async def _anext_(self):_ _data = await self.fetchdata()_ _if data:_ _return data_ _else:_ _raise StopAsyncIteration_ _async def fetchdata(self):_ _..._ _New Syntax_ _''''''''''_ _A new statement for iterating through asynchronous iterators is_ _proposed::_ _async for TARGET in ITER:_ _BLOCK_ _else:_ _BLOCK2_ _which is semantically equivalent to::_ _iter = (ITER)_ _iter = await type(iter)._aiter_(iter)_ _running = True_ _while running:_ _try:_ _TARGET = await type(iter)._anext_(iter)_ _except StopAsyncIteration:_ _running = False_ _else:_ _BLOCK_ _else:_ _BLOCK2_ _It is a TypeErrorto pass a regular iterable without_aiter__ _method to async for. It is a SyntaxErrorto useasync for_ _outside of a coroutine._ _As for with regular forstatement,async for has an optional_ _elseclause._ _(Not because we're particularly fond of it, but because its absence_ _would_ _just introduce more special cases. :-)_ _Example 1_ _'''''''''_ _With asynchronous iteration protocol it is possible to asynchronously_ _buffer data during iteration::_ _async for data in cursor:_ _..._ _Wherecursoris an asynchronous iterator that prefetchesNrows_ _of data from a database after everyNiterations._ _The following code illustrates new asynchronous iteration protocol::_ _class Cursor:_ _def _init_(self):_ _self.buffer = collections.deque()_ _def prefetch(self):_ _..._ _async def _aiter_(self):_ _return self_ _async def _anext_(self):_ _if not self.buffer:_ _self.buffer = await self.prefetch()_ _if not self.buffer:_ _raise StopAsyncIteration_ _return self.buffer.popleft()_ _then theCursorclass can be used as follows::_ _async for row in Cursor():_ _print(row)_ _which would be equivalent to the following code::_ _i = await Cursor()._aiter_()_ _while True:_ _try:_ _row = await i._anext_()_ _except StopAsyncIteration:_ _break_ _else:_ _print(row)_ _Example 2_ _'''''''''_ _The following is a utility class that transforms a regular iterable to_ _an asynchronous one. While this is not a very useful thing to do, the_ _code illustrates the relationship between regular and asynchronous_ _iterators._ _::_ _class AsyncIteratorWrapper:_ _def _init_(self, obj):_ _self.it = iter(obj)_ _async def _aiter_(self):_ _return self_ _async def _anext_(self):_ _try:_ _value = next(self.it)_ _except StopIteration:_ _raise StopAsyncIteration_ _return value_ _async for letter in AsyncIteratorWrapper("abc"):_ _print(letter)_ _Why StopAsyncIteration?_ _'''''''''''''''''''''''_ _I keep wanting to propose to rename this to AsyncStopIteration. I know_ _it's about stopping an async iteration, but in my head I keep_ _referring to_ _it as AsyncStopIteration, probably because in other places we use_ _async (or_ _'a') as a prefix._ _I'd be totally OK with that. Should I rename it?_ _Coroutines are still based on generators internally. So, before PEP_ _479, there was no fundamental difference between_ _::_ _def g1():_ _yield from fut_ _return 'spam'_ _and_ _::_ _def g2():_ _yield from fut_ _raise StopIteration('spam')_ _And since PEP 479 is accepted and enabled by default for coroutines,_ _the following example will have itsStopIteration wrapped into a_ _RuntimeError_ _::_ _async def a1():_ _await fut_ _raise StopIteration('spam')_ _The only way to tell the outside code that the iteration has ended is_ _to raise something other than StopIteration. Therefore, a new_ _built-in exception class StopAsyncIterationwas added._ _Moreover, with semantics from PEP 479, allStopIterationexceptions_ _raised in coroutines are wrapped inRuntimeError._ _Debugging Features_ _------------------_ _One of the most frequent mistakes that people make when using_ _generators as coroutines is forgetting to use yield from::_ _@asyncio.coroutine_ _def useful():_ _asyncio.sleep(1) # this will do noting without 'yield from'_ _Might be useful to point out that this was the one major advantage_ _of PEP_ _3152 -- although it wasn't enough to save that PEP, and in your response_ _you pointed out that this mistake is not all that common. Although_ _you seem_ _to disagree with that here ("One of the most frequent mistakes ...")._ _I think it's a mistake that a lot of beginners may make at some_ _point (and in this sense it's frequent). I really doubt that_ _once you were hit by it more than two times you would make it_ _again._ _This is a small wart, but we have to have a solution for it._ _For debugging this kind of mistakes there is a special debug mode in_ _asyncio, in which @coroutinedecorator wraps all functions with a_ _special object with a destructor logging a warning. Whenever a wrapped_ _generator gets garbage collected, a detailed logging message is_ _generated with information about where exactly the decorator function_ _was defined, stack trace of where it was collected, etc. Wrapper_ _object also provides a convenient_repr_function with detailed_ _information about the generator._ _The only problem is how to enable these debug capabilities. Since_ _debug facilities should be a no-op in production mode,@coroutine_ _decorator makes the decision of whether to wrap or not to wrap based on_ _an OS environment variable PYTHONASYNCIODEBUG. This way it is_ _possible to run asyncio programs with asyncio's own functions_ _instrumented. EventLoop.setdebug, a different debug facility, has_ _no impact on @coroutinedecorator's behavior._ _With this proposal, coroutines is a native, distinct from generators,_ _concept. New methodssetcoroutinewrapper and_ _getcoroutinewrapperare added to thesysmodule, with which_ _frameworks can provide advanced debugging facilities._ _These two appear to be unspecified except by example._ _Will add a subsection specifically for them._ _It is also important to make coroutines as fast and efficient as_ _possible, therefore there are no debug features enabled by default._ _Example::_ _async def debugme():_ _await asyncio.sleep(1)_ _def asyncdebugwrap(generator):_ _return asyncio.CoroWrapper(generator)_ _sys.setcoroutinewrapper(asyncdebugwrap)_ _debugme() # <- this line will likely GC the coroutine object and_ _# trigger asyncio.CoroWrapper's code._ _assert isinstance(debugme(), asyncio.CoroWrapper)_ _sys.setcoroutinewrapper(None) # <- this unsets any_ _# previously set wrapper_ _assert not isinstance(debugme(), asyncio.CoroWrapper)_ _Ifsys.setcoroutinewrapper()is called twice, the new wrapper_ _replaces the previous wrapper.sys.setcoroutinewrapper(None)_ _unsets the wrapper._ _inspect.iscoroutine() and inspect.iscoroutineobject()_ _-----------------------------------------------------_ _Two new functions are added to the inspectmodule:_ _*inspect.iscoroutine(obj)returnsTrueifobjis a_ _coroutine object._ _*inspect.iscoroutinefunction(obj)returnsTrueisobjis a_ _coroutine function._ _Maybe isawaitable() and isawaitablefunction() are also useful? (Or only_ _isawaitable()?)_ _I think that isawaitable would be really useful. Especially,_ _to check if an object implemented with C API has a tpawait_ _function._ _isawaitablefunction() looks a bit confusing to me:_ _def foo(): return fut_ _is awaitable, but there is no way to detect that._ _def foo(arg):_ _if arg == 'spam':_ _return fut_ _is awaitable sometimes._ _Differences between coroutines and generators_ _---------------------------------------------_ _A great effort has been made to make sure that coroutines and_ _generators are separate concepts:_ _1. Coroutine objects do not implement_iter_and_next__ _methods. Therefore they cannot be iterated over or passed to_ _iter(), list(), tuple()and other built-ins. They_ _also cannot be used in afor..inloop._ _2.yield fromdoes not accept coroutine objects (unless it is used_ _in a generator-based coroutine decorated withtypes.coroutine.)_ _How does yield fromknow that it is occurring in a generator-based_ _coroutine?_ _I check that in 'ceval.c' in the implementation of YIELDFROM_ _opcode._ _If the current code object doesn't have a COCOROUTINE flag_ _and the opcode arg is a generator-object with COCOROUTINE --_ _we raise an error._ _3.yield from does not accept coroutine objects from plain Python_ _generators (*not* generator-based coroutines.)_ _I am worried about this. PEP 380 gives clear semantics to "yield from_ _<generator>" and I don't think you can revert that here. Or maybe I am_ _misunderstanding what you meant here? (What exactly are "coroutine_ _objects_ _from plain Python generators"?)_ _# *Not* decorated with @coroutine_ _def somealgorithmimpl():_ _yield 1_ _yield from nativecoroutine() # <- this is a bug_ _"somealgorithmimpl" is a regular generator. By mistake_ _someone could try to use "yield from" on a native coroutine_ _(which is 99.9% is a bug)._ _So we can rephrase it to:_ _yield fromdoes not accept *native coroutine objects*_ _from regular Python generators_ _I also agree that raising an exception in this case in 3.5_ _might break too much existing code. I'll try warnings, and_ _if it doesn't work we might want to just let this restriction_ _slip._ _4.inspect.isgenerator()andinspect.isgeneratorfunction()_ _return False for coroutine objects and coroutine functions._ _Coroutine objects_ _-----------------_ _Coroutines are based on generators internally, thus they share the_ _implementation. Similarly to generator objects, coroutine objects have_ _throw, sendandclosemethods. StopIteration and_ _GeneratorExitplay the same role for coroutine objects (although_ _PEP 479 is enabled by default for coroutines)._ _Does send() make sense for a native coroutine? Check PEP 380. I think_ _the_ _only way to access the send() argument is by usingyieldbut that's_ _disallowed. Or is this about send() being passed to theyield that_ _ultimately suspends the chain of coroutines? (You may just have to_ _rewrite_ _the section about that -- it seems a bit hidden now.)_ _Yes, 'send()' is needed to push values to the 'yield' statement_ _somewhere (future) down the chain of coroutines (suspension point)._ _This has to be articulated in a clear way, I'll think how to_ _rewrite this section without replicating PEP 380 and python_ _documentation on generators._ _Glossary_ _========_ _:Coroutine:_ _A coroutine function, or just "coroutine", is declared with_ _async_ _def. It uses awaitandreturn value``; see `New Coroutine Declaration Syntaxfor details._ _:Coroutine object:_ _Returned from a coroutine function. SeeAwait Expressionfor_ _details._ _:Future-like object:_ _An object with an ``_await_`` method, or a C object with_ _``tpawait`` function, returning an iterator. Can be consumed by_ _an ``await`` expression in a coroutine. A coroutine waiting for a_ _Future-like object is suspended until the Future-like object's_ _``_await_`` completes, and returns the result. SeeAwait Expressionfor details._ _:Awaitable:_ _A *Future-like* object or a *coroutine object*. SeeAwait Expressionfor details._ _:Generator-based coroutine:_ _Coroutines based in generator syntax. Most common example is_ _``@asyncio.coroutine``._ _:Asynchronous context manager:_ _An asynchronous context manager has ``_aenter_`` and_ _``_aexit_``_ _methods and can be used with ``async with``. SeeAsynchronous Context Managers and "async with"for details._ _:Asynchronous iterable:_ _An object with an ``_aiter_`` method, which must return an_ _*asynchronous iterator* object. Can be used with ``async for``._ _SeeAsynchronous Iterators and "async for" for details._ _:Asynchronous iterator:_ _An asynchronous iterator has an ``_anext_`` method. See_ _Asynchronous Iterators and "async for"for details._ _List of functions and methods_ _=============================_ _(I'm not sure of the utility of this section.)_ _It's a little bit hard to understand that "awaitable" is_ _a general term that includes native coroutine objects, so_ _it's OK to write both:_ _def _aenter_(): return fut_ _async def _aenter_(): ..._ _We (Victor and I) decided that it might be useful to have an_ _additional section that explains it._ _================= =================================== =================_ _Method Can contain Can't contain_ _================= =================================== =================_ _async def func await, return value yield, yield from_ _async def _a*_ await, return value yield, yield from_ _(This line seems redundant.)_ _def _a*_ return awaitable await_ _def _await_ yield, yield from, return iterable await_ _generator yield, yield from, return value await_ _================= =================================== =================_ _Where:_ _* "async def func": coroutine;_ _* "async def _a*_": ``_aiter_``, ``_anext_``, ``_aenter_``,_ _``_aexit_`` defined with the ``async`` keyword;_ _* "def _a*_": ``_aiter_``, ``_anext_``, ``_aenter_``,_ _``_aexit_`` defined without the ``async`` keyword, must return an_ _*awaitable*;_ _* "def _await_": ``_await_`` method to implement *Future-like*_ _objects;_ _* generator: a "regular" generator, function defined with ``def`` and_ _which contains a least one ``yield`` or ``yield from`` expression._ _Transition Plan_ _===============_ _This may need to be pulled forward or at least mentioned earlier (in_ _the_ _Abstract or near the top of the Specification)._ _To avoid backwards compatibility issues with ``async`` and ``await``_ _keywords, it was decided to modify ``tokenizer.c`` in such a way, that_ _it:_ _* recognizes ``async def`` name tokens combination (start of a_ _coroutine);_ _* keeps track of regular functions and coroutines;_ _* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with_ _``AWAIT`` when in the process of yielding tokens for coroutines._ _This approach allows for seamless combination of new syntax features_ _(all of them available only in ``async`` functions) with any existing_ _code._ _An example of having "async def" and "async" attribute in one piece of_ _code::_ _class Spam:_ _async = 42_ _async def ham():_ _print(getattr(Spam, 'async'))_ _# The coroutine can be executed and will print '42'_ _Backwards Compatibility_ _-----------------------_ _This proposal preserves 100% backwards compatibility._ _Is this still true with the proposed restrictions on what ``yield from``_ _accepts? (Hopefully I'm the one who is confused. :-)_ _True for the code that uses @coroutine decorators properly._ _I'll see what I can do with warnings, but I'll update the_ _section anyways._ _Grammar Updates_ _---------------_ _Grammar changes are also fairly minimal::_ _decorated: decorators (classdef | funcdef | asyncfuncdef)_ _asyncfuncdef: ASYNC funcdef_ _compoundstmt: (ifstmt | whilestmt | forstmt | trystmt |_ _withstmt_ _| funcdef | classdef | decorated | asyncstmt)_ _asyncstmt: ASYNC (funcdef | withstmt | forstmt)_ _power: atomexpr ['**' factor]_ _atomexpr: [AWAIT] atom trailer*_ _Transition Period Shortcomings_ _------------------------------_ _There is just one._ _Are you OK with this thing?_ _Until ``async`` and ``await`` are not proper keywords, it is not_ _possible (or at least very hard) to fix ``tokenizer.c`` to recognize_ _them on the **same line** with ``def`` keyword::_ _# async and await will always be parsed as variables_ _async def outer(): # 1_ _def nested(a=(await fut)):_ _pass_ _async def foo(): return (await fut) # 2_ _Since ``await`` and ``async`` in such cases are parsed as ``NAME``_ _tokens, a ``SyntaxError`` will be raised._ _To workaround these issues, the above examples can be easily rewritten_ _to a more readable form::_ _async def outer(): # 1_ _adefault = await fut_ _def nested(a=adefault):_ _pass_ _async def foo(): # 2_ _return (await fut)_ _This limitation will go away as soon as ``async`` and ``await`` ate_ _proper keywords. Or if it's decided to use a future import for this_ _PEP._ _Deprecation Plans_ _-----------------_ _``async`` and ``await`` names will be softly deprecated in CPython 3.5_ _and 3.6. In 3.7 we will transform them to proper keywords. Making_ _``async`` and ``await`` proper keywords before 3.7 might make it harder_ _for people to port their code to Python 3._ _asyncio_ _-------_ _``asyncio`` module was adapted and tested to work with coroutines and_ _new statements. Backwards compatibility is 100% preserved._ _The required changes are mainly:_ _1. Modify ``@asyncio.coroutine`` decorator to use new_ _``types.coroutine()`` function._ _2. Add ``_await_ = _iter_`` line to ``asyncio.Future`` class._ _3. Add ``ensuretask()`` as an alias for ``async()`` function._ _Deprecate ``async()`` function._ _Design Considerations_ _=====================_ _PEP 3152_ _--------_ _PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines_ _(called "cofunctions"). Some key points:_ _1. A new keyword ``codef`` to declare a *cofunction*. *Cofunction* is_ _always a generator, even if there is no ``cocall`` expressions_ _inside it. Maps to ``async def`` in this proposal._ _2. A new keyword ``cocall`` to call a *cofunction*. Can only be used_ _inside a *cofunction*. Maps to ``await`` in this proposal (with_ _some differences, see below.)_ _3. It is not possible to call a *cofunction* without a ``cocall``_ _keyword._ _4. ``cocall`` grammatically requires parentheses after it::_ _atom: cocall | <existing alternatives for atom>_ _cocall: 'cocall' atom cotrailer* '(' [arglist] ')'_ _cotrailer: '[' subscriptlist ']' | '.' NAME_ _5. ``cocall f(*args, **kwds)`` is semantically equivalent to_ _``yield from f._cocall_(*args, **kwds)``._ _Differences from this proposal:_ _1. There is no equivalent of ``_cocall_`` in this PEP, which is_ _called and its result is passed to ``yield from`` in the ``cocall``_ _expression. ``await`` keyword expects an *awaitable* object,_ _validates the type, and executes ``yield from`` on it. Although,_ _``_await_`` method is similar to ``_cocall_``, but is only used_ _to define *Future-like* objects._ _2. ``await`` is defined in almost the same way as ``yield from`` in the_ _grammar (it is later enforced that ``await`` can only be inside_ _``async def``). It is possible to simply write ``await future``,_ _whereas ``cocall`` always requires parentheses._ _3. To make asyncio work with PEP 3152 it would be required to modify_ _``@asyncio.coroutine`` decorator to wrap all functions in an object_ _with a ``_cocall_`` method, or to implement ``_cocall_`` on_ _generators. To call *cofunctions* from existing generator-based_ _coroutines it would be required to use ``costart(cofunc, *args,_ _**kwargs)`` built-in._ _4. Since it is impossible to call a *cofunction* without a ``cocall``_ _keyword, it automatically prevents the common mistake of forgetting_ _to use ``yield from`` on generator-based coroutines. This proposal_ _addresses this problem with a different approach, seeDebugging Features`._ _5. A shortcoming of requiring a cocall keyword to call a coroutine_ _is that if is decided to implement coroutine-generators --_ _coroutines with yield or async yield expressions -- we_ _wouldn't need a cocall keyword to call them. So we'll end up_ _having _cocall_ and no _call_ for regular coroutines,_ _and having _call_ and no _cocall_ for coroutine-_ _generators._ _6. Requiring parentheses grammatically also introduces a whole lot_ _of new problems._ _The following code::_ _await fut_ _await functionreturningfuture()_ _await asyncio.gather(coro1(arg1, arg2), coro2(arg1, arg2))_ _would look like::_ _cocall fut() # or cocall costart(fut)_ _cocall (functionreturningfuture())()_ _cocall asyncio.gather(costart(coro1, arg1, arg2),_ _costart(coro2, arg1, arg2))_ _7. There are no equivalents of async for and async with in PEP_ _3152._ _Coroutine-generators_ _--------------------_ _With async for keyword it is desirable to have a concept of a_ _*coroutine-generator* -- a coroutine with yield and yield from_ _expressions. To avoid any ambiguity with regular generators, we would_ _likely require to have an async keyword before yield, and_ _async yield from would raise a StopAsyncIteration exception._ _While it is possible to implement coroutine-generators, we believe that_ _they are out of scope of this proposal. It is an advanced concept that_ _should be carefully considered and balanced, with a non-trivial changes_ _in the implementation of current generator objects. This is a matter_ _for a separate PEP._ _No implicit wrapping in Futures_ _-------------------------------_ _There is a proposal to add similar mechanism to ECMAScript 7 [2]. A_ _key difference is that JavaScript "async functions" always return a_ _Promise. While this approach has some advantages, it also implies that_ _a new Promise object is created on each "async function" invocation._ _We could implement a similar functionality in Python, by wrapping all_ _coroutines in a Future object, but this has the following_ _disadvantages:_ _1. Performance. A new Future object would be instantiated on each_ _coroutine call. Moreover, this makes implementation of await_ _expressions slower (disabling optimizations of yield from)._ _2. A new built-in Future object would need to be added._ _3. Coming up with a generic Future interface that is usable for any_ _use case in any framework is a very hard to solve problem._ _4. It is not a feature that is used frequently, when most of the code_ _is coroutines._ _Why "async" and "await" keywords_ _--------------------------------_ _async/await is not a new concept in programming languages:_ _* C# has it since long time ago [5];_ _* proposal to add async/await in ECMAScript 7 [2];_ _see also Traceur project [9];_ _* Facebook's Hack/HHVM [6];_ _* Google's Dart language [7];_ _* Scala [8];_ _* proposal to add async/await to C++ [10];_ _* and many other less popular languages._ _This is a huge benefit, as some users already have experience with_ _async/await, and because it makes working with many languages in one_ _project easier (Python with ECMAScript 7 for instance)._ _Why "_aiter_" is a coroutine_ _------------------------------_ _In principle, _aiter_ could be a regular function. There are_ _several good reasons to make it a coroutine:_ _* as most of the _anext_, _aenter_, and _aexit__ _methods are coroutines, users would often make a mistake defining it_ _as async anyways;_ _* there might be a need to run some asynchronous operations in_ __aiter_, for instance to prepare DB queries or do some file_ _operation._ _Importance of "async" keyword_ _-----------------------------_ _While it is possible to just implement await expression and treat_ _all functions with at least one await as coroutines, this approach_ _makes APIs design, code refactoring and its long time support harder._ _Let's pretend that Python only has await keyword::_ _def useful():_ _..._ _await log(...)_ _..._ _def important():_ _await useful()_ _If useful() function is refactored and someone removes all_ _await expressions from it, it would become a regular python_ _function, and all code that depends on it, including important()_ _would be broken. To mitigate this issue a decorator similar to_ _@asyncio.coroutine has to be introduced._ _Why "async def"_ _---------------_ _For some people bare async name(): pass syntax might look more_ _appealing than async def name(): pass. It is certainly easier to_ _type. But on the other hand, it breaks the symmetry between async_ _def, async with and async for, where async is a modifier,_ _stating that the statement is asynchronous. It is also more consistent_ _with the existing grammar._ _Why "async for/with" instead of "await for/with"_ _------------------------------------------------_ _async is an adjective, and hence it is a better choice for a_ _*statement qualifier* keyword. await for/with would imply that_ _something is awaiting for a completion of a for or with_ _statement._ _Why "async def" and not "def async"_ _-----------------------------------_ _async keyword is a *statement qualifier*. A good analogy to it are_ _"static", "public", "unsafe" keywords from other languages. "async_ _for" is an asynchronous "for" statement, "async with" is an_ _asynchronous "with" statement, "async def" is an asynchronous function._ _Having "async" after the main statement keyword might introduce some_ _confusion, like "for async item in iterator" can be read as "for each_ _asynchronous item in iterator"._ _Having async keyword before def, with and for also_ _makes the language grammar simpler. And "async def" better separates_ _coroutines from regular functions visually._ _Why not a _future_ import_ _---------------------------_ __future_ imports are inconvenient and easy to forget to add._ _Also, they are enabled for the whole source file. Consider that there_ _is a big project with a popular module named "async.py". With future_ _imports it is required to either import it using _import_() or_ _importlib.importmodule() calls, or to rename the module. The_ _proposed approach makes it possible to continue using old code and_ _modules without a hassle, while coming up with a migration plan for_ _future python versions._ _Why magic methods start with "a"_ _--------------------------------_ _New asynchronous magic methods _aiter_, _anext_,_ __aenter_, and _aexit_ all start with the same prefix "a"._ _An alternative proposal is to use "async" prefix, so that _aiter__ _becomes _asynciter_. However, to align new magic methods with_ _the existing ones, such as _radd_ and _iadd_ it was decided_ _to use a shorter version._ _Why not reuse existing magic names_ _----------------------------------_ _An alternative idea about new asynchronous iterators and context_ _managers was to reuse existing magic methods, by adding an async_ _keyword to their declarations::_ _class CM:_ _async def _enter_(self): # instead of _aenter__ _..._ _This approach has the following downsides:_ _* it would not be possible to create an object that works in both_ _with and async with statements;_ _* it would break backwards compatibility, as nothing prohibits from_ _returning a Future-like objects from _enter_ and/or_ __exit_ in Python <= 3.4;_ _* one of the main points of this proposal is to make coroutines as_ _simple and foolproof as possible, hence the clear separation of the_ _protocols._ _Why not reuse existing "for" and "with" statements_ _--------------------------------------------------_ _The vision behind existing generator-based coroutines and this proposal_ _is to make it easy for users to see where the code might be suspended._ _Making existing "for" and "with" statements to recognize asynchronous_ _iterators and context managers will inevitably create implicit suspend_ _points, making it harder to reason about the code._ _Comprehensions_ _--------------_ _For the sake of restricting the broadness of this PEP there is no new_ _syntax for asynchronous comprehensions. This should be considered in a_ _separate PEP, if there is a strong demand for this feature._ _Async lambdas_ _-------------_ _Lambda coroutines are not part of this proposal. In this proposal they_ _would look like async lambda(parameters): expression. Unless there_ _is a strong demand to have them as part of this proposal, it is_ _recommended to consider them later in a separate PEP._ _Performance_ _===========_ _Overall Impact_ _--------------_ _This proposal introduces no observable performance impact. Here is an_ _output of python's official set of benchmarks [4]:_ _::_ _python perf.py -r -b default ../cpython/python.exe_ _../cpython-aw/python.exe_ _[skipped]_ _Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0:_ _Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASEX8664_ _x8664 i386_ _Total CPU cores: 8_ _### etreeiterparse ###_ _Min: 0.365359 -> 0.349168: 1.05x faster_ _Avg: 0.396924 -> 0.379735: 1.05x faster_ _Significant (t=9.71)_ _Stddev: 0.01225 -> 0.01277: 1.0423x larger_ _The following not significant results are hidden, use -v to_ _show them:_ _djangov2, 2to3, etreegenerate, etreeparse, etreeprocess,_ _fastpickle,_ _fastunpickle, jsondumpv2, jsonload, nbody, regexv8,_ _tornadohttp._ _Tokenizer modifications_ _-----------------------_ _There is no observable slowdown of parsing python files with the_ _modified tokenizer: parsing of one 12Mb file_ _(Lib/test/testbinop.py repeated 1000 times) takes the same amount_ _of time._ _async/await_ _-----------_ _The following micro-benchmark was used to determine performance_ _difference between "async" functions and generators::_ _import sys_ _import time_ _def binary(n):_ _if n <= 0:_ _return 1_ _l = yield from binary(n - 1)_ _r = yield from binary(n - 1)_ _return l + 1 + r_ _async def abinary(n):_ _if n <= 0:_ _return 1_ _l = await abinary(n - 1)_ _r = await abinary(n - 1)_ _return l + 1 + r_ _def timeit(gen, depth, repeat):_ _t0 = time.time()_ _for in range(repeat):_ _list(gen(depth))_ _t1 = time.time()_ _print('{}({}) * {}: total {:.3f}s'.format(_ _gen._name_, depth, repeat, t1-t0))_ _The result is that there is no observable performance difference._ _Minimum timing of 3 runs_ _::_ _abinary(19) * 30: total 12.985s_ _binary(19) * 30: total 12.953s_ _Note that depth of 19 means 1,048,575 calls._ _Reference Implementation_ _========================_ _The reference implementation can be found here: [3]._ _List of high-level changes and new protocols_ _--------------------------------------------_ _1. New syntax for defining coroutines: async def and new await_ _keyword._ _2. New _await_ method for Future-like objects, and new_ _tpawait slot in PyTypeObject._ _3. New syntax for asynchronous context managers: async with. And_ _associated protocol with _aenter_ and _aexit_ methods._ _4. New syntax for asynchronous iteration: async for. And_ _associated protocol with _aiter_, _aexit_ and new_ _built-_ _in exception StopAsyncIteration._ _5. New AST nodes: AsyncFunctionDef, AsyncFor, AsyncWith,_ _Await._ _6. New functions: sys.setcoroutinewrapper(callback),_ _sys.getcoroutinewrapper(), types.coroutine(gen),_ _inspect.iscoroutinefunction(), and inspect.iscoroutine()._ _7. New COCOROUTINE bit flag for code objects._ _While the list of changes and new things is not short, it is important_ _to understand, that most users will not use these features directly._ _It is intended to be used in frameworks and libraries to provide users_ _with convenient to use and unambiguous APIs with async def,_ _await, async for and async with syntax. Working example --------------- All concepts proposed in this PEP are implemented [3] and can be tested. :: import asyncio async def echoserver(): print('Serving on localhost:8000') await asyncio.startserver(handleconnection, 'localhost', 8000) async def handleconnection(reader, writer): print('New connection...') while True: data = await reader.read(8192) if not data: break print('Sending {:.10}... back'.format(repr(data))) writer.write(data) loop = asyncio.geteventloop() loop.rununtilcomplete(echoserver()) try: loop.runforever() finally: loop.close() References ========== .. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:asyncfunctions .. [3] https://github.com/1st1/cpython/tree/await .. [4] https://hg.python.org/benchmarks .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx .. [6] http://docs.hhvm.com/manual/en/hack.async.php .. [7] https://www.dartlang.org/articles/await-async/ .. [8] http://docs.scala-lang.org/sips/pending/async.html .. [9] https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF) Acknowledgments =============== I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew Svetlov, and Łukasz Langa for their initial feedback. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

More information about the Python-Dev mailing list