[Python-Dev] PEP 550 v4: coroutine policy (original) (raw)

Yury Selivanov yselivanov.ml at gmail.com
Tue Aug 29 15:59:19 EDT 2017

On Tue, Aug 29, 2017 at 3:32 PM, Antoine Pitrou <antoine at python.org> wrote:

Le 29/08/2017 à 21:18, Yury Selivanov a écrit : On Tue, Aug 29, 2017 at 2:40 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: On Mon, 28 Aug 2017 17:24:29 -0400 Yury Selivanov <yselivanov.ml at gmail.com> wrote:

Long story short, I think we need to rollback our last decision to prohibit context propagation up the call stack in coroutines. In PEP 550 v3 and earlier, the following snippet would work just fine:

var = newcontextvar() async def bar(): var.set(42) async def foo(): await bar() assert var.get() == 42 # with previous PEP 550 semantics rununtilcomplete(foo()) But it would break if a user wrapped "await bar()" with "waitfor()": var = newcontextvar() async def bar(): var.set(42) async def foo(): await waitfor(bar(), 1) assert var.get() == 42 # AssertionError !!! rununtilcomplete(foo()) [...] Why wouldn't the bar() coroutine inherit the LC at the point it's instantiated (i.e. where the synchronous bar() call is done)? We want tasks to have their own isolated contexts. When a task is started, it runs its code in parallel with its "parent" task. I'm sorry, but I don't understand what it all means. To pose the question differently: why is example #1 supposed to be different, philosophically, than example #2? Both spawn a coroutine, both wait for its execution to end. There is no reason that adding a waitfor() intermediary (presumably because the user wants to add a timeout) would significantly change the execution semantics of bar().

I see your point. The currently published version of the PEP (v4) fixes this by saying: each coroutine has its own LC. Therefore, "var.set(42)" cannot be visible to the code that calls "bar()". And therefore, "await wait_for(bar())" and "await bar()" work the same way with regards to execution context semantics.

Unfortunately, while this fixes above examples to work the same way, setting context vars in "aenter" stops working:

 class MyAsyncCM:

         def __aenter__(self):
                var.set(42)

 async with MyAsyncCM():
       assert var.get() == 42

Because aenter has its own LC, the code wrapped in "async with" will not see the effect of "var.set(42)"!

This absolutely needs to be fixed, and the only way (that I know) it can be fixed is to revert the "every coroutine has its own LC" statement (going back to the semantics coroutines had in PEP 550 v2 and v3).

waitfor() in the above example creates an asyncio.Task implicitly, and that's why we don't see 'var' changed to '42' in foo(). I don't understand why a non-obvious behaviour detail (the fact that waitfor() creates an asyncio.Task implicitly) should translate into a fundamental difference in observable behaviour. I find it counter-intuitive and error-prone.

"await bar()" and "await wait_for(bar())" are actually quite different. Let me illustrate with an example:

b1 = bar()
# bar() is not running yet
await b1

b2 = wait_for(bar())
# bar() was wrapped into a Task and is being running right now
await b2

Usually this difference is subtle, but in asyncio it's perfectly fine to never await on b2, just let it run until it completes. If you don't "await b1" -- b1 simply will never run.

All in all, we can't say that "await bar()" and "await wait_for(bar())" are equivalent. The former runs bar() synchronously within the coroutine that awaits it. The latter runs bar() in a completely separate and detached task in parallel to the coroutine that spawned it.

This is a slightly complicated case, but it's addressable with a good documentation and recommended best practices. It would be better addressed with consistent behaviour that doesn't rely on specialist knowledge, though :-/

I agree. But I don't see any other solution that would solve the problem and satisfy the following requirements:

  1. Context variables set in "CM.aenter" and "CM.aexit" should be visible to code that is wrapped in "async with CM()".

  2. Tasks must have isolated contexts -- changes that coroutines do to the EC in one Task, should not be visible to other Tasks.

Yury

More information about the Python-Dev mailing list