[Python-Dev] async/await in Python; v2 (original) (raw)

Paul Sokolovsky pmiscml at gmail.com
Thu Apr 23 01:12:36 CEST 2015

Hello,

On Wed, 22 Apr 2015 09:53:39 -0700 Rajiv Kumar <rajiv.kumar at gmail.com> wrote:

I'd like to suggest another way around some of the issues here, with apologies if this has already been discussed sometime in the past.

From the viewpoint of a Python programmer, there are two distinct reasons for wanting to suspend execution in a block of code: 1. To yield a value from an iterator, as Python generators do today. 2. To cede control to the event loop while waiting for an asynchronous task to make progress in a coroutine. As of today both of these reasons to suspend are supported by the same underlying mechanism, i.e. a "yield" at the end of the chain of "yield from"s. PEPs 492 and 3152 introduce "await" and "cocall", but at the bottom of it all there's effectively still a yield as I understand it. I think that the fact that these two concepts use the same mechanism is what leads to the issues with coroutine-generators that Greg and Yury have raised. With that in mind, would it be possible to introduce a second form of suspension to Python to specifically handle the case of ceding to the event loop?

Barring adding an adhoc statement "yield_to_a_main_loop", there's a generic programming device to do it: symmetric coroutines. But it's unlikely to help with your sentiment that the same device is used for different purposes. At least with asymmetric coroutines as currently in Python, you have next() to "call" a coroutine and yield to "return" from. With symmetric coroutines, you don't have a place to return - you can only "call" another coroutine, and then have freedom to call any (including a main loop), but need to bother to always know whom you want to call.

But I guess it's already sounds confusing enough for folks who didn't hear about symmetric coroutines, whereas call/return paradigm is much more familiar and understandable. That's certainly why Python implements asymmetric model. And having both asymmetric and symmetric would quite confusing, especially that symmetric are more powerful and asymmetric can be easily implemented in terms of symmetric using continuation-passing style. On the last occurrence of "easily" mere users of course start to run away, shouting that if they wanted to use Scheme, they'd have taken classes on it and used long ago.

So, the real problem with dichotomy you describe above is not technical, but rather educational/documentational. And the current approach asyncio takes is "you should not care if a coroutine yields to main loop, or how it is done". Actually, the current approach is to forbid and deny you knowledge how it is done, quoting Victor Stinner from another mail: "(A) asyncio coroutine in Python 3.4: use yield from, yield denied". So, just pretend that there's no yield, only yield from, problem solved. But people know there's yield - they knew it for long time before "yield from". And there're valid usages for yield in a coroutine, like implementing your, application-level generator/generation. Currently, any generation ability is usurped by asyncio's main loop.

Much better approach IMHO is given in David Beazley's presentations on generators and coroutines, http://www.dabeaz.com/generators/ . He says that coroutines provided by framework are essentially "system calls". And that's why you don't want to know how they work, and shouldn't care - because users usually don't care how OS kernel implements system calls while sitting in the web browser. But if you want, you can, and you will discover that they're implemented by yield'ing objects of a special class. That's why you're suggested not to use yield's in coroutines - because if you want to catch yours, application-level, yields, you may also get any time a system yield object. You would need to expect such possibility, filter such yields and pass them up (re-yield). But there's no forbidden magic in all that, and understanding that helps a lot IMHO.

-- Best regards, Paul mailto:pmiscml at gmail.com

More information about the Python-Dev mailing list