[Python-Dev] Tricky way of of creating a generator via a comprehension expression (original) (raw)
Yury Selivanov yselivanov.ml at gmail.com
Sun Nov 26 20:01:00 EST 2017
- Previous message (by thread): [Python-Dev] Tricky way of of creating a generator via a comprehension expression
- Next message (by thread): [Python-Dev] Tricky way of of creating a generator via a comprehension expression
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On Sun, Nov 26, 2017 at 6:51 PM, Guido van Rossum <guido at python.org> wrote:
On Sun, Nov 26, 2017 at 12:29 PM, Nathaniel Smith <njs at pobox.com> wrote: [..]
- async/await has associated thread-global state like sys.setcoroutinewrapper and sys.setasyncgenhooks. Generally async libraries assume that they own these, and arbitrarily weird things may happen if you have multiple async/await coroutine runners in same thread with no coordination between them.
The existence of these is indeed a bit unfortunate for this use case. I'm CC'ing Yury to ask him if he can think of a different way to deal with the problems that these are supposed to solve. For each, the reason they exist is itself an edge case -- debugging for the former, finalization for the latter. A better solution for these problem may also be important for situations where multiple event loops exist (in the same thread, e.g. running alternately). Maybe a context manager could be used to manage this state better?
Yeah, both of them are for solving edge cases:
sys.set_coroutine_wrapper() is a debug API: asyncio uses it to slightly enhance the warning about non-awaited coroutines by showing where they were created. Capturing traceback when we create every coroutine is expensive, hence we only do that in asyncio debug mode.
sys.set_asyncgen_hooks() is more important, we use it to let event loops finalize partially iterated and then abandoned asynchronous generators.
The rule of thumb is to get the previous coro-wrapper/asyncgen-hook, set your own (both are thread specific), and after you're done -- restore the saved old versions back. This should work fine in all use cases (even a trio event loop nested in an asyncio event loop). But nested coroutine loops is a sign of bad design, the nested loop will completely block the execution of the outer event loop, so situations like that should be avoided. Not to mention that debugging becomes much harder when you have complex nested coroutine runners.
If someone wants to run some async/await code without an event loop for educational purposes or to implement some pattern, they likely don't even need to use any of the above APIs.
For me the status quo is OK: asyncio/twisted/curio/trio use these APIs internally, and for them it shouldn't be a problem to use a couple low-level functions from sys.
That said I have a couple ideas:
We can simplify these low-level APIs by combining set_coroutine_wrapper() and set_asyncgen_hooks() into one function -- set_async_hooks(coroutine_wrapper=, firstiter=, etc). It could return a context-manager, so the following would be possible:
with sys.set_async_hooks(...): # run async code or an event loop
Another option is to design an API that would let you to stack your coro-wrapper/asyncgen-hooks, so that many coroutine runners can control coroutines/generators simultaneously. This sounds very complex to me though, and I haven't seen any compelling real-world use case that would require this kind of design.
sys.set_coroutine_wrapper() is marked as an experimental debug API; sys.set_asyncgen_hooks() is still provisional. So we can in theory change/improve both of them.
Yury
- Previous message (by thread): [Python-Dev] Tricky way of of creating a generator via a comprehension expression
- Next message (by thread): [Python-Dev] Tricky way of of creating a generator via a comprehension expression
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]