bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio') by 1st1 · Pull Request #13472 · python/cpython (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation38 Commits6 Checks0 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

This makes it easy to play with asyncio APIs with simply
using async/await in the REPL.

@asvetlov would you mind playing with this? Pull the branch, make sure to rebuild Python, and then ./python -m asyncio. I quite like this and I think it would be a valuable addition to asyncio 3.8.

https://bugs.python.org/issue37028

csabella, tirkarthi, winterjung, Jackenmen, daniil-konovalenko, dmerejkowsky, dizballanze, bsolomon1124, decaz, mhchia, and 2 more reacted with hooray emoji

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I test it (debian 9) and this is great!

Looks good, works fine!

I think unittests are needed.
./Lib/test_code_module.py can be used as an example of interactive console testing.

$ python -m asyncio
asyncio REPL

Python 3.8.0a4+ (heads/arepl-dirty:0efd6fb39b, May 20 2019, 19:17:43)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import itertools
>>> import asyncio
>>> async def ticker():
...     for i in itertools.count():
...         print(i)
...         await asyncio.sleep(1)
...
>>> asyncio.ensure_future(ticker())
<Task pending name='Task-1' coro=<ticker() running at <console>:1>>
>>> await asyncio.sleep(5)
# no output

You should be able to keep the same eventloop running right ?
Drawback : that would prevent people from running asyncio.run() from within this REPL.



if __name__ == '__main__':
console = AsyncIOInteractiveConsole(locals())

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the locals() here on purpose ? Seem to leak some values...

$ python -m asyncio
asyncio REPL

Python 3.8.0a4+ (heads/arepl-dirty:0efd6fb39b, May 20 2019, 19:17:43)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> code
<module 'code' from '/Users/bussonniermatthias/dev/cpython/Lib/code.py'>
>>> inspect
<module 'inspect' from '/Users/bussonniermatthias/dev/cpython/Lib/inspect.py'>
>>> sys
<module 'sys' (built-in)>
>>> console
<__main__.AsyncIOInteractiveConsole object at 0x109ab5be0>
>>> banner
'asyncio REPL\n\nPython 3.8.0a4+ (heads/arepl-dirty:0efd6fb39b, May 20 2019, 19:17:43) \n[Clang 9.0.0 (clang-900.0.39.2)] on darwin\nType "help", "copyright", "credits" or "license" for more information.\n'
>>>

And to extend on my comment, here is what I am suggesting as a behavior (but maybe we do that after beta1), where the background coroutines run when there are foreground ones:

>>> import asyncio
>>> import itertools
>>> async def ticker():
...    for i in itertools.count():
...        print(i)
...        await asyncio.sleep(1)
>>> asyncio.ensure_future(ticker())
<Task pending name='Task-2' coro=<ticker() running at <empty co_filename>:1>>
>>> await asyncio.sleep(5)
0
1
2
3
4

The drawback would be :

>>> asyncio.run(ticker())
...
Traceback (most recent call last):
  File "minirepl.py", line 31, in <module>
    asyncio.run(repl())
  [...snip]
  File "", line 1, in <module>
  File "/Users/bussonniermatthias/dev/cpython/Lib/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop

As a guideline here is one of the bug report we got when implementing this on IPython, it will give you an idea of the kind of stuff people will try in async-repl.

As a guideline here is one of the bug report we got when implementing this on IPython, it will give you an idea of the kind of stuff people will try in async-repl.

Ah, this is quite interesting. The loop is indeed not running in the background which might confuse some users.

@asvetlov Andrew what do you think? Should we run asyncio event loop in the main thread, and implement the REPL in another non-main thread?

No, REPL in non-main thread is a bad idea.
At least this solution is very different from my expectations: when I start an interactive session my thread is the main.

Yeah, I think pausing the event loop while waiting for input is going to give a frustrating experience. Also consider:

async def ticker(): ... while True: ... await asyncio.sleep(1) ... print("tick!") asyncio.create_task(ticker)

(Or more realistically, something like starting a server in a background task from the REPL.)

You could put the RPL parts in another thread and the E in the main thread, but then exiting may be tricky and control-C may act pretty wonky. You could put RPL in the main thread and E in another thread, not sure if that would help or not...

Oh hah, sorry I skimmed the thread and missed that @Carreau used that exact example already :-)

And the other general approach would be to write the whole REPL async, but that requires support for async stdio, which is complicated (see also).

REPL not in main thread is bad from experience. Many library expect to be in main.

There are two things in play here:
Starting/stoping the loop between each input:

def run():
      input()
      co = compile()
      asyncio.run_untin_complete(co)

Or blocking the loop when requesting input:

async def run():
       while True:
           input()
           co = compile()
           await Function(co)
asyncio.run(run())

I believe they have similar– but not quite identical behavior.
I think that blocking the loop between inputs is more reasonable, as at least you can await sleep(...) to get the bg tasks moving. Which current implementation does not allow.

There is also the possibility "let's not put that in 3.8"; it's small enough that it could be it's own package to iterate on – and well, it works already with IPython :-P

@njsmith

Yeah, I think pausing the event loop while waiting for input is going to give a frustrating experience.

Agree. I was super confused why @Carreau's examples with while True: print didn't work "as expected". It's important that less obvious examples like "starting a TCP server" would work, i.e. the server should actually accept connections even if there's no user input.

@asvetlov

No, REPL in non-main thread is a bad idea.

I assume you meant "event loop in non-main thread is a bad idea"?

@Carreau

REPL not in main thread is bad from experience. Many library expect to be in main.

Why? What libraries have to do with this?

There is also the possibility "let's not put that in 3.8"; it's small enough that it could be it's own package to iterate on – and well

I'm OK to iterate. If there are open issues we won't push this in 3.8. However if there's no actual iteration on design, I prefer to push things and let people use them. asyncio REPL isn't critical or dependable piece of software, we can fix/tweak it.

As for IPython -- I'm glad that it exists, but I still can't get accustomed to it and always want a quick built-in asyncio repl to test things.

@ALL: I've pushed an updated implementation that runs the REPL in a separate thread, keeping an asyncio event loop always running in the main thread. Seems to work just fine, please try it out.

Some feedback:

>>> import asyncio

>>> async def f():
...     while(True):
...         print('.')
...         await asyncio.sleep(1)
... 
>>> await f()
.
.
^CTraceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 434, in result
    raise CancelledError()
concurrent.futures._base.CancelledError

Traceback for keyboard interrupt is confusing a little.
Should I see it if press Ctrl+C to break long-running await func()?

>>> def g():
...     while True:
...         pass
... 
>>> g()
^CTraceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 436, in result
    return self.__get_result()
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/andrew/projects/cpython/Lib/asyncio/__main__.py", line 29, in callback
    coro = func()
  File "<console>", line 1, in <module>
  File "<console>", line 3, in g
KeyboardInterrupt

The same question. Maybe REPL can hide it's own part of stack traceback?

The worst part:

>>> t = asyncio.create_task(f())
.
>>> .
.
asd.
sdsd.
s.
d.
.
.
.
sds.
.
[1]    30723 quit (core dumped)  ./python -m asyncio

I've started a task for f() function from the first snippet.

First, dots printed by the task are mixed with my typing. It makes editing REPL string almost impossible.

Also, pressing Ctrl+C when editing does nothing.
In the standard Python REPL it stops editing, prints KeyboardInterrupt without traceback and opens a fresh edit line.

In general, not bad at all if mentioned inconveniences can be fixed.

>>> import asyncio

>>> async def f():
...     pass
... 
>>> asyncio.run(f())
Traceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 436, in result
    return self.__get_result()
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/andrew/projects/cpython/Lib/asyncio/__main__.py", line 29, in callback
    coro = func()
  File "<console>", line 1, in <module>
  File "/home/andrew/projects/cpython/Lib/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop

The problem was mentioned by @Carreau
Not sure if asyncio specific loop needs to allow asyncio.run() (and family like loop.run_until_complete()) calls but the behavior worth to be mentioned.

Some feedback:

Thanks for trying it out :)

Traceback for keyboard interrupt is confusing a little.

Both ^C outputs can be fixed.

The worst part:

This is tricky. I guess the only answer I have to this is "don't print in background tasks".

The exact behaviour you see is easy to replicate in the standard Python repl just by spawning a background thread that prints. Since this is OK for the standard repl, I assume we can dismiss this usability annoyance for the asyncio one.

The problem was mentioned by @Carreau
Not sure if asyncio specific loop needs to allow asyncio.run() (and family like loop.run_until_complete()) calls but the behavior worth to be mentioned.

Yeah. I think we should document it, because allowing both asyncio.run and await a_thing in asyncio REPL is just not possible.

The exact behaviour you see is easy to replicate in the standard Python repl just by spawning a background thread that prints. Since this is good for the standard repl, I assume we can dismiss this for the asyncio one.

Agree.

Yeah. I think we should document it, because allowing both asyncio.run and await a_thing in asyncio REPL is just not possible.

Agree again.

>>> import asyncio

>>> async def f():
...     pass
... 
>>> asyncio.run(f())
Traceback (most recent call last):
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 436, in result
    return self.__get_result()
  File "/home/andrew/projects/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/home/andrew/projects/cpython/Lib/asyncio/__main__.py", line 29, in callback
    coro = func()
  File "<console>", line 1, in <module>
  File "/home/andrew/projects/cpython/Lib/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
>>> 
exiting asyncio REPL...
sys:1: RuntimeWarning: coroutine 'f' was never awaited

Please take a look on the last line. It is printed after the REPL exit.

Please take a look on the last line. It is printed after the REPL exit.

What's wrong with it? The coroutine wasn't actually awaited. I don't think that silencing this warning in the REPL is a good idea. Here's an example interaction where having this warning will hint to the user that they should probably add an "await":

>>> async def foo(): print('hello')
>>> foo()
<coroutine ...>
>>> foo()
<coroutine ...>
sys:1: RuntimeWarning: coroutine 'foo' was never awaited

One other possibility is to filter this warning out while the REPL is exiting...

I mean it is printed after exiting asyncio REPL... message.
I would like to see the print before the exit message. Pretty sure it requires a very small change :)

I love that debug mode just works!

andrew•~/projects/cpython(1st1-arepl⚡)» ./python -X dev -m asyncio     
>>> import asyncio

>>> import time
>>> async def f():
...     time.sleep(1)
... 
>>> await f()
Executing <Task finished name='Task-1' coro=<<module>() done, defined at <console>:1> result=None created at /home/andrew/projects/cpython/Lib/asyncio/__main__.py:39> took 1.002 seconds

This makes it easy to play with asyncio APIs with simply using async/await in the REPL.

@asvetlov The latest update should address your comments.

Why? What libraries have to do with this?

from a top of my head i've seem people using open-cv which seemed to not like bing imported in not-main-thread. I'll see if I can your new version to crash :-)

Yeah. I think we should document it, because allowing both asyncio.run and await a_thing in asyncio REPL is just not possible.

Well, it is kinda-half-possible; IPython does it; if there are no asyncio task running, and no top-level await we just use a dumb-coroutine iterator. But that's a detail; I agree it's overkill to try-to get both to work.

@Carreau seems like you fall into the same trap as me.
REPL is for Read–Eval–Print Loop
The current version of PR does Read part in a thread but Eval and Print is executed inside the main thread.
So everything that you are typing is executed in the main thread context.

@njsmith mentioned this way:

You could put the RPL parts in another thread and the E in the main thread, but then exiting may be tricky and control-C may act pretty wonky. You could put RPL in the main thread and E in another thread, not sure if that would help or not...

Looks like "wonky" approach works pretty well.

pasting multiple lines misprints the ...:

$ python -m asyncio
asyncio REPL 3.8.0a4+ (remotes/origin/pr/13472:7a973fb6ec, May 23 2019, 13:31:27)
[Clang 4.0.1 (tags/RELEASE_401/final)] on darwin

Use "await" directly instead of asyncio.run().
Type "help", "copyright", "credits" or "license" for more information.

>>> import asyncio

>>> async def ticker():
    while True:
        await asyncio.sleep(1)
        print("tick!")... ... ...

Nevermind, this is my system that have an issue with readline.

@asvetlov Please feel free to review this. If you green light it I'll merge and add tests later.

1st1 changed the title~~Implement asyncio REPL (activated via 'python -m asyncio')~~ bpo-37028: Implement asyncio REPL (activated via 'python -m asyncio')