[Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted (original) (raw)

Guido van Rossum guido at python.org
Fri Jan 4 23:59:40 CET 2013

On Fri, Jan 4, 2013 at 2:38 PM, Dustin Mitchell <djmitche at gmail.com> wrote:

As the maintainer of a pretty large, complex app written in Twisted, I think this is great. I look forward to a future of being able to select from a broad library of async tools, and being able to write tools that can be used outside of Twisted.

Thanks. Me too. :-)

Buildbot began, lo these many years ago, doing a lot of things in memory on on local disk, neither of which require asynchronous IO. So a lot of API methods did not originally return Deferreds. Those methods are then used by other methods, many of which also do not return Deferreds. Now, we want to use a database backend, and parallelize some of the operations, meaning that the methods need to return a Deferred. Unfortunately, that requires a complete tree traversal of all of the methods and methods that call them, rewriting them to take and return Deferreds. There's no "halfway" solution. This is a little easier with generators (@inlineCallbacks), since the syntax doesn't change much, but it's a significant change to the API (in fact, this is a large part of the reason for the big rewrite for Buildbot-0.9.x).

I bring all this up to say, this PEP will introduce a new "kind" of method signature into standard Python, one which the caller must know, and the use of which changes the signature of the caller. That can cause sweeping changes, and debugging those changes can be tricky.

Yes, and this is the biggest unproven point of the PEP. (The rest is all backed by a decade or more of experience.)

Two things can help:

First, yield from somemeth() should work fine even if somemeth is not a coroutine function, and authors of async tools should be encouraged to use this form to assist future-compatibility. Second, somemeth() without a yield should fail loudly if somemeth is a coroutine function. Otherwise, the effects can be pretty confusing.

That would be nice. But the way yield from and generators work, that's hard to accomplish without further changes to the language -- and I don't want to have to change the language again (at least not immediately -- maybe in a few releases, after we've learned what the real issues are). The best I can do for the first requirement is to define @coroutine in a way that if the decorated function isn't a generator, it is wrapped in one. For the second requirement, if you call somemeth() and ignore the result, nothing happens at all -- this is indeed infuriating but I see no way to change this.(*) If you use the result, well, Futures have different attributes than most other objects so hopefully you'll get a loud AttributeError or TypeError soon, but of course if you pass it into something else which uses it, it may still be difficult to track. Hopefully these error messages provide a hint:

f.foo Traceback (most recent call last): File "", line 1, in AttributeError: 'Future' object has no attribute 'foo' f() Traceback (most recent call last): File "", line 1, in TypeError: 'Future' object is not callable

(*) There's a heavy gun we might use, but I would make this optional, as a heavy duty debugging mode only. @coroutine could wrap generators in a lightweight object with a del method and an iter method. If del is called before iter is ever called, it could raise an exception or log a warning. But this probably adds too much overhead to have it always enabled.

In http://code.google.com/p/uthreads, I accomplished the latter by taking advantage of garbage collection: if the generator is garbage collected before it's begun, then it's probably not been yielded. This is a bit gross, but good enough as a debugging technique.

Eh, yeah, what I said. :-)

On the topic of debugging, I also took pains to make sure that tracebacks looked reasonable, filtering out scheduler code[1]. I haven't looked closely at Tulip to see if that's a problem. Most of the "noise" in the tracebacks came from the lack of 'yield from', so it may not be an issue at all.

One of the great advantages of using yield from is that the tracebacks automatically look nice.

Dustin

[1] http://code.google.com/p/uthreads/source/browse/trunk/uthreads/core.py#253

-- --Guido van Rossum (python.org/~guido)

More information about the Python-Dev mailing list