[Python-ideas] [Python-Dev] Python needs a standard asynchronous return object (original) (raw)

Guido van Rossum guido at python.org
Sun Sep 12 17:49:56 CEST 2010

On Sun, Sep 12, 2010 at 4:03 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

On Sat, 11 Sep 2010 19:26:50 -0700 Guido van Rossum <guido at python.org> wrote:

But thinking about this more I don't know that it will be easy to mix PEP 3148, which is solidly thread-based, with a PEP 342 style scheduler (whether or not the PEP 380 enhancements are applied, or even PEP 3152). I'm not sure why. The implementation is certainly thread-based, but functions such as wait(fs, timeout=None, returnwhen=ALLCOMPLETED) could be implemented in termes of a single-threaded event loop / job scheduler.

Sure, but the tricky thing is to make it pluggable so that PEP 3148 and Twisted and other frameworks can use it all together, and a single call will accept a mixture of Futures.

I also worry that "impure" code will have a hard time -- e.g. when mixing generator-based coroutines and thread-based futures, it would be quite bad if a coroutine called .result() on a Future or the .wait() function instead of yielding to the scheduler.

Actually, Twisted has a similar primitive in DeferredList, although more powerful since the DeferredList itself is a Deferred, and can therefore be further combined, etc.:

http://twistedmatrix.com/documents/10.0.0/api/twisted.internet.defer.DeferredList.html

This sounds similar to the way you can create derived futures in Java.

And comparing the blog's examples to PEP 3148, I find Twisted's terminology rather confusing compared to the PEP's clean Futures API (where IMO you can ignore almost everything except result()). Well, apart from the API which may be considered a taste issue (I have used Deferreds long before I heard about Futures, so perhaps I'm a bit biased),

I heard of Deferred long before PEP 3148 was even conceived, but I find Twisted's terminology terribly confusing while I find the PEP's names easy to understand.

the following API doc in PEP 3148 shows that the Future model of callbacks is less rich than Twisted's:

“adddonecallback(fn)  Attaches a callable fn to the future that will be called when the  future is cancelled or finishes running. fn will be called with the  future as its only argument.  Added callables are called in the order that they were added and  are always called in a thread belonging to the process that added  them. If the callable raises an Exception then it will be logged  and ignored. If the callable raises another BaseException then  behavior is not defined.” With Twisted Deferreds, when a callback or errback raises an error, its exception isn't “logged and ignored”, it is passed to the remaining errback chain attached to the Deferred. This is part of what makes Deferreds more complicated to understand, but it also makes them more powerful.

Yeah, please do explain why Twisted has so much machinery to handle exceptions?

ISTM that the main difference is that add_done_callback() isn't meant for callbacks that return a value. So then the exceptions that might be raised are kind of "out of band". For any API that returns a value I agree that raising an exception should be handled -- but in the PEP 342 world we can do that by passing exceptions back into coroutine using throw(), so no separate "success" and "failure" callbacks are needed.

Another key point is that a callback can itself return another Deferred object, in which case the next callback (or errback, in case of error) will be called only once the other Deferred produces a result. This is all handled transparently and you can freely mix callbacks that immediately return a value, and callbacks that return a Deferred whose final value will be available later. And the other Deferred can have its own callback/errback chain, etc.

Yeah, that is part of what makes it so utterly confusing. PEP 380 supports a similar thing but much cleaner, without ever using callbacks.

(just for the record, the “final value” of a Deferred is the value returned by the last callback in the chain)

I think the main reason, though, that people find Deferreds inconvenient is that they force you to think in terms of asynchronicity (well, almost: you can of course hack yourself some code which blocks until a Deferred has a value, but it's extremely discouraged). They would like to have officially supported methods like result(timeout=None) which make simple things (like quick scripts to fetch a bunch of URLs) simpler. Twisted is generally used for server applications where such code is out of question (in an async model, that is).

Actually I think the main reason is historic: Twisted introduced callback-based asynchronous (thread-less) programming when there was no alternative in Python, and they invented both the mechanisms and the terminology as they were figuring it all out. That is no mean feat. But with PEP 342 (generator-based coroutines) and especially PEP 380 (yield from) there is an alternative, and while Twisted has added APIs to support generators, it hasn't started to deprecate its other APIs, and its terminology becomes hard to follow for people (like me, frankly) who first learned this stuff through PEP 342.

-- --Guido van Rossum (python.org/~guido)

More information about the Python-ideas mailing list