[Python-Dev] [PEP 3148] futures - execute computations asynchronously (original) (raw)

P.J. Eby pje at telecommunity.com
Sun Mar 7 18:57:31 CET 2010

At 08:39 AM 3/7/2010 -0800, Jeffrey Yasskin wrote:

On Sun, Mar 7, 2010 at 7:48 AM, P.J. Eby <pje at telecommunity.com> wrote: > At 02:49 PM 3/7/2010 +1000, Nick Coghlan wrote: >> >> P.J. Eby wrote: >> > (Personally, I think it would be better to just drop the ambitious title >> > and scope, and go for the "nice task queue" scope. Â I imagine, too, that >> > in that case Jean-Paul wouldn't need to worry about it being raised as a >> > future objection to Deferreds or some such getting into the stdlib.) >> >> This may be a terminology thing - to me futures are just a nice way to >> handle farming tasks out to worker threads or processes. You seem to see >> them as something more comprehensive than that. > > Actual futures are, yes. Â Specifically, futures are a mechanism for > asynchronous computation, whereas the PEP seems to be all about > synchronously managing parallel tasks. Â That's a huge difference. > > Technically, the things in the PEP (and by extension, Java's futures) match > the letter of the definition of a future, but not (IMO) the spirit. Â There's > no clean way to compose them, and at base they're more about parallelism > than asynchrony.

Do you have an example of a language or library that uses the term "future" to refer to what you're talking about? I'm curious to see what it looks like.

The wikipedia page menetioned earlier in the discussion has several examples. Twisted's Deferreds are probably the best example of such a system in Python, though, as far as I know. The features proposed in the PEP are basically akin to Twisted's deferToThread(), i.e. "return a future for running this code in some other thread".

However, you could easily make your own thread or process pool for executing tasks, using Deferreds as your "future" objects; deferToThread() is not particularly special, it's just one possible "executor", to use the PEP's terminology. Likewise, deferreds are not limited to representing parallel task execution - they can also represent asynchronous events such as incoming I/O or a waited-for mouse click.

Deferreds allow composing calculation by stacking callbacks (in continuation-passing style), but they're most Pythonic (IMO) when used with a coroutine wrapping system like Eventlet or my earlier peak.events framework, so that you can use them in "yield" expressions. That way you can write synchronous-looking code that nonetheless executes asynchronously.

Twisted uses Deferreds extensively, both for I/O and IPC, and of course for database access. However, the 'defer' module itself is conceptually distinct from both the I/O system, event loop, and threads: it's a largely-generic framework for managing asynchronous computation. Parallel task queueing is simply one possible application of Deferreds; their actual purpose is to provide a common API for working with values that aren't yet known.

That is to say, futures.

In contrast, the PEP manages only a very limited sort of "future" - the completion of a single task, which must be synchronously waited for. If one "future" in the PEP needs to wait for another, you tie up that thread or process in a busy loop... which rather limits how much parallelism you can have.

Deferreds also allow you to write code that doesn't care whether the value has been determined or not... you can simply stack on what code you want to have run when the value is available, and it runs immediately if the value is there.

Likewise, a deferred is not tied to the execution of a single piece of code - it's more of a rendezvous or synchronization point, that can be given a value to "fire" with. (In contrast, the PEP's "futures" are tightly coupled to a task execution model: they have a notion of running and being cancelled, for example.)

For these reasons, I'd say that Deferreds are much more deserving of the title "futures", especially since there is no need to call a simple task queue a "futures" system. (Unless you're Java, I suppose, in which case you might want the name for marketing reasons. ;-) But in Python I assume we'd like to call a spade a spade, and a task queue a task queue.)

More information about the Python-Dev mailing list