[Python-Dev] microthreading vs. async io (original) (raw)

dustin at v.igoro.us dustin at v.igoro.us
Thu Feb 15 06:00:31 CET 2007

I've steered clear of this conversation for a while, because it drifted a little bit off my original topic. But I did want to straighten a little bit of terminology out, if only to resolve some of my own confusion over all the hubbub. I don't pretend to define the words others use; these definitions are mine, and apply to what I write below.

cooperative multitasking: Dynamically executing multiple tasks in a single thread; comes in two varieties:

continuations: Breaking tasks into discrete "chunks", and passing references those chunks around as a means of scheduling.

microtreading: Exploiting language features to use cooperative multitasking in tasks that "read" like they are single-threaded.

asynchronous IO: Performing IO to/from an application in such a way that the application does not wait for any IO operations to complete, but rather polls for or is notified of the readiness of any IO operations.

Twisted is, by the above definitions, a continuation-based cooperative multitasking library that includes extensive support for asynchronous IO, all the way up the network stack for an impressive array of protocols. It does not itself implement microthreading, but Phillip provided a nice implementation of such on top of Twisted[1].

Asyncore only implements asynchronous IO -- any "tasks" performed in its context are the direct result of an IO operation, so it's hard to say it implements cooperative multitasking (and Josiah can correct me if I'm wrong, but I don't think it intends to).

Much of the discussion here has been about creating a single, unified asynchronous IO mechanism that would support any kind of cooperative multitasking library. I have opinions on this ($0.02 each, bulk discounts available), but I'll keep them to myself for now.

Instead, I would like to concentrate on producing a small, clean, consistent, generator-based microthreading library. I've seen several such libraries (including the one in PEP 342, which is fairly skeletal), and they all work almost the same way, but differ in, for example, the kinds of values that can be yielded, their handling of nested calls, and the names for the various "special" values one can yield.

That similar mouldes are being written repeatedly, and presumably applications and frameworks are being built on top of those modules, seems to me to suggest a new "standard" implementation should be added to the standard library.

I realize that I'm all talk and no code -- I've been busy, but I hope to rectify the imbalance tonight.

Dustin

[1] http://mail.python.org/pipermail/python-dev/2007-February/071076.html

More information about the Python-Dev mailing list