[Python-Dev] Pythonic concurrency (original) (raw)

Michael Sparks ms at cerenity.org
Thu Oct 6 21:54:56 CEST 2005

Hi Bruce,

On Thursday 06 October 2005 18:12, Bruce Eckel wrote:

Although I hope our conversation isn't done, as he suggests! ... At some point when more ideas have been thrown about (and TIJ4 is done) I hope to summarize what we've talked about in an article.

I don't know if you saw my previous post[1] to python-dev on this topic, but Kamaelia is specifically aimed at making concurrency simple and easy to use. Initially we were focussed on using scheduled generators for co-operative CSP-style (but with buffers) concurrency. [1] http://tinyurl.com/dfnah, http://tinyurl.com/e4jfq

We've tested the system so far on 2 relatively inexperienced programmers (as well as experienced, but the more interesting group is novices). The one who hadn't done much programming at all (a little bit of VB, pre-university) actually fared better IMO. This is probably because concurrency became part of his standard toolbox of approaches.

I've placed the slides I've produced for Euro OSCON on Kamaelia here:

The corrected URL for the whitepaper based on work now 6 months old (we've come quite a way since then!) is here:

Consider a simple server for sending text (generated by a user typing into the server) to multiple clients connecting to a server. This is a naturally concurrent problem in various ways (user interaction, splitting, listening for connections, serving connections, etc). Why is that interesting to us? It's effectively a microcosm of how subtitling works. (I work at the BBC)

In Kamaelia this looks like this:

=== start === class ConsoleReader(threadedcomponent): def run(self): while 1: line = raw_input(">>> ") line = line + "\n" self.outqueues["outbox"].put(line)

Backplane("subtitles").activate() pipeline( ConsoleReader(), publishTo("subtitles"), ).activate() def subtitles_protocol(): return subscribeTo("subtitles")

SimpleServer(subtitles_protocol, 5000).run() === end ===

The ConsoleReader is threaded to allow the use of the naive way of reading from the input, whereas the server, backplane (a named splitter component in practice), pipelines, publishing, subscribing, splitting, etc are all single threaded co-operative concurrency.

A possible client for this text service might be:

pipeline( TCPClient("subtitles.rd.bbc.co.uk", 5000), Ticker(), ).run()

(Though that would be a bit bare, even if it does use pygame :)

The entire system is based around communicating generators, but we also have threads for blocking operations. (Though the entire network subsystem is non-blocking)

What I'd be interested in, is hearing how our system doesn't match with the goals of the hypothetical concurrency system you'd like to see (if it doesn't). The main reason I'm interested in hearing this, is because the goals you listed are ones we want to achieve. If you don't think our system matches it (we don't have process migration as yet, so that's one area) I'd be interested in hearing what areas you think are deficient.

However, the way we're beginning to refer to the project is to refer to just the component aspect rather than concurrency - for one simple reason - we're getting to stage where we can ignore /most/ concurrency issues(not all).

If you have any time for feedback, it'd be appreciated. If you don't I hope it's useful food for thought!

Best Regards,

Michael

Michael Sparks, Senior R&D Engineer, Digital Media Group Michael.Sparks at rd.bbc.co.uk, http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP

This e-mail may contain personal views which are not the views of the BBC.

More information about the Python-Dev mailing list