[Python-Dev] Pythonic concurrency (original) (raw)

Bruce Eckel BruceEckel-Python3234 at mailblocks.com
Thu Oct 6 22:06:37 CEST 2005

This does look quite fascinating, and I know there's a lot of really interesting work going on at the BBC now -- looks like some really pioneering stuff going on with respect to TV show distribution over the internet, new compression formats, etc.

So yes indeed, this is quite high on my list to research. Looks like people there have been doing some interesting work.

Right now I'm just trying to cast a net, so that people can put in ideas, for when the Java book is done and I can spend more time on it.

Thursday, October 6, 2005, 1:54:56 PM, Michael Sparks wrote:

Hi Bruce,

On Thursday 06 October 2005 18:12, Bruce Eckel wrote:

Although I hope our conversation isn't done, as he suggests! ... At some point when more ideas have been thrown about (and TIJ4 is done) I hope to summarize what we've talked about in an article.

I don't know if you saw my previous post[1] to python-dev on this topic, but Kamaelia is specifically aimed at making concurrency simple and easy to use. Initially we were focussed on using scheduled generators for co-operative CSP-style (but with buffers) concurrency. [1] http://tinyurl.com/dfnah, http://tinyurl.com/e4jfq

We've tested the system so far on 2 relatively inexperienced programmers (as well as experienced, but the more interesting group is novices). The one who hadn't done much programming at all (a little bit of VB, pre-university) actually fared better IMO. This is probably because concurrency became part of his standard toolbox of approaches.

I've placed the slides I've produced for Euro OSCON on Kamaelia here: * http://cerenity.org/KamaeliaEuroOSCON2005.pdf

The corrected URL for the whitepaper based on work now 6 months old (we've come quite a way since then!) is here: * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml

Consider a simple server for sending text (generated by a user typing into the server) to multiple clients connecting to a server. This is a naturally concurrent problem in various ways (user interaction, splitting, listening for connections, serving connections, etc). Why is that interesting to us? It's effectively a microcosm of how subtitling works. (I work at the BBC)

In Kamaelia this looks like this:

=== start === class ConsoleReader(threadedcomponent): def run(self): while 1: line = rawinput(">>> ") line = line + "\n" self.outqueues["outbox"].put(line)

Backplane("subtitles").activate() pipeline( ConsoleReader(), publishTo("subtitles"), ).activate() def subtitlesprotocol(): return subscribeTo("subtitles")

SimpleServer(subtitlesprotocol, 5000).run() === end ===

The ConsoleReader is threaded to allow the use of the naive way of reading from the input, whereas the server, backplane (a named splitter component in practice), pipelines, publishing, subscribing, splitting, etc are all single threaded co-operative concurrency.

A possible client for this text service might be:

pipeline( TCPClient("subtitles.rd.bbc.co.uk", 5000), Ticker(), ).run()

(Though that would be a bit bare, even if it does use pygame :)

The entire system is based around communicating generators, but we also have threads for blocking operations. (Though the entire network subsystem is non-blocking)

What I'd be interested in, is hearing how our system doesn't match with the goals of the hypothetical concurrency system you'd like to see (if it doesn't). The main reason I'm interested in hearing this, is because the goals you listed are ones we want to achieve. If you don't think our system matches it (we don't have process migration as yet, so that's one area) I'd be interested in hearing what areas you think are deficient.

However, the way we're beginning to refer to the project is to refer to just the component aspect rather than concurrency - for one simple reason - we're getting to stage where we can ignore /most/ concurrency issues(not all).

If you have any time for feedback, it'd be appreciated. If you don't I hope it's useful food for thought!

Best Regards,

Michael

Bruce Eckel http://www.BruceEckel.com mailto:BruceEckel-Python3234 at mailblocks.com Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e" Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel Subscribe to my newsletter: http://www.mindview.net/Newsletter My schedule can be found at: http://www.mindview.net/Calendar

More information about the Python-Dev mailing list