[Python-Dev] Pythonic concurrency (original) (raw)

Steve Holden steve at holdenweb.com
Tue Oct 11 13:06:39 CEST 2005

Bruce Eckel wrote: [Bill Janssen]

Yes, there's a troublesome meme in the world: "threads are hard". They aren't, really. You just have to know what you're doing. But that begs the question, because there is a significant amount of evidence that when it comes to threads "knowing what you are doing" is hard to the point that people can think they do when they demonstrably don't! I would say that the troublesome meme is that "threads are easy." I posted an earlier, rather longish message about this. The gist of which was: "when someone says that threads are easy, I have no idea what they mean by it." I would suggest that the truth lies in the middle ground, and would say that "you can get yourself into a lot of trouble using threads without considering the subtleties". It's an area where anything but the most simplistic solutions are almost always wrong at some point.

Perhaps this means "threads in Python are easier than threads in other languages."

But I just finished a 150-page chapter on Concurrency in Java which took many months to write, based on a large chapter on Concurrency in C++ which probably took longer to write. I keep in reasonably good touch with some of the threading experts. I can't get any of them to say that it's easy, even though they really do understand the issues and think about it all the time. Because of that, they say that it's hard. So alright, I'll take the bait that you've laid down more than once, now. Perhaps you can go beyond saying that "threads really aren't hard" and explain the aspects of them that seem so easy to you. Perhaps you can give a nice clear explanation of cache coherency and memory barriers in multiprocessor machines? Or explain atomicity, volatility and visibility? Or, even better, maybe you can come up with a better concurrency model, which is what I think most of us are looking for in this discussion. The nice thing about Python threads (or rather threading.threads) is that since each thread is an instance it's relatively easy to ensure that a thread restricts itself to manipulating thread-local resources (i.e. instance members).

This makes it possible to write algorithms parameterized for the number of "worker threads" where the workers are taking their tasks off a Queue with entries generated by a single producer thread. With care, multiple producers can be used. More complex inter-thread communications are problematic, and arbitrary access to foreign-thread state is a nightmare (although the position has been somewhat alleviated by the introduction of threading.local).

Beyond the single-producer many-consumers model there is still plenty of room to shoot yourself in the foot. In the case of threads true sophistication is staying away from the difficult cases, an option which unfortunately isn't always available in the real world.

regards Steve

Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC www.holdenweb.com PyCon TX 2006 www.python.org/pycon/

More information about the Python-Dev mailing list