[Python-Dev] Experimenting with STM on CPython (original) (raw)

Armin Rigo arigo at tunes.org
Wed Apr 11 13:47:42 CEST 2012

Hi all,

This is an update on the (so far PyPy-only) project of adding "Automatic Mutual Exclusion" to Python, via STM (Software Transactional Memory). For the motivation, see here:

http://morepypy.blogspot.com/2012/03/call-for-donations-for-software.html """The point is that [with STM/AME] your program is always correct, and can be tweaked to improve performance. This is the opposite from what explicit threads and locks give you, which is a performant program which you need to tweak to remove bugs. Arguably, this approach is the reason for why you use Python in the first place :-)"""

The update is: I now believe that it might be (reasonably) possible to apply the same techniques to CPython, and not only to PyPy. For now I am experimenting with applying them in a simple CPython-like interpreter. If it works, it might end up as a patch to the core parts of CPython. The interesting property is that it would still be able to run unmodified C extension modules --- the Python code gets the benefits of multi-core STM/AME only if it involves only the patched parts of the C code, but in all cases it still works correctly, falling back to single-core usage.

I did not try to hack CPython so far, but only this custom interpreter for a Lisp language, whose implementation should be immediately familiar to anyone who knows CPython C code: https://bitbucket.org/arigo/duhton . The non-standard built-in function is "transaction", which schedules a transaction to run later (see test/test_transaction.py).

The code contains the necessary tweaks to reference counting, and seems to work on all examples, but leaks some of the objects so far. Fixing this directly might be possible, but I'm not sure yet (it might require interaction with the cycle-detecting GC of CPython). Moreover the performance hit is well below 2x, more like 20%.

If anyone is interested, I could create a dedicated mailing list in order to discuss this in more details. From experience I would think that this has the potential to become a Psyco-like experiment, but unlike 10 years ago, today I'm not ready any more to dive completely alone into a project of that scale :-)

A bientôt,

Armin.

More information about the Python-Dev mailing list