[Python-Dev] Cloning threading.py using proccesses (original) (raw)

Josiah Carlson jcarlson at uci.edu
Tue Oct 10 17:58:47 CEST 2006

"Richard Oudkerk" <r.m.oudkerk at googlemail.com> wrote:

I am not sure how sensible the idea is, but I have had a first stab at writing a module processing.py which is a near clone of threading.py but uses processes and sockets for communication. (It is one way of avoiding the GIL.)

On non-windows platforms, you should check on unix domain sockets, I've found they can run a couple times faster than standard sockets on the local machine. And if you are using fork or a variant of subprocess to start processes on linux or Windows, you should consider using pipes, they can be competitive with sockets (though using a bunch on Windows can be a pain).

I have tested it on unix and windows and it seem to work pretty well. (Getting round the lack of os.fork on windows is a bit awkward.) There is also another module dummyprocessing.py which has the same api but is just a wrapper round threading.py.

Queues, Locks, RLocks, Conditions, Semaphores and some other shared objects are implemented. People are welcome to try out the tests in testprocessing.py contained in the zipfile. More information is included in the README file. As a quick example, the code

[snip]

Looks interesting. Maybe it would become clearer with docs (I hope you've written some). Right now there is a difference, and it is basically that there are tokens and proxies, which could confuse some users.

Presumably with this library you have created, you have also written a fast object encoder/decoder (like marshal or pickle). If it isn't any faster than cPickle or marshal, then users may bypass the module and opt for fork/etc. + XML-RPC; which works pretty well and gets them multi-machine calling, milti-language interoperability, and some other goodies, though it is a bit slow in terms of communication.

More information about the Python-Dev mailing list