[Python-Dev] Making python C-API thread safe (try 2) (original) (raw)

Harri Pesonen fuerte at sci.fi
Thu Sep 11 21:47:49 EDT 2003

Phillip J. Eby wrote:

At 07:53 PM 9/11/03 +0300, Harri Pesonen wrote:

Sorry about the previous post! :-)

Background: I'm new to Python. I just embedded Python into a multi-threaded application, and extended Python by callbacks as well. During this process I realised that Python is not internally as beautiful as externally. It is not thread safe. This surprised me, because Python is a quite young programming language. I started thinking how to fix this, how to make Python thread safe. Currently Python does not support multiprocessor machines or hyperthreading processors. It is only a matter of time when this has to be fixed, better soon than later. How to make C-API thread safe: It's quite simple, in fact. Thread should first call PyThreadState* PyNewInterpreter() and use this thread state pointer in every following API call. This thread state pointer should point to a memory structure, that holds everything that Python interpreter needs. There should be no global variables, not even None, except This only makes individual Python interpreters thread-safe. That might certainly be useful for some embedded applications, but it does nothing for thread-safety within an interpreter. In other words, Python would still need a global interpreter lock, in order for threaded programs written in Python to behave properly. So, your solution would help only creators of multithreaded embedded applications, wherein each thread desired an isolated Python interpreter. Meanwhile, it would impose both a one-time and ongoing cost penalty on all other Python users, without giving them any benefit in return.

You are right, I thought only about making interpreters thread safe, thanks for pointing this out. But this is still needed, it is unavoidable in the long run.

What do you mean about the penalty? If you mean having the state pointer in stack in every call, I don't think so.

If we think about making Python threads thread-safe, within the interpreter, then I think that it should be left for the application deverloper, in principle. He should create "critical sections" and other locks when needed (I don't know how it is done currently). I can imagine that None and other values will cause problems, when two threads increment or decrement the reference count at the same time. Of course there is a simple solution for that, InterlockedIncrement and InterlockedDecrement in Win32 at least.

I can imagine that there are other problems with garbage collector, and so on. I don't want to go into details about this, I got my message through, I was almost losing sleep when I realised the current situation.

But my basic message is this: Python needs to be made thread safe. Making the individual interpreters thread safe is trivial, and benefits many people, and is a necessary first step; making threads within interpreter thread safe is possible as well, at least if you leave something for the developer, as you should, as you do in every other programming language as well.

Harri

More information about the Python-Dev mailing list