[Python-Dev] Making python C-API thread safe (try 2) (original) (raw)

Harri Pesonen fuerte at sci.fi
Thu Sep 11 20:53:10 EDT 2003

Sorry about the previous post! :-)

Background: I'm new to Python. I just embedded Python into a multi-threaded application, and extended Python by callbacks as well. During this process I realised that Python is not internally as beautiful as externally. It is not thread safe. This surprised me, because Python is a quite young programming language. I started thinking how to fix this, how to make Python thread safe. Currently Python does not support multiprocessor machines or hyperthreading processors. It is only a matter of time when this has to be fixed, better soon than later.

How to make C-API thread safe: It's quite simple, in fact. Thread should first call

PyThreadState* Py_NewInterpreter()

and use this thread state pointer in every following API call. This thread state pointer should point to a memory structure, that holds everything that Python interpreter needs. There should be no global variables, not even None, except

const char* Py_GetVersion()

and such. So every Python function should have this thread state as the first argument:

int PyRun_SimpleString(PyThreadState*tstate, char *command)

and so on. Also callbacks should have it, about every API function.

Advantages: The following functions would not be needed any more, it would make the API much simpler:

void PyEval_InitThreads( ) void PyEval_AcquireLock( ) void PyEval_ReleaseLock( ) void PyEval_AcquireThread( PyThreadState *tstate) void PyEval_ReleaseThread( PyThreadState tstate) PyThreadState PyEval_SaveThread( ) void PyEval_RestoreThread( PyThreadState tstate) Py_BEGIN_ALLOW_THREADS Py_END_ALLOW_THREADS Py_BLOCK_THREADS Py_UNBLOCK_THREADS PyInterpreterState PyInterpreterState_New( ) void PyInterpreterState_Clear( PyInterpreterState *interp) void PyInterpreterState_Delete( PyInterpreterState interp) PyThreadState PyThreadState_New( PyInterpreterState *interp) void PyThreadState_Clear( PyThreadState tstate) void PyThreadState_Delete( PyThreadState tstate) PyThreadState PyThreadState_Get( ) PyThreadState PyThreadState_Swap( PyThreadState *tstate)

The current documentation says: "In order to support multi-threaded Python programs, the interpreter regularly releases and reacquires the lock -- by default, every 100 bytecode instructions". This would not be needed anymore. It would make the interpreter faster.

Making this change would take a couple of weeks, if the job would be divided between modules to different persons. The change is trivial, but it would have to be made everywhere.

Disadvantages: Everything old would break. Thread safe Python should have a version number significantly higher, like Python 3. Old applications would work with old Python versions, with old Python DLLs, but they would need to be rewritten for the new API. It would be a trivial task, but still.

Disclaimer: I already got mail saying: "

This issue gets discussed regularly on comp.lang.python; python-dev is for discussion of specific changes to the Python language and libraries. Until you are familiar with the way threading works in Python and have some concrete proposal to make, please keep the discussion on c.l.py."

The current situation is very unfortunate, having a single threaded implementation in the modern computer age. I have a concrete proposal, which would break everything :-) but I think that is the only solution.

Thanks for reading,

Harri

More information about the Python-Dev mailing list