[Python-Dev] Eureka! (Re: test_fork fails --with-thread) (original) (raw)

Charles G Waldman cgw@fnal.gov
Fri, 18 Aug 2000 23:31:06 -0500 (CDT)

Last month there was a flurry of discussion, around

http://www.python.org/pipermail/python-dev/2000-July/014208.html

about problems arising when combining threading and forking. I've been reading through the python-dev archives and as far as I can tell this problem has not yet been resolved.

Well, I think I understand what's going on and I have a patch that fixes the problem.

Contrary to some folklore, you can use fork() in threaded code; you just have to be a bit careful about locks...

Rather than write up a long-winded explanation myself, allow me to quote:

from "man pthread_atfork":

   ... recall that fork(2) duplicates the whole memory space,
   including mutexes in their current locking state, but only the
   calling thread: other threads are not running in the child
   process. Thus, if a mutex is locked by a thread other than
   the thread calling fork, that  mutex  will  remain  locked
   forever in the child process, possibly blocking the execu-
   tion of the child process.

and from http://www.lambdacs.com/newsgroup/FAQ.html#Q120

Q120: Calling fork() from a thread

Can I fork from within a thread ?

Absolutely.

If that is not explicitly forbidden, then what happens to the other threads in the child process ?

There ARE no other threads in the child process. Just the one that forked. If your application/library has background threads that need to exist in a forked child, then you should set up an "atfork" child handler (by calling pthread_atfork) to recreate them. And if you use mutexes, and want your application/library to be "fork safe" at all, you also need to supply an atfork handler set to pre-lock all your mutexes in the parent, then release them in the parent and child handlers. Otherwise, ANOTHER thread might have a mutex locked when one thread forks -- and because the owning thread doesn't exist in the child, the mutex could never be released. (And, worse, whatever data is protected by the mutex is in an unknown and inconsistent state.)

Below is a patch (I will also post this to SourceForge)

Notes on the patch:

  1. I didn't make use of pthread_atfork, because I don't know how portable it is. So, if somebody uses "fork" in a C extension there will still be trouble.

  2. I'm deliberately not cleaning up the old lock before creating the new one, because the lock destructors also do error-checking. It might be better to add a PyThread_reset_lock function to all the thread_*.h files, but I'm hesitant to do this because of the amount of testing required.

Patch:

Index: Modules/signalmodule.c

RCS file: /cvsroot/python/python/dist/src/Modules/signalmodule.c,v retrieving revision 2.53 diff -c -r2.53 signalmodule.c *** Modules/signalmodule.c 2000/08/03 02:34:44 2.53 --- Modules/signalmodule.c 2000/08/19 03:37:52 *************** *** 667,672 **** --- 667,673 ---- PyOS_AfterFork(void) { #ifdef WITH_THREAD + PyEval_ReInitThreads(); main_thread = PyThread_get_thread_ident(); main_pid = getpid(); #endif Index: Parser/intrcheck.c

RCS file: /cvsroot/python/python/dist/src/Parser/intrcheck.c,v retrieving revision 2.39 diff -c -r2.39 intrcheck.c *** Parser/intrcheck.c 2000/07/31 15:28:04 2.39 --- Parser/intrcheck.c 2000/08/19 03:37:54 *************** *** 206,209 **** --- 206,212 ---- void PyOS_AfterFork(void) { + #ifdef WITH_THREAD + PyEval_ReInitThreads(); + #endif } Index: Python/ceval.c

RCS file: /cvsroot/python/python/dist/src/Python/ceval.c,v retrieving revision 2.191 diff -c -r2.191 ceval.c *** Python/ceval.c 2000/08/18 19:53:25 2.191 --- Python/ceval.c 2000/08/19 03:38:06

*** 142,147 **** --- 142,165 ---- Py_FatalError("PyEval_ReleaseThread: wrong thread state"); PyThread_release_lock(interpreter_lock); }

+