Issue 7316: Add a timeout functionality to common locking operations (original) (raw)

diff -r 8089902215a5 Doc/library/_thread.rst --- a/Doc/library/_thread.rst   Fri Jan 08 18:54:23 2010 +0100 +++ b/Doc/library/_thread.rst   Fri Jan 08 20:33:54 2010 +0100 @@ -103,18 +103,29 @@ It defines the following constant and fu    Availability: Windows, systems with POSIX threads.

+.. data:: TIMEOUT_MAX

Above here, it says, "It defines the following constant and functions:", which should be updated to "constants" now that there are 2.

Do we want to document what this value is likely to be? Or guarantee that it's at least 2000?

I believe we can support arbitrary values here, subject to floating point rounding errors, by calling lock-with-timeout in a loop. I'm not sure whether that's a good idea, but it fits better with python's arbitrary-precision ints.

-.. method:: lock.acquire([waitflag]) +.. method:: lock.acquire(waitflag=1, timeout=-1)

   Without the optional argument, this method acquires the lock unconditionally, if

Since there are now 2 optional arguments, this needs to be updated.

   necessary waiting until it is released by another thread (only one thread at a    time can acquire a lock --- that's their reason for existence). If the integer    waitflag argument is present, the action depends on its value: if it is zero,    the lock is only acquired if it can be acquired immediately without waiting,

You might mention that "lock.acquire(timeout=0)" is equivalent to "lock.acquire(waitflag=0)", and that a missing or negative timeout causes an unbounded wait.

 .. method:: lock.release() diff -r 8089902215a5 Doc/library/threading.rst --- a/Doc/library/threading.rst Fri Jan 08 18:54:23 2010 +0100 +++ b/Doc/library/threading.rst Fri Jan 08 20:33:54 2010 +0100 @@ -155,6 +155,16 @@ This module defines the following functi    Availability: Windows, systems with POSIX threads.

+This module also defines the following constant: + +.. data:: TIMEOUT_MAX +

 The design of this module is loosely based on Java's threading model. However, @@ -349,7 +359,7 @@ and may vary across implementations.  All methods are executed atomically.

-.. method:: Lock.acquire(blocking=True) +.. method:: Lock.acquire(blocking=True, timeout=-1)

   Acquire a lock, blocking or non-blocking.

@@ -363,6 +373,13 @@ All methods are executed atomically.    without an argument would block, return false immediately; otherwise, do the    same thing as when called without arguments, and return true.

s/and as long as the lock cannot be acquired./and return False if the lock couldn't be acquired by then./ ? Also consider an equivalent comment about timeout<=0 as I suggested for _thread.

 .. method:: Lock.release()

@@ -396,7 +413,7 @@ pair) resets the lock to unlocked and al  :meth:acquire to proceed.

-.. method:: RLock.acquire(blocking=True) +.. method:: RLock.acquire(blocking=True, timeout=-1)

   Acquire a lock, blocking or non-blocking.

@@ -415,6 +432,11 @@ pair) resets the lock to unlocked and al    without an argument would block, return false immediately; otherwise, do the    same thing as when called without arguments, and return true.

True and False? And same comment as for Lock.acquire.

 .. method:: RLock.release()

diff -r 8089902215a5 Include/pythread.h --- a/Include/pythread.h        Fri Jan 08 18:54:23 2010 +0100 +++ b/Include/pythread.h        Fri Jan 08 20:33:54 2010 +0100 @@ -23,6 +23,30 @@ PyAPI_FUNC(void) PyThread_free_lock(PyTh  PyAPI_FUNC(int) PyThread_acquire_lock(PyThread_type_lock, int);  #define WAIT_LOCK      1  #define NOWAIT_LOCK    0 + +#if defined(HAVE_LONG_LONG) +#define PY_TIMEOUT_T PY_LONG_LONG +#define PY_TIMEOUT_MAX PY_LLONG_MAX

I think this deserves a comment that it's not the same as _thread.TIMEOUT_MAX and why.

+#else +#define PY_TIMEOUT_T long +#define PY_TIMEOUT_MAX LONG_MAX +#endif + +/* In the NT API, the timeout is a DWORD and is expressed in milliseconds / +#if defined (NT_THREADS) && (0xFFFFFFFFLL * 1000 < PY_TIMEOUT_MAX) +#undef PY_TIMEOUT_MAX +#define PY_TIMEOUT_MAX (0xFFFFFFFFLL * 1000) +#endif + +/ If microseconds == 0, the call is non-blocking: it returns immediately

 PyAPI_FUNC(size_t) PyThread_get_stacksize(void); diff -r 8089902215a5 Lib/_dummy_thread.py --- a/Lib/_dummy_thread.py      Fri Jan 08 18:54:23 2010 +0100 +++ b/Lib/_dummy_thread.py      Fri Jan 08 20:33:54 2010 +0100 @@ -17,6 +17,10 @@ all = ['error', 'start_new_thread',            'interrupt_main', 'LockType']

 import traceback as _traceback +import time + +# A dummy value +TIMEOUT_MAX = 2**31

This should probably be the same as the typical value for _thread.TIMEOUT_MAX.

 class error(Exception):     """Dummy implementation of _thread.error.""" @@ -92,7 +96,7 @@ class LockType(object):     def init(self):         self.locked_status = False

        For blocking calls, self.locked_status is automatically set to @@ -111,6 +115,8 @@ class LockType(object):                 self.locked_status = True                 return True             else:

    enter = acquire diff -r 8089902215a5 Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py       Fri Jan 08 18:54:23 2010 +0100 +++ b/Lib/multiprocessing/pool.py       Fri Jan 08 20:33:54 2010 +0100 @@ -379,10 +379,10 @@ class Pool(object):                 p.terminate()

        debug('joining task handler')

Why is this change here? (Mostly curiosity)

        debug('joining result handler')

        if pool and hasattr(pool[0], 'terminate'):             debug('joining pool workers') diff -r 8089902215a5 Lib/test/lock_tests.py --- a/Lib/test/lock_tests.py    Fri Jan 08 18:54:23 2010 +0100 +++ b/Lib/test/lock_tests.py    Fri Jan 08 20:33:54 2010 +0100 @@ -4,7 +4,7 @@ Various tests for synchronization primit

 import sys  import time -from _thread import start_new_thread, get_ident +from _thread import start_new_thread, get_ident, TIMEOUT_MAX  import threading  import unittest

@@ -62,6 +62,14 @@ class BaseTestCase(unittest.TestCase):         support.threading_cleanup(*self._threads)         support.reap_children()

 class BaseLockTests(BaseTestCase):     """ @@ -143,6 +151,31 @@ class BaseLockTests(BaseTestCase):         Bunch(f, 15).wait_for_finished()         self.assertEqual(n, len(threading.enumerate()))

Please add this to the documentation.

This is just a sanity-check that a successful acquire finishes in a sane amount of time, right? Please comment that.

 class LockTests(BaseLockTests):     """ @@ -178,7 +211,7 @@ class LockTests(BaseLockTests):         b.wait_for_finished()         lock.acquire()         lock.release()

 class RLockTests(BaseLockTests):     """ @@ -284,14 +317,14 @@ class EventTests(BaseTestCase):         def f():             results1.append(evt.wait(0.0))             t1 = time.time()

 class BaseSemaphoreTests(BaseTestCase): diff -r 8089902215a5 Lib/threading.py --- a/Lib/threading.py  Fri Jan 08 18:54:23 2010 +0100 +++ b/Lib/threading.py  Fri Jan 08 20:33:54 2010 +0100 @@ -31,6 +31,7 @@ try:     _CRLock = _thread.RLock  except AttributeError:     _CRLock = None +TIMEOUT_MAX = _thread.TIMEOUT_MAX  del _thread

@@ -107,14 +108,14 @@ class _RLock(_Verbose):         return "<%s owner=%r count=%d>" % (                 self.class.name, owner, self._count)

 static PyObject * -lock_PyThread_acquire_lock(lockobject *self, PyObject *args) +lock_PyThread_acquire_lock(lockobject *self, PyObject *args, PyObject *kwds)  {

I believe it's possible for this comparison to return false, but for the conversion to PY_TIMEOUT_T to still overflow:

$ cat test.c #include <stdio.h> #include <limits.h>

int main() { double d_ll_max = (double)LONG_LONG_MAX; if (d_ll_max > LONG_LONG_MAX) printf("Bigger\n"); if (d_ll_max == LONG_LONG_MAX) printf("Equal\n"); printf("%lld %lf %lld\n", LONG_LONG_MAX, d_ll_max, (long long)d_ll_max); return 0; }

$ ./test Equal 9223372036854775807 9223372036854775808.000000 -9223372036854775808

Unfortunately, that overflowing cast back to long long is undefined behavior, and I don't know how to check for that overflow before it happens.

 PyDoc_STRVAR(acquire_doc, @@ -105,9 +134,9 @@ Return whether the lock is in the locked

 static PyMethodDef lock_methods[] = {        {"acquire_lock", (PyCFunction)lock_PyThread_acquire_lock,

       if (self->rlock_count > 0 ||            !PyThread_acquire_lock(self->rlock_lock, 0)) {

       /* Initialize types: */        if (PyType_Ready(&localtype) < 0) @@ -1027,6 +1082,12 @@ PyInit__thread(void)        if (m == NULL)                return NULL;

 DWORD -EnterNonRecursiveMutex(PNRMUTEX mutex, BOOL wait) +EnterNonRecursiveMutex(PNRMUTEX mutex, DWORD milliseconds)  {        /* Assume that the thread waits successfully */        DWORD ret ;

       /* InterlockedIncrement(&mutex->owned) == 0 means that no thread currently owns the mutex */

Use ==0 now that this in a real integer?

       {                if (InterlockedCompareExchange(&mutex->owned, 0, -1) != -1)                        return WAIT_TIMEOUT ; @@ -49,7 +49,7 @@ EnterNonRecursiveMutex(PNRMUTEX mutex, B        else                ret = InterlockedIncrement(&mutex->owned) ?                        /* Some thread owns the mutex, let's wait... */

       mutex->thread_id = GetCurrentThreadId() ; /* We own it */        return ret ; @@ -249,17 +249,34 @@ PyThread_free_lock(PyThread_type_lock aL  * if the lock has already been acquired by this thread!  */  int +PyThread_acquire_lock_timed(PyThread_type_lock aLock, PY_TIMEOUT_T microseconds) +{

Can (microseconds+999) overflow?

 void diff -r 8089902215a5 Python/thread_pthread.h --- a/Python/thread_pthread.h   Fri Jan 08 18:54:23 2010 +0100 +++ b/Python/thread_pthread.h   Fri Jan 08 20:33:54 2010 +0100 @@ -83,6 +83,14 @@  #endif

+/* We assume all modern POSIX systems have gettimeofday() */

Famous last words. ;) (Not saying you should change anything)

+#ifdef GETTIMEOFDAY_NO_TZ +#define GETTIMEOFDAY(ptv) gettimeofday(ptv) +#else +#define GETTIMEOFDAY(ptv) gettimeofday(ptv, (struct timezone )NULL) +#endif + +  / A pthread mutex isn't sufficient to model the Python lock type  * because, according to Draft 5 of the docs (P1003.4a/D5), both of the  * following are undefined: @@ -335,34 +343,61 @@ fix_status(int status)        return (status == -1) ? errno : status;  }

-int -PyThread_acquire_lock(PyThread_type_lock lock, int waitflag) +int +PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds)  {        int success;        sem_t *thelock = (sem_t *)lock;        int status, error = 0;

       success = (status == 0) ? 1 : 0;

+int +PyThread_acquire_lock(PyThread_type_lock lock, int waitflag) +{

-int -PyThread_acquire_lock(PyThread_type_lock lock, int waitflag) +int +PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds)  {        int success;        pthread_lock *thelock = (pthread_lock *)lock;        int status, error = 0;

       status = pthread_mutex_lock( &thelock->mut );        CHECK_STATUS("pthread_mutex_lock[1]");        success = thelock->locked == 0;

Pull this into a helper function so it's not duplicated between the sem and mutex implementations?

               /* mut must be locked by me -- part of the condition                 * protocol */

       if (error) success = 0;

+int +PyThread_acquire_lock(PyThread_type_lock lock, int waitflag) +{