| msg153744 - (view) |
Author: Matt Joiner (anacrolix) |
Date: 2012-02-20 00:52 |
| There is no Barrier in multiprocessing corresponding to threading.Barrier. |
|
|
| msg153785 - (view) |
Author: Richard Oudkerk (sbt) *  |
Date: 2012-02-20 17:16 |
| Here is an initial implementation. Differences from threading.Barrier: - I have not implemented reset(). - wait() returns 0 or -1. One thread returns 0, the remainder return -1. This is different to threading.Barrier where each of the N threads returns a unique index in range(N). - I added an "action_args" parameter to the constructor. This is a tuple which provides argument to the "action" callback. (This is because closures are not picklable, making no-argument callbacks rather limiting for multiprocessing.) |
|
|
| msg153786 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-02-20 17:20 |
| barrier_tests.py contains minor modifications of the unit tests for threading.Barrier. (The two tests using reset() are commented out.) The implementation passes for me on Linux and Windows. |
|
|
| msg153857 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-02-21 07:50 |
| > Here is an initial implementation. Wouldn't it be simpler with a mp.Condition? Otherwise, this should be added to Lib/multiprocesing.synchronize.py, and the tests to test_multiprocessing. |
|
|
| msg153958 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-02-22 15:28 |
| > Wouldn't it be simpler with a mp.Condition? Well, it is a fair bit shorter than the implementation in threading.py. But that is not a fair comparison because it does implement reset(). I was trying to avoid using shared memory/ctypes since multiprocessing.synchronize does not currently use them. However, I think it would be better (and much simpler) to just subclass threading.Barrier, making self._state and self._counter properties which delegate to RawValue objects. That gets rid of the differences in behaviour. I have this working, although I had to monkey patch multiprocessing.Condition to add a wait_for() method. See Issue 14087. > Otherwise, this should be added to Lib/multiprocesing.synchronize.py, > and the tests to test_multiprocessing. I will provide a patch later. |
|
|
| msg154073 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-02-23 17:19 |
| Patch which subclasses threading.Barrier. |
|
|
| msg154078 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-02-23 19:06 |
| Forgot to mention, mp_barrier.patch needs to be applied on top of cond_wait_for.patch for Issue14087. |
|
|
| msg159335 - (view) |
Author: Charles-François Natali (neologix) * |
Date: 2012-04-25 20:27 |
| The patch looks good. However, I've had two failures while testing it: - a BrokenBarrierError on test_default_timeout: I see you've already increased the timeout from the original threading code, but you can probably double it (we have some slow buildbots, and creating processes is expensive on Windows) - TestNumberOfObjects is failing too: """ test_number_of_objects (test.test_multiprocessing.WithManagerTestZZZNumberOfObjects) ... b6cb776c: refcount=1 <threading.Barrier object at 0xb6cb776c> b6d72dfc: refcount=2 <multiprocessing.pool.Pool object at 0xb6d72dfc> b6cb776c: refcount=1 <threading.Barrier object at 0xb6cb776c> b6d72dfc: refcount=2 <multiprocessing.pool.Pool object at 0xb6d72dfc> FAIL """ |
|
|
| msg159455 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-04-27 14:09 |
| RawValue uses ctypes, right? That's problematic for platforms which don't support ctypes. |
|
|
| msg161671 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-05-26 17:42 |
| > RawValue uses ctypes, right? That's problematic for platforms which don't > support ctypes. Are there many posix systems (we care about) where ctypes doesn't work? It would be fairly easy to use memoryview instead of ctypes. (In fact Value/RawValue could be implemented that way when using a typecode instead of a ctypes type.) |
|
|
| msg161673 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-05-26 17:56 |
| > > RawValue uses ctypes, right? That's problematic for platforms which don't > > support ctypes. > > Are there many posix systems (we care about) where ctypes doesn't > work? It depends what you call "caring about" :-) But proprietary Unix C compilers generally don't mix very well with ctypes. > It would be fairly easy to use memoryview instead of ctypes. (In fact > Value/RawValue could be implemented that way when using a typecode > instead of a ctypes type.) That would be a good idea then. Also, not using ctypes could make code theoretically more robust. |
|
|
| msg162371 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2012-06-05 20:30 |
| The attached patch uses memoryview instead of ctypes. If the patch for Issue #14953 (reimplementing RawArray/RawValue in terms of memoryview) is applied, then it could be simplified a bit. |
|
|
| msg162923 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2012-06-15 17:33 |
| New changeset 2d2f206d040e by Richard Oudkerk in branch 'default': Issue #14059: Implement multiprocessing.Barrier http://hg.python.org/cpython/rev/2d2f206d040e |
|
|