| msg105013 - (view) |
Author: (yig) |
Date: 2010-05-05 12:00 |
| Calling uuid.uuid4() while using the multiprocessing module leads to the same exact UUIDs being generating in each process. It is an artifact resulting from the built-in uuid_generate_random() of my underlying platform, Mac OS X 10.6.3. A Linux machine I have does not exhibit this problem. I have tested it with both Python 2.5 and 2.6. |
|
|
| msg105018 - (view) |
Author: (yig) |
Date: 2010-05-05 12:34 |
| For the record, I filed a bug against the underlying platform. (I wrote a simple program to reproduce this in C and filed a bug report with Apple rdar://7944700. The OpenRadar page for it is here: http://openradar.appspot.com/radar?id=334401 ) |
|
|
| msg105024 - (view) |
Author: Ronald Oussoren (ronaldoussoren) *  |
Date: 2010-05-05 13:18 |
| As the bug is in the underlying platform the best we can do is to warn about this in the documentation, as in the attached patch. BTW. I've updated the title to be slightly more informative. |
|
|
| msg105027 - (view) |
Author: (yig) |
Date: 2010-05-05 13:34 |
| Why not default to not use the Python implementation on darwin instead of the underlying platform's uuid_generate_random(), until it's proven safe? |
|
|
| msg105028 - (view) |
Author: (yig) |
Date: 2010-05-05 13:35 |
| Ahem. Why not use the Python implementation on darwin until its uuid_generate_random() is deemed to be safe? |
|
|
| msg105029 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-05-05 13:52 |
| Because I didn't look closely enough at the source :-( The attached patch disabled the C implementation on OSX 10.6 or later. I've tested that 10.5 is not affected by the issue. |
|
|
| msg105036 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-05-05 14:05 |
| Also added a testcase that should warn if other unix-y platforms start to suffer from the same issue. BTW. .patch uses a runtime test in the uuid module instead of a configure-check because a binary might be created on 10.5 (without the issue) and run on 10.6 (with the issue) and that should not result in a broken library. |
|
|
| msg105039 - (view) |
Author: (yig) |
Date: 2010-05-05 14:21 |
| Great work! Very thorough patches. Strange that it's a regression versus 10.5. |
|
|
| msg105051 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-05-05 15:34 |
| There are way more interesting regressions in OSX, is one example: basicly getgroups(2) does not reflect the results of setgroups(2) with the compiler settings we use. Committed in r80784 (trunk), r80785 (2.6), r80786 (3.2) and r80788 (3.1) |
|
|
| msg108326 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2010-06-21 22:31 |
| Reopening since test failures are reported on python-dev: [...] test_uuid test test_uuid failed -- Traceback (most recent call last): File "/private/tmp/Python-2.7rc2/Lib/test/test_uuid.py", line 472, in testIssue8621 self.assertNotEqual(parent_value, child_value) AssertionError: '8395a08e40454895be537a180539b7fb' == '8395a08e40454895be537a180539b7fb' [...] |
|
|
| msg108332 - (view) |
Author: Bill Janssen (janssen) * |
Date: 2010-06-22 01:04 |
| This is on an Intel machine running OS X 10.5.8. I downloaded and built 2.7rc2 from source with "./configure ; make". I then ran the tests with "make test". test_uuid fails with this output: test test_uuid failed -- Traceback (most recent call last): File "/private/tmp/Python-2.7rc2/Lib/test/test_uuid.py", line 472, in testIssue8621 self.assertNotEqual(parent_value, child_value) AssertionError: '751ca85de22f4450b7f95dd3f82c7e5f' == '751ca85de22f4450b7f95dd3f82c7e5f' However, when I run the test standalone with this command-line, it passes: % ./python.exe -Wd -3 -E -tt ./Lib/test/regrtest.py -l test_uuid test_uuid 1 test OK. Not sure what's going on. |
|
|
| msg108348 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-06-22 09:40 |
| Odd, unless someone already checked in a fix on the trunk. I currently have a clean test run on the trunk on OSX 10.6.4 (intel). I haven't checked this on my 10.5 VM yet. |
|
|
| msg108349 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2010-06-22 09:45 |
| It is disputed on http://openradar.appspot.com/radar?id=334401 that this is an OS bug. If I understand correctly, the test program is not guaranteed to work if threads are involved. See also: http://webcache.googleusercontent.com/search?q=cache:hJo0u_Lc8wkJ:www.opengroup.org/onlinepubs/000095399/functions/fork.html+http://www.opengroup.org/onlinepubs/009695399/functions/fork.html&hl=en&strip=1 "A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. [THR] [Option Start] Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls." This could explain why running the complete test suite fails but running the individual test does not. |
|
|
| msg108350 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-06-22 09:49 |
| Stefan: we already new that, see . This issue was closed as fixed because the uuid module contains a workaround for this issue (by not using the broken C API on OSX 10.6). It seems that OSX 10.5.8 and 10.4 is also affected by this issue. I'll test on 10.5, and if I can reproduce the issue there I'll adjust the workaround for this. |
|
|
| msg108351 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2010-06-22 09:56 |
| Bill, could you try to add this to the tests and see if they also fail when you run them standalone? Index: Lib/test/test_uuid.py =================================================================== --- Lib/test/test_uuid.py (revision 82109) +++ Lib/test/test_uuid.py (working copy) @@ -479,4 +479,7 @@ test_support.run_unittest(TestUUID) if __name__ == '__main__': + import threading + t = threading.Thread(target=lambda: None) + t.start() test_main() |
|
|
| msg108352 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-06-22 09:59 |
| Bill, please test the patch below on a 10.5 system: Index: Lib/uuid.py =================================================================== --- Lib/uuid.py (revision 82148) +++ Lib/uuid.py (working copy) @@ -416,7 +416,7 @@ import sys if sys.platform == 'darwin': import os - if int(os.uname()[2].split('.')[0]) >= 10: + if int(os.uname()[2].split('.')[0]) >= 9: _uuid_generate_random = _uuid_generate_time = None # On Windows prior to 2000, UuidCreate gives a UUID containing the This extends the workaround for the bug in the system uuid libraries to OSX 10.5. |
|
|
| msg108368 - (view) |
Author: Stefan Krah (skrah) * |
Date: 2010-06-22 11:21 |
| Ronald Oussoren <report@bugs.python.org> wrote: > Stefan: we already new that, see . > > This issue was closed as fixed because the uuid module contains a workaround for this issue (by not using the broken C API on OSX 10.6). Ok, my comment was partly meant to give an explanation why the test suite failed but the individual tests did not. If uuid_generate_random() is specified as async-signal-safe, then yes, the C API would be broken on OSX. |
|
|
| msg108753 - (view) |
Author: Bill Janssen (janssen) * |
Date: 2010-06-26 22:01 |
| Stefan, I tried your suggestion of starting threading. Test still succeeds on my 10.5.8 system when test_uuid is run separately. Ronald, your fix works on my 10.5.8 system. Why not check it in, and let's see if the buildbots turn green again? |
|
|
| msg108783 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-06-27 12:46 |
| Committed a fix in r82276 (2.7), r82277 (2.6), r82278 (3.2) and r82279 (3.1). Please reopen the issue if you notice that the problem does not go away on the buildbots. |
|
|