| msg147299 - (view) |
Author: Ronald Oussoren (ronaldoussoren) *  |
Date: 2011-11-08 14:11 |
| test_ctypes fails when building and testing on OSX 10.7 (Lion). This is likely caused by changes in the compiler w.r.t. the calling convention. I've worked around this problem in PyObjC, and will try to port that workaround to ctypes. BTW. Using --with-system-ffi does not fix the issue for me |
|
|
| msg147301 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2011-11-08 14:12 |
| > test_ctypes fails when building and testing on OSX 10.7 (Lion). What is the error message? |
|
|
| msg147302 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2011-11-08 14:19 |
| I should have included the failures in my first message, sorry about that. IIRC the workaround in PyObjC I mention in my initial message does explicit sign extension when needed. W.r.t to the first couple of failures, not that: struct.unpack('b', struct.pack('B', 130)) == (-126,) ====================================================================== FAIL: test_byte (ctypes.test.test_cfuncs.CFunctions) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_cfuncs.py", line 21, in test_byte self.assertEqual(self.S(), -126) AssertionError: 130 != -126 ====================================================================== FAIL: test_byte_plus (ctypes.test.test_cfuncs.CFunctions) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_cfuncs.py", line 27, in test_byte_plus self.assertEqual(self.S(), -126) AssertionError: 130 != -126 ====================================================================== FAIL: test_short (ctypes.test.test_cfuncs.CFunctions) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_cfuncs.py", line 45, in test_short self.assertEqual(self.S(), -32766) AssertionError: 32770 != -32766 ====================================================================== FAIL: test_short_plus (ctypes.test.test_cfuncs.CFunctions) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_cfuncs.py", line 51, in test_short_plus self.assertEqual(self.S(), -32766) AssertionError: 32770 != -32766 ====================================================================== FAIL: test_doubleresult (ctypes.test.test_functions.FunctionTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_functions.py", line 143, in test_doubleresult self.assertEqual(result, -21) AssertionError: 65771.0 != -21 ====================================================================== FAIL: test_floatresult (ctypes.test.test_functions.FunctionTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_functions.py", line 131, in test_floatresult self.assertEqual(result, -21) AssertionError: 65771.0 != -21 ====================================================================== FAIL: test_intresult (ctypes.test.test_functions.FunctionTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_functions.py", line 105, in test_intresult self.assertEqual(result, -21) AssertionError: 65771 != -21 ====================================================================== FAIL: test_longdoubleresult (ctypes.test.test_functions.FunctionTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_functions.py", line 155, in test_longdoubleresult self.assertEqual(result, -21) AssertionError: 65771.0 != -21 |
|
|
| msg152662 - (view) |
Author: Meador Inge (meador.inge) * |
Date: 2012-02-05 06:46 |
| I can't reproduce this on Lion using the 2.7 or default branches. I can't build with GCC at the moment because of . I am using the clang that ships with Lion: motherbrain:cpython meadori$ sw_vers ProductName: Mac OS X ProductVersion: 10.7.2 BuildVersion: 11C74 motherbrain:cpython meadori$ clang --version Apple clang version 3.0 (tags/Apple/clang-211.12) (based on LLVM 3.0svn) Target: x86_64-apple-darwin11.2.0 Thread model: posix |
|
|
| msg152690 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2012-02-05 15:17 |
| It looks like the test failures do not show up with debug enabled (--with-pydebug). I'm able to reproduce them with: ./configure MACOSX_DEPLOYMENT_TARGET=10.7 CC=clang |
|
|
| msg152699 - (view) |
Author: Meador Inge (meador.inge) * |
Date: 2012-02-05 19:22 |
| I can reproduce it without --with-pydebug. Thanks. I will investigate. |
|
|
| msg152714 - (view) |
Author: Ezra Berch (ezberch) |
Date: 2012-02-06 00:32 |
| The reason for this issue appears to be a bug in the clang optimizer. This is why it only shows up with debug disabled. When I compile with -O0 instead of -O3, the test passes. This issue from the clang tracker may be the same issue: http://llvm.org/bugs/show_bug.cgi?id=9524 |
|
|
| msg152717 - (view) |
Author: Meador Inge (meador.inge) * |
Date: 2012-02-06 01:29 |
| So I have debugged the first failure and it has to do with the sign extending of 'signed char' parameters. The first test case basically does this: _dll = CDLL(_ctypes_test.__file__) _dll.tf_b.restype = c_byte _dll.tf_b.argtypes = (c_byte,) _dll.tf_b(-126) where 'tf_b' is defined in C code like: signed char tf_b(signed char c) { return c/3; } Clang with -O3 generates code for 'tf_b' like: 0x0000000100000ed0 <tf_b+0>: push %rbp 0x0000000100000ed1 <tf_b+1>: mov %rsp,%rbp 0x0000000100000ed4 <tf_b+4>: movslq %edi,%rax 0x0000000100000ed7 <tf_b+7>: imul 0x55555556,0x55555556,%rax,%rax 0x0000000100000ede <tf_b+14>: mov %rax,%rcx 0x0000000100000ee1 <tf_b+17>: shr 0x55555556,0x3f,%rcx 0x0000000100000ee5 <tf_b+21>: shr $0x20,%rax 0x0000000100000ee9 <tf_b+25>: add %ecx,%eax 0x0000000100000eeb <tf_b+27>: movsbl %al,%eax 0x0000000100000eee <tf_b+30>: pop %rbp 0x0000000100000eef <tf_b+31>: retq See how 'movslq' is used to sign extend the first argument? Since the first argument is a 'signed char', that should be 'movsbq'. I am pretty sure this is a clang bug. I am going to see if this is fixed in later versions of clang. I will also search the clang tracker to see if this has been reported. |
|
|
| msg152718 - (view) |
Author: Meador Inge (meador.inge) * |
Date: 2012-02-06 01:40 |
| > Since the first argument is a 'signed char', that should be 'movsbq'. Hmmm, on second thought I am not 100% sure about that exact statement, but I still think this is most likely a clang bug having something to do with sign extension. |
|
|
| msg152801 - (view) |
Author: Meador Inge (meador.inge) * |
Date: 2012-02-07 02:12 |
| It turns out that this isn't OS X specific. I can reproduce the exact same issue by building with clang on a 64-bit Fedora 16 box. |
|
|
| msg152804 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2012-02-07 05:00 |
| Thanks for the analysis, Meador. So perhaps as a temporary workaround, we could tweak setup.py to set -O0 for building ctypes with clang? |
|
|
| msg152923 - (view) |
Author: Meador Inge (meador.inge) * |
Date: 2012-02-09 03:24 |
| > So perhaps as a temporary workaround, we could tweak setup.py to set > -O0 for building ctypes with clang? That would make the tests pass, but it still won't fix the fundamental issue. clang and GCC have a difference in opinion as to when function parameters and return values should be sign extended. libffi follows the GCC model. I am talking to the LLVM folks right now to see why things are implemented this way. Whether LLVM or libffi should be fixed depends on the outcome of that discussion. Due to this ABI difference compiling _ctypes with -O0 for clang will not fix the problem because any ctypes code that interacts with C code that was compiled by clang with optimizations has the potential to break just like our test cases. |
|
|
| msg155148 - (view) |
Author: Meador Inge (meador.inge) * |
Date: 2012-03-08 03:23 |
| LLVM/clang bug report here: http://llvm.org/bugs/show_bug.cgi?id=12207. |
|
|
| msg156216 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2012-03-17 23:49 |
| An update: test_ctypes continues to fail on 2.7.3rc2 with exactly the same failures as originally reported when compiled non-debug 64-bit (x86_64) with either the latest clang *or* llvm-gcc from Xcode 4.3.1. It does not fail when compiled with gcc-4.2 from Xcode 3.2.6. Not does it fail with any of the three compilers when compiled and run as 32-bit (i386). So it does seem to come down to a difference between standard gcc and with the llvm-based compilers (clang, llvm-gcc), and not just clang. Python 2.7.3rc2 (2.7:f59ab537a5dc, Mar 17 2012, 15:26:37) [GCC 4.2.1 Compatible Apple Clang 3.1 (tags/Apple/clang-318.0.54)] on darwin Python 2.7.3rc2 (2.7:f59ab537a5dc, Mar 17 2012, 15:47:25) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.9.00)] on darwin Python 2.7.3rc2 (2.7:f59ab537a5dc, Mar 17 2012, 16:08:58) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin |
|
|
| msg168203 - (view) |
Author: Stan Seibert (Stan.Seibert) |
Date: 2012-08-14 15:14 |
| FYI: This bug still exists using the version of clang shipped with OS X 10.8. Tested with Python 3.3.0b2 and: Apple clang version 4.0 (tags/Apple/clang-421.0.57) (based on LLVM 3.1svn) Target: x86_64-apple-darwin12.0.0 Thread model: posix |
|
|
| msg168275 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-15 10:45 |
| The attached patch syncs libffi_osx with the copy of libffi that's included with PyObjC. With this patch test_ctypes passes for me when using clang to build on OSX 10.8. I haven't run the tests with other python releases yet, and also not on a OSX 10.5 system (where I can test if the patch works with GCC as well) |
|
|
| msg168672 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-20 15:25 |
| I've tested the patch on OSX 10.5 with GCC and all tests pass. I'd like to get this into 3.3rc1 as the test failure indicate a problem that could cause very vague problems for users of the ctypes library that use clang on OSX to build python (which includes anyone building python from source on OSX 10.7 or later) |
|
|
| msg168676 - (view) |
Author: Stan Seibert (Stan.Seibert) |
Date: 2012-08-20 16:22 |
| I can also confirm that the patch applied to the current tip (663de4dbb88a) passes test_ctypes now with Clang 4.0 on OS X 10.8. |
|
|
| msg168753 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2012-08-21 10:43 |
| The patch as it stands causes _ctypes build failures when using the 10.4u SDK as in the traditional 10.3+ 32-bit-only installer configuration. Apparently, in ppc-ffi_darwin.c, the skip that the patch removes was because OSCacheControl.h doesn't exist in the 10.4u SDK. It is in the 10.5 SDK and _ctypes.so builds and tests OK on both PPC and i386 with it. --- a/Modules/_ctypes/libffi_osx/powerpc/ppc-ffi_darwin.c +++ b/Modules/_ctypes/libffi_osx/powerpc/ppc-ffi_darwin.c @@ -37,19 +37,10 @@ #include <ppc-darwin.h> #include <architecture/ppc/mode_independent_asm.h> -#if 0 #if defined(POWERPC_DARWIN) #include <libkern/OSCacheControl.h> // for sys_icache_invalidate() #endif -#else - -#pragma weak sys_icache_invalidate -extern void sys_icache_invalidate(void *start, size_t len); - -#endif ./Modules/_ctypes/libffi_osx/powerpc/ppc-ffi_darwin.c:41:67: error: libkern/OSCacheControl.h: No such file or directory |
|
|
| msg168852 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-22 08:02 |
| The change to ppc-ffi_darwin.c was unintentional and should be removed. I must have been sleeping during the merge, as I only intended to copy the files for i386/x86_64 and not for PPC as the latter architecture didn't have problems when compiled with clang. I've copied the ppc-ffi_darwin.c file from Python's tree into the PyObjC tree to avoid problems when there are future merges. |
|
|
| msg168923 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2012-08-23 05:46 |
| I just noticed another anomaly in the patch. In x86-ffi_darwin.c there appears to be some bogus code duplicating ffi_prep_args. Could you review and clean up the patch before we apply it? |
|
|
| msg168950 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-23 17:56 |
| I didn't have time to look into this today and will look into this tomorrow. |
|
|
| msg169005 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-24 12:28 |
| What code duplication do you mean? The code in the switch statement at line 71 looks similar for the various case, but it isn't: there is variation in the types uses in the casts (and particularly in the cast used to read the input value). |
|
|
| msg169006 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-24 12:31 |
| Attached version of the patch without the change to the powerpc specific code. I will commit this version later today (after another testing round). |
|
|
| msg169071 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2012-08-24 18:59 |
| Sorry, I meant this part of the patch: diff --git a/Modules/_ctypes/libffi_osx/x86/x86-ffi_darwin.c b/Modules/_ctypes/libffi_osx/ --- a/Modules/_ctypes/libffi_osx/x86/x86-ffi_darwin.c +++ b/Modules/_ctypes/libffi_osx/x86/x86-ffi_darwin.c @@ -35,6 +35,8 @@ /* ffi_prep_args is called by the assembly routine once stack space has been allocated for the function's arguments */ +void ffi_prep_args(char *stack, extended_cif *ecif); + void ffi_prep_args(char *stack, extended_cif *ecif) { register unsigned int i; |
|
|
| msg169089 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-24 21:15 |
| That's not code duplication, but a way to silence a compiler warning. I added it while trying to get pyobjc to compile without any warnings. |
|
|
| msg169094 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-24 21:49 |
| I noticed earlier today that test_ctypes passes for Python 3.3 both with and without my patch. This is on OSX 10.8.1 with an up-to-date Xcode. Configure arguments: '--enable-framework' '--with-pydebug' '--enable-universalsdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk' '--with-universal-archs=intel' '--with-framework-name=PythonDev' 'MACOSX_DEPLOYMENT_TARGET=10.8' 'CC=clang' 'CXX=clang++' That's why I haven't commited the patch yet. I'm installing Xcode in a OSX 10.7 VM to check if the patch is necessary on that platform. |
|
|
| msg169098 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2012-08-24 23:29 |
| Bah, I just noticed that you built --with-pydebug enabled. That hides the problem. I was fooled myself there. But I've since verified that the failure still exists when built on 10.7 with Xcode 4.4.1 so the patch should go in. |
|
|
| msg169117 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-25 08:35 |
| Duh... 3.3 without patch fails when I leave out '--with-pydebug'. I'm now running the testsuite for 2.7 and 3.2 with patch and will commit once those are ready, which should be within an hour from now. |
|
|
| msg169120 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2012-08-25 09:24 |
| New changeset 58a87c02692c by Ronald Oussoren in branch '2.7': Fix issue 13370: Ensure that ctypes works on Mac OS X when Python is compiled using the clang compiler http://hg.python.org/cpython/rev/58a87c02692c New changeset a425f2697273 by Ronald Oussoren in branch '3.2': Fix issue 13370: Ensure that ctypes works on Mac OS X when Python is compiled using the clang compiler http://hg.python.org/cpython/rev/a425f2697273 New changeset bed8dd30309d by Ronald Oussoren in branch 'default': Fix issue 13370: Ensure that ctypes works on Mac OS X when Python is http://hg.python.org/cpython/rev/bed8dd30309d |
|
|
| msg169121 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2012-08-25 09:25 |
| The issue should be gone in the upcoming 3.3rc1 release and the next stable releases of 3.2 and 2.7. |
|
|