| msg183156 - (view) |
Author: Matt Clarke (Matt.Clarke) |
Date: 2013-02-27 15:51 |
| I have had an issue arise with ctypes callbacks with 64bit Python on Windows. Note: everything works fine with 32bit Python on Windows and on 32bit and 64bit Linux. I have created a simple example to illustrate the issue I have (see attachment), but the real-life issue occurs with using Python to interact with the EPICS control software (http://www.aps.anl.gov/epics/) used at many major scientific institutes. Basically, if I have a C callback that takes a struct (by value) greater than 8 bytes then the callback returns nonsense. 8 bytes or less works fine. Stepping through with the Windows debugger, if appears that something goes amiss between the callback being called in C and the closure_fcn(ffi_cif *cif, void *resp, void **args, void *userdata) function in ctypes's callback.c file. Unfortunately, the debugger won't let me step in between those two points. Looking at the memory I can see the original data in memory at some memory address, X, and a copy of the data at X+40 bytes, but the args in the closure_fcn points at X-40 bytes (which is junk). Using 32bit Python the data copy is at X-40 bytes and the args pointer in the closure_fcn also points at this. EPICS has some 64bit C/C++ clients that work fine using callbacks on Windows. Likewise, doing the same sort of thing as ctypes does with EPICS from C# using PInvoke works fine. Any help would be much appreciated. |
|
|
| msg184078 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *  |
Date: 2013-03-13 13:19 |
| Is "_pack_ = 1" correct? Did you compile your C library with /Zp1 or similar? Also check that ctypes.sizeof(callback_t) matches the one given by the C compiler. |
|
|
| msg184444 - (view) |
Author: Matt Clarke (Matt.Clarke) |
Date: 2013-03-18 11:15 |
| Hi Amaury. I have tried removing pack and using /Zp1, but it makes no difference. The size of callback_t and the one in C are 8 bytes. Thanks, Matt On 13 March 2013 13:19, Amaury Forgeot d'Arc <report@bugs.python.org> wrote: > > Amaury Forgeot d'Arc added the comment: > > Is "_pack_ = 1" correct? Did you compile your C library with /Zp1 or > similar? > Also check that ctypes.sizeof(callback_t) matches the one given by the C > compiler. > > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue17310> > _______________________________________ > |
|
|
| msg184457 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2013-03-18 14:37 |
| Sorry, I asked the wrong question; callback_t is a function pointer, so 8 bytes is expected. What is sizeof(myst_args) both in C and Python? |
|
|
| msg184460 - (view) |
Author: Matt Clarke (Matt.Clarke) |
Date: 2013-03-18 15:19 |
| Hi Amaury. They are both 12 bytes. Matt |
|
|
| msg187650 - (view) |
Author: Matt Clarke (Matt.Clarke) |
Date: 2013-04-23 14:50 |
| It seems that any argument greater than 8 bytes is automatically converted to a references. Thus, changing to using ctypes.POINTER works. For example: callback_t = ctypes.CFUNCTYPE(None, ctypes.POINTER(myst_args)) Quite a simple solution in the end. Is it worth documenting this on the ctypes page? Thanks for your help, Matt On 18 March 2013 15:19, Matt Clarke <report@bugs.python.org> wrote: > > Matt Clarke added the comment: > > Hi Amaury. > > They are both 12 bytes. > > Matt > > ---------- > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue17310> > _______________________________________ > |
|
|
| msg245146 - (view) |
Author: Vinay Sajip (vinay.sajip) * |
Date: 2015-06-10 22:25 |
| Although Matt was able to work around his problem, this problem seems to still be present in 2.7 as well as all 3.x versions. I think I've found the cause of the problem: in 64-bit code, the calling conventions for passing structures by value are different. From this page: https://msdn.microsoft.com/en-us/library/zthk2dkh(v=vs.90).aspx I would point to "Structs/unions of size 8, 16, 32, or 64 bits and __m64 will be passed as if they were integers of the same size. Structs/unions other than these sizes will be passed as a pointer to memory allocated by the caller. For these aggregate types passed as a pointer (including __m128), the caller-allocated temporary memory will be 16-byte aligned." The code in ffi_prep_incoming_args_SYSV (see https://hg.python.org/cpython/file/a1b76c1c3be8/Modules/_ctypes/libffi_msvc/ffi.c#l368) assumes (see lines 399, 402) that all value parameters are always passed inline on the stack - which is plainly not as per the documentation I linked to, for 64-bit code. |
|
|
| msg245148 - (view) |
Author: Vinay Sajip (vinay.sajip) * |
Date: 2015-06-10 22:30 |
| I note that the code for CFFI has a fix in ffi_prep_incoming_args_SYSV, as follows: #ifdef _WIN64 if (z > 8) { /* On Win64, if a single argument takes more than 8 bytes, then it is always passed by reference. */ *p_argv = *((void**) argp); z = 8; } else #endif *p_argv = (void*) argp; /* The original code, works for 32-bit */ (Source: https://bitbucket.org/cffi/cffi/src/abc8ff5b2885b3d9f22bbb314a011b8dd63c9e14/c/libffi_msvc/ffi.c?at=default) |
|
|
| msg245151 - (view) |
Author: Vinay Sajip (vinay.sajip) * |
Date: 2015-06-10 22:46 |
| I can confirm that the CFFI patch works as expected on Python 2.7.10. |
|
|
| msg270900 - (view) |
Author: Patrick Stewart (Patrick Stewart) |
Date: 2016-07-21 01:51 |
| This also fixes python 3.5 |
|
|
| msg270917 - (view) |
Author: Patrick Stewart (Patrick Stewart) |
Date: 2016-07-21 12:10 |
| I've attached a patch with an extra fix to the duplicated issue 20160 http://bugs.python.org/issue20160 |
|
|
| msg271906 - (view) |
Author: Vinay Sajip (vinay.sajip) * |
Date: 2016-08-03 14:27 |
| I'm closing this as a duplicate of issue 20160 - it appears to be the same bug. |
|
|