Ctypes - SciPy wiki dump (original) (raw)

Contents

Table of Contents
Introduction
Getting Started with ctypes
NumPy arrays' ctypes property
NumPy's ndpointer with ctypes argtypes
Dynamic allocation through callbacks
More useful code frags
Heterogeneous Types Example
Fibonacci example (using NumPy arrays, C and Scons)
Pertinent Mailing List Threads
Documentation

Introduction

ctypes is an advanced Foreign Function Interface package for Python 2.3 and higher. It is included in the standard library for Python 2.5.

ctypes allows to call functions exposed from DLLs/shared libraries and has extensive facilities to create, access and manipulate simple and complicated C data types in Python - in other words: wrap libraries in pure Python. It is even possible to implement C callback functions in pure Python.

ctypes also includes a code generator tool chain which allows automatic creation of library wrappers from C header files. ctypes works on Windows, Mac OS X, Linux, Solaris, FreeBSD, OpenBSD and other systems.

Ensure that you have at least ctypes version 1.0.1 or later.

Other possibilities to call or run C code in python include: SWIG, Cython, Weave, etc

Getting Started with ctypes

The ctypes tutorial and the ctypes documentation for Python provide extensive information on getting started with ctypes.

Assuming you've built a library called foo.dll or libfoo.so containing a function called bar that takes a pointer to a buffer of doubles and an int as arguments and returns an int, the following code should get you up and running. The following sections cover some possible build scripts, C code and Python code.

If you would like to build your DLL/shared library with distutils, take a look at the SharedLibrary distutils extension included with OOF2. This should probably be included in numpy.distutils at some point.

Nmake Makefile (Windows)

Run nmake inside the Visual Studio Command Prompt to build with the following file.

You should be able to build the DLL with any version of the Visual Studio compiler regardless of the compiler used to compile Python. Keep in mind that you shouldn't allocate/deallocate memory across different debug/release and single-threaded/multi-threaded runtimes or operate on FILE*s from different runtimes.

CXX = cl.exe LINK = link.exe

CPPFLAGS = -D_WIN32 -D_USRDLL -DFOO_DLL -DFOO_EXPORTS CXXFLAGSALL = -nologo -EHsc -GS -W3 -Wp64 $(CPPFLAGS) CXXFLAGSDBG = -MDd -Od -Z7 -RTCcsu CXXFLAGSOPT = -MD -O2 #CXXFLAGS = (CXXFLAGSALL)(CXXFLAGSALL) (CXXFLAGSALL)(CXXFLAGSDBG) CXXFLAGS = (CXXFLAGSALL)(CXXFLAGSALL) (CXXFLAGSALL)(CXXFLAGSOPT)

LINKFLAGSALL = /nologo /DLL LINKFLAGSDBG = /DEBUG LINKFLAGSOPT = #LINKFLAGS = (LINKFLAGSALL)(LINKFLAGSALL) (LINKFLAGSALL)(LINKFLAGSDBG) LINKFLAGS = (LINKFLAGSALL)(LINKFLAGSALL) (LINKFLAGSALL)(LINKFLAGSOPT)

all: foo.dll

foo.dll: foo.obj (LINK)(LINK) (LINK)(LINKFLAGS) foo.obj /OUT:foo.dll

svm.obj: svm.cpp svm.h (CXX)(CXX) (CXX)(CXXFLAGS) -c foo.cpp

clean: -erase /Q *.obj *.dll *.exp *.lib

SConstruct (GCC)

You can use the following file with SCons to build a shared library.

1 env = Environment() 2 env.Replace(CFLAGS=['-O2','-Wall','-ansi','-pedantic']) 3 env.SharedLibrary('foo', ['foo.cpp'])

foo.cpp

1 #include <stdio.h> 2 3 #ifdef FOO_DLL 4 #ifdef FOO_EXPORTS 5 #define FOO_API __declspec(dllexport) 6 #else 7 #define FOO_API __declspec(dllimport) 8 #endif 9 #else 10 #define FOO_API extern 11 #endif 12 13 #ifdef __cplusplus 14 extern "C" { 15 #endif 16 17 extern FOO_API int bar(double* data, int len) { 18 int i; 19 printf("data = %p\n", (void*) data); 20 for (i = 0; i < len; i++) { 21 printf("data[%d] = %f\n", i, data[i]); 22 } 23 printf("len = %d\n", len); 24 return len + 1; 25 } 26 27 #ifdef __cplusplus 28 } 29 #endif 30

When building the DLL for foo on Windows, define FOO_DLL and FOO_EXPORTS (this is what you want to do when building a DLL for use with ctypes). When linking against the DLL, define FOO_DLL. When linking against a static library that contains foo, or when including foo in an executable, don't define anything.

If you're unclear about what extern "C" is for, read section 3 of the C++ dlopen mini HOWTO. This allows you to write function wrappers with C linkage on top of a bunch of C++ classes so that you can use them with ctypes. Alternatively, you might prefer to write C code.

foo.py

1 import numpy as N 2 import ctypes as C 3 _foo = N.ctypeslib.load_library('libfoo', '.') 4 _foo.bar.restype = C.c_int 5 _foo.bar.argtypes = [C.POINTER(C.c_double), C.c_int] 6 def bar(x): 7 return _foo.bar(x.ctypes.data_as(C.POINTER(C.c_double)), len(x)) 8 x = N.random.randn(10) 9 n = bar(x)

NumPy arrays' ctypes property

A ctypes property was recently added to NumPy arrays:

In [18]: x = N.random.randn(2,3,4)

In [19]: x.ctypes.data Out[19]: c_void_p(14394256)

In [21]: x.ctypes.data_as(ctypes.POINTER(c_double))

In [24]: x.ctypes.shape Out[24]: <ctypes._endian.c_long_Array_3 object at 0x00DEF2B0>

In [25]: x.ctypes.shape[:3] Out[25]: [2, 3, 4]

In [26]: x.ctypes.strides Out[26]: <ctypes._endian.c_long_Array_3 object at 0x00DEF300>

In [27]: x.ctypes.strides[:3] Out[27]: [96, 32, 8]

In general, a C function might take a pointer to the array's data, an integer indicating the number of array dimensions, (pass the value of the ndim property here) and two int pointers to the shapes and stride information.

If your C function assumes contiguous storage, you might want to wrap it with a Python function that calls NumPy's ascontiguousarray function on all the input arrays.

NumPy's ndpointer with ctypes argtypes

Starting with ctypes 0.9.9.9, any class implementing the from_param method can be used in the argtypes list of a function. Before ctypes calls a C function, it uses the argtypes list to check each parameter.

Using NumPy's ndpointer function, some very useful argtypes classes can be constructed, for example:

1 from numpy.ctypeslib import ndpointer 2 arg1 = ndpointer(dtype='<f4') 3 arg2 = ndpointer(ndim=2) 4 arg3 = ndpointer(shape=(10,10)) 5 arg4 = ndpointer(flags='CONTIGUOUS,ALIGNED') 6 7 arg5 = ndpointer(dtype='>i4', flags='CONTIGUOUS') 8 func.argtypes = [arg1,arg2,arg3,arg4,arg5]

Now, if an argument doesn't meet the requirements, a TypeError is raised. This allows one to make sure that arrays passed to the C function is in a form that the function can handle.

See also the mailing list thread on ctypes and ndpointer.

Dynamic allocation through callbacks

ctypes supports the idea of callbacks, allowing C code to call back into Python through a function pointer. This is possible because ctypes releases the Python Global Interpreter Lock (GIL) before calling the C function.

We can use this feature to allocate NumPy arrays if and when we need a buffer for C code to operate on. This could avoid having to copy data in certain cases. You also don't have to worry about freeing the C data after you're done with it. By allocating your buffers as NumPy arrays, the Python garbage collector can take care of this.

Python code:

1 from ctypes import * 2 ALLOCATOR = CFUNCTYPE(c_long, c_int, POINTER(c_int)) 3 4 lib.baz.restype = None 5 lib.baz.argtypes = [c_float, c_int, ALLOCATOR]

This isn't the prettiest way to define the allocator (I'm also not sure if c_long is the right return type), but there are a few bugs in ctypes that seem to make this the only way at present. Eventually, we'd like to write the allocator like this (but it doesn't work yet):

1 from numpy.ctypeslib import ndpointer 2 ALLOCATOR = CFUNCTYPE(ndpointer('f4'), c_int, POINTER(c_int))

The following also seems to cause problems:

1 ALLOCATOR = CFUNCTYPE(POINTER(c_float), c_int, POINTER(c_int)) 2 ALLOCATOR = CFUNCTYPE(c_void_p, c_int, POINTER(c_int)) 3 ALLOCATOR = CFUNCTYPE(None, c_int, POINTER(c_int), POINTER(c_void_p))

Possible failures include a SystemError exception being raised, the interpreter crashing or the interpreter hanging. Check these mailing list threads for more details:

Time for an example. The C code for the example:

1 #ifndef CSPKREC_H 2 #define CSPKREC_H 3 #ifdef FOO_DLL 4 #ifdef FOO_EXPORTS 5 #define FOO_API __declspec(dllexport) 6 #else 7 #define FOO_API __declspec(dllimport) 8 #endif 9 #else 10 #define FOO_API 11 #endif 12 #endif 13 #include <stdio.h> 14 #ifdef __cplusplus 15 extern "C" { 16 #endif 17 18 typedef void*(allocator_t)(int, int); 19 20 extern FOO_API void foo(allocator_t allocator) { 21 int dim = 2; 22 int shape[] = {2, 3}; 23 float* data = NULL; 24 int i, j; 25 printf("foo calling allocator\n"); 26 data = (float*) allocator(dim, shape); 27 printf("allocator returned in foo\n"); 28 printf("data = 0x%p\n", data); 29 for (i = 0; i < shape[0]; i++) { 30 for (j = 0; j < shape[1]; j++) { 31 *data++ = (i + 1) * (j + 1); 32 } 33 } 34 } 35 36 #ifdef __cplusplus 37 } 38 #endif 39

Check the The Function Pointer Tutorials if you're new to function pointers in C or C++. And the Python code:

1 from ctypes import * 2 import numpy as N 3 4 allocated_arrays = [] 5 def allocate(dim, shape): 6 print 'allocate called' 7 x = N.zeros(shape[:dim], 'f4') 8 allocated_arrays.append(x) 9 ptr = x.ctypes.data_as(c_void_p).value 10 print hex(ptr) 11 print 'allocate returning' 12 return ptr 13 14 lib = cdll['callback.dll'] 15 lib.foo.restype = None 16 ALLOCATOR = CFUNCTYPE(c_long, c_int, POINTER(c_int)) 17 lib.foo.argtypes = [ALLOCATOR] 18 19 print 'calling foo' 20 lib.foo(ALLOCATOR(allocate)) 21 print 'foo returned' 22 23 print allocated_arrays[0]

The allocate function creates a new NumPy array and puts it in a list so that we keep a reference to it after the callback function returns. Expected output:

calling foo foo calling allocator allocate called 0xaf5778 allocate returning allocator returned in foo data = 0x00AF5778 foo returned [[ 1. 2. 3.] [ 2. 4. 6.]]

Here's another idea for an Allocator class to manage this kind of thing. In addition to dimension and shape, this allocator function takes a char indicating what type of array to allocate. You can get these typecodes from the ndarrayobject.h header, in the NPY_TYPECHAR enum.

1 from ctypes import * 2 import numpy as N 3 4 class Allocator: 5 CFUNCTYPE = CFUNCTYPE(c_long, c_int, POINTER(c_int), c_char) 6 7 def init(self): 8 self.allocated_arrays = [] 9 10 def call(self, dims, shape, dtype): 11 x = N.empty(shape[:dims], N.dtype(dtype)) 12 self.allocated_arrays.append(x) 13 return x.ctypes.data_as(c_void_p).value 14 15 def getcfunc(self): 16 return self.CFUNCTYPE(self) 17 cfunc = property(getcfunc)

Use it like this in Python:

1 lib.func.argtypes = [..., Allocator.CFUNCTYPE] 2 def func(): 3 alloc = Allocator() 4 lib.func(..., alloc.cfunc) 5 return tuple(alloc.allocated_arrays[:3])

Corresponding C code:

1 typedef void*(allocator_t)(int, int, char); 2 3 void func(..., allocator_t allocator) { 4
5 int dims[] = {2, 3, 4}; 6 double* data = (double*) allocator(3, dims, 'd'); 7
8 }

None of the allocators presented above are thread safe. If you have multiple Python threads calling the C code that invokes your callbacks, you will have to do something a bit smarter.

More useful code frags

Suppose you have a C function like the following, which operates on a pointer-to-pointers data structure.

1 void foo(float** data, int len) { 2 float** x = data; 3 for (int i = 0; i < len; i++, x++) { 4
5 } 6 }

You can create the necessary structure from an existing 2-D NumPy array using the following code:

1 x = N.array([[10,20,30], [40,50,60], [80,90,100]], 'f4') 2 f4ptr = POINTER(c_float) 3 data = (f4ptrlen(x))([row.ctypes.data_as(f4ptr) for row in x])

f4ptr*len(x) creates a ctypes array type that is just large enough to contain a pointer to every row of the array.

Heterogeneous Types Example

Here's a simple example when using heterogeneous dtypes (record arrays).

But, be warned that NumPy recarrays and corresponding structs in C may not be congruent.

Also structs are not standardized across platforms ...In other words, be aware of padding issues!

sample.c

1 #include <stdio.h> 2 3 typedef struct Weather_t { 4 int timestamp; 5 char desc[12]; 6 } Weather; 7 8 void print_weather(Weather* w, int nelems) 9 { 10 int i; 11 for (i=0;i<nelems;++i) { 12 printf("timestamp: %d\ndescription: %s\n\n", w[i].timestamp, w[i].desc); 13 } 14 }

SConstruct

1 env = Environment() 2 env.Replace(CFLAGS=['-O2','-Wall','-ansi','-pedantic']) 3 env.SharedLibrary('sample', ['sample.c'])

sample.py

1 import numpy as N 2 import ctypes as C 3 4 dat = [[1126877361,'sunny'], [1126877371,'rain'], [1126877385,'damn nasty'], [1126877387,'sunny']] 5 6 dat_dtype = N.dtype([('timestamp','i4'),('desc','|S12')]) 7 arr = N.rec.fromrecords(dat,dtype=dat_dtype) 8 9 _sample = N.ctypeslib.load_library('libsample','.') 10 _sample.print_weather.restype = None 11 _sample.print_weather.argtypes = [N.ctypeslib.ndpointer(dat_dtype, flags='aligned, contiguous'), C.c_int] 12 13 14 def print_weather(x): 15 _sample.print_weather(x, x.size) 16 17 18 19 if name=='main': 20 print_weather(arr)

Fibonacci example (using NumPy arrays, C and Scons)

The following was tested and works on Windows (using MinGW) and GNU/Linux 32-bit OSs (last tested 13-08-2009). Copy all three files to the same directory.

The C code (this calculates the Fibonacci number recursively):

1 2 3 4 5 6 7 8 int fib(int a); 9 void fibseries(int *a, int elements, int *series); 10 void fibmatrix(int *a, int rows, int columns, int *matrix); 11 12 int fib(int a) 13 { 14 if (a <= 0) 15 return -1; 16 else if (a==1) 17 return 0; 18 else if ((a==2)||(a==3)) 19 return 1; 20 else 21 return fib(a - 2) + fib(a - 1); 22 } 23 24 void fibseries(int *a, int elements, int *series) 25 { 26 int i; 27 for (i=0; i < elements; i++) 28 { 29 series[i] = fib(a[i]); 30 } 31 } 32 33 void fibmatrix(int *a, int rows, int columns, int *matrix) 34 { 35 int i, j; 36 for (i=0; i<rows; i++) 37 for (j=0; j<columns; j++) 38 { 39 matrix[i * columns + j] = fib(a[i * columns + j]); 40 } 41 }

The Python code:

1 """ 2 Filename: fibonacci.py 3 Demonstrates the use of ctypes with three functions: 4 5 (1) fib(a) 6 (2) fibseries(b) 7 (3) fibmatrix(c) 8 """ 9 10 import numpy as nm 11 import ctypes as ct 12 13 14 15 16 _libfibonacci = nm.ctypeslib.load_library('libfibonacci', '.') 17 18 _libfibonacci.fib.argtypes = [ct.c_int] 19 _libfibonacci.fib.restype = ct.c_int
20 21 _libfibonacci.fibseries.argtypes = [nm.ctypeslib.ndpointer(dtype = nm.int),
22 ct.c_int,
23 nm.ctypeslib.ndpointer(dtype = nm.int)] 24 _libfibonacci.fibseries.restype = ct.c_void_p 25 26 _libfibonacci.fibmatrix.argtypes = [nm.ctypeslib.ndpointer(dtype = nm.int),
27 ct.c_int, ct.c_int,
28 nm.ctypeslib.ndpointer(dtype = nm.int)] 29 _libfibonacci.fibmatrix.restype = ct.c_void_p 30 31 def fib(a): 32 """Compute the n'th Fibonacci number. 33 34 ARGUMENT(S): 35 An integer. 36 37 RESULT(S): 38 The n'th Fibonacci number. 39 40 EXAMPLE(S): 41 >>> fib(8) 42 13 43 >>> fib(23) 44 17711 45 >>> fib(0) 46 -1 47 """ 48 return _libfibonacci.fib(int(a)) 49 50 def fibseries(b): 51 """Compute an array containing the n'th Fibonacci number of each entry. 52 53 ARGUMENT(S): 54 A list or NumPy array (dim = 1) of integers. 55 56 RESULT(S): 57 NumPy array containing the n'th Fibonacci number of each entry. 58 59 EXAMPLE(S): 60 >>> fibseries([1,2,3,4,5,6,7,8]) 61 array([ 0, 1, 1, 2, 3, 5, 8, 13]) 62 >>> fibseries(range(1,12)) 63 array([ 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55]) 64 """ 65 b = nm.asarray(b, dtype=nm.intc) 66 result = nm.empty(len(b), dtype=nm.intc) 67 _libfibonacci.fibseries(b, len(b), result) 68 return result 69 70 def fibmatrix(c): 71 """Compute a matrix containing the n'th Fibonacci number of each entry. 72 73 ARGUMENT(S): 74 A nested list or NumPy array (dim = 2) of integers. 75 76 RESULT(S): 77 NumPy array containing the n'th Fibonacci number of each entry. 78 79 EXAMPLE(S): 80 >>> from numpy import array 81 >>> fibmatrix([[3,4],[5,6]]) 82 array([[1, 2], 83 [3, 5]]) 84 >>> fibmatrix(array([[1,2,3],[4,5,6],[7,8,9]])) 85 array([[ 0, 1, 1], 86 [ 2, 3, 5], 87 [ 8, 13, 21]]) 88 """ 89 tmp = nm.asarray(c) 90 rows, cols = tmp.shape 91 c = tmp.astype(nm.intc) 92 result = nm.empty(c.shape, dtype=nm.intc) 93 _libfibonacci.fibmatrix(c, rows, cols, result) 94 return result

Here's the SConstruct file contents (filename: SConstruct):

1 env = Environment() 2 env.Replace(CFLAGS=['-O2', '-Wall', '-ansi', '-pedantic']) 3 env.SharedLibrary('libfibonacci', ['fibonacci.c'])

In Python interpreter (or whatever you use), do:

import fibonacci as fb fb.fib(8) 13 fb.fibseries([5,13,2,6] array([ 3, 144, 1, 5])

etc.

Pertinent Mailing List Threads

Some useful threads on the ctypes-users mailing list:

Thomas Heller's answers are particularly insightful.

Documentation

CategoryCookbook CategoryCookbook