[Python-Dev] SWIG (was Re: Ctypes and the stdlib) (original) (raw)

David Cournapeau cournape at gmail.com
Mon Aug 29 20:37:31 CEST 2011

On Mon, Aug 29, 2011 at 7:14 PM, Eli Bendersky <eliben at gmail.com> wrote:

I've sometimes thought it might be interesting to create a Swig replacement purely in Python.  When I work on the PLY project, this is often what I think about.   In that project, I've actually built a number of the parsing tools that would be useful in creating such a thing.   The only catch is that when I start thinking along these lines, I usually reach a point where I say "nah, I'll just write the whole application in Python." Anyways, this is probably way more than anyone wants to know about Swig. Getting back to the original topic of using it to make standard library modules, I just don't know.   I think you probably could have some success with an automatic code generator of some kind.  I'm just not sure it should take the Swig approach of parsing C++ headers.  I think you could do better. Dave, Having written a full C99 parser (http://code.google.com/p/pycparser/) based on your (excellent) PLY library, my impression is that the problem is with the problem, not with the solution. Strange sentence, I know :-) What I mean is that parsing C++ (even its headers) is inherently hard, which is why the solutions tend to grow so complex. Even with the modest C99, clean and simple solutions based on theoretical approaches (like PLY with its generated LALR parsers) tend to run into walls [*]. C++ is an order of magnitude harder. If I went to implement something like SWIG today, I would almost surely base my implementation on Clang (http://clang.llvm.org/). They have a full C++ parser (carefully hand-crafted, quite admirably keeping a relatively comprehensible code-base for such a task) used in a real compiler front-end, and a flexible library structure aimed at creating tools. There are also Python bindings that would allow to do most of the interesting Python-interface-specific work in Python - parse the C++ headers using Clang's existing parser into ASTs - then generate ctypes / extensions from that, in Python. The community is also gladly accepting contributions. I've had some fixes committed for the Python bindings and the C interfaces that tie them to Clang, and got the impression from Clang's core devs that further contributions will be most welcome. So whatever is missing from the Python bindings can be easily added.

Agreed, I know some people have looked into that direction in the scientific python community (to generate .pxd for cython). I wrote one of the hack Stefan refered to (based on ctypeslib using gccxml), and using clang makes so much more sense.

To go back to the initial issue, using cython to wrap C code makes a lot of sense. In the scipy community, I believe there is a broad agreement that most of code which would requires C/C++ should be done in cython instead (numpy and scipy already do so a bit). I personally cannot see man situations where writing wrappers in C by hand works better than cython (especially since cython handles python2/3 automatically for you).

cheers,

David

More information about the Python-Dev mailing list