getargs.c (original) (raw)

M.-A. Lemburg mal at egenix.com
Fri Mar 26 10:20:08 CET 2010

Larry Hastings wrote:

M.-A. Lemburg wrote: Backporting PyCapsule is fine, but the changes you made to all those PyCObject uses does not look backwards compatible.

The C APIs exposed by the modules (e.g. the datetime module) are used in lots of 3rd party extension modules and changing them from PyCObject to PyCapsule is a major change in the module API. You're right, my changes aren't backwards compatible. I thought it was reasonable for four reasons:

Just as reminder of the process we have in place for such changes: Please discuss any major breakage on python-dev before checking in the patch.

1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a few lines by mixing and matching CObjects. Switching Python to capsules prevents a class of exploits. I've included a script at the bottom of this message that demonstrates three such crashes. The script runs in Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

That's good, but then again: deliberate wrong use of APIs will always cause crashes and at least I don't know of any report about PyCObjects posing a problem in all their years of existence.

2. As I just mentioned, Python 3.1 already uses capsules everywhere instead of CObjects. Since part of the purpose of Python 2.7 is to prepare developers for the to upgrade to 3.1, getting them to switch to capsules now is just one more way they are prepared.

Sure, but forcing them is not a good idea, even less so if you can easily expose both a PyCObject and PyCapsule interface to the same C API.

3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we ever made a 2.8 I want to remove it completely.

Please remember that PyCObjects are not only used internally in CPython, but also in other 3rd party modules to expose C APIs and those will generally have to support more than just the latest Python release.

If you deprecate those C APIs, the process will at least have to cover one more release, i.e. run through the whole deprecation process:

pending deprecation (2.7)
deprecation (2.8)
removal (2.9)

I think it's better to add a -3 warning when using PyCObjects in 2.7.

4. When Python publishes an API using a CObject, it describes the thing the CObject points to in a header file. In nearly all cases that header file also provides a macro or inline function that does the importing work for you. I changed those to use capsules too. So if the third-party code uses the macro or inline function, all you need do is recompile it against 2.7 and it works fine. Sadly I know of one exception: pyexpat.expatCAPI. The header file just describes the struct pointed to by the CObject, but callers

I know about those macros... I introduced that idea with mxDateTime and then added the same logic in CPython in a couple of places a long while ago, e.g. socketmodule.h. IIRC, Jim Fulton added PyCObjects for exchanging module C APIs a few years before that.

Later on PyCObject_Import() was added to simplify the C API import a bit.

A recompile is certainly a possibility to have 3rd party modules switch to the capsule interfaces of the stdlib modules, but we should still be more careful about this.

Wouldn't it be possible to have PyCObject_AsVoidPtr() et al. work with PyCapsules as well ?

After all, the caller is only interested in the pointer and doesn't really care about what object was used to wrap it.

I can suggest four ways to ameliorate the problem.

First, we could do as Antoine Pitrou suggests on the bug (issue 7992): wherever the CObject used to be published as a module attribute to expose an API, we could provide both a CObject and a capsule; internally Python would only use the capsules. This would allow third-party libraries to run against 2.7 unchanged. The major problem with this is that third-party libraries would still be vulnerable to the mix-and-match CObject crash.

True, but we've been happy with this vulnerability for years, just as we've been happy with the fact that it's easy to crash the VM by passing hand-crafted byte-code to it, or using ctypes to call an OS function with the wrong parameters, etc.

Like I said: there are many ways to deliberately crash Python.

We don't have a concept of interface signatures in Python, it's mostly based on trust.

A secondary, minor concern: obviously we'd store the CObject attribute with the existing name, and the capsule attribute would have to get some new name. But in Python 3.1, these attributes already expose a capsule. Therefore, people who convert to using the capsules now would have to convert again when moving to 3.1.

This should be manageable with some aliasing.

Second, we could make CObject internally support unpacking capsules. If you gave a capsule to PyCObjectAsVoidPtr() it would unpack it and return the pointer within. (We could probably also map the capsule "context" to the CObject "desc", if any of the Python use cases needed it.) I wouldn't change anything else about CObjects; creating and using them would continue to work as normal. This would also allow third-party libraries to run against Python 2.7 unchanged. The only problem is that it's unsafe, as indeed allowing any use of PyCObjectAsVoidPtr() is unsafe.

That's a good idea. Yes, playing with fire is unsafe, but fire is useful in a lot of places as well :-)

Third, I've been pondering writing a set of preprocessor macros, shipped in their own header file distributed independently of Python and released to the public domain, that would make it easy to use either CObjects or capsules depending on what version of Python you were compiling against. Obviously, using these macros would require a source code change in the third-party library. But these macros would make it a five-minute change. This could compliment the first or second approaches.

I'm not sure whether that would worth the effort. 3rd party modules for Python 2.x are likely not going to use them, since they typically have to support more than just Python 2.7.

E.g. look at Plone - they are still running on Python 2.4 and use lots of 3rd party modules. It took Zope years to switch from 2.4 to 2.6. You have the same situation with other large systems, whether OSS or not.

If a required 3rd party module would suddenly only support Python 2.7, developers would either have to find a replacement that (still) works for them (rather unlikely), or try to port their whole app to 2.7, which often enough is not easily possible due to other restrictions.

This aspect is often not considered when discussing such backwards incompatible changes - at least not in the early stages. Fortunately, we have so far - and most of the time - found a way to keep everyone happy.

Fourth, we could back out of the changes to published APIs and convert them back to CObjects. -1.

See above. That's still an option as well, but I agree that either making PyCObjects compatible with PyCapsules (via changes to the PyCObject functions) or having them exist side-by-side is a better option.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Mar 26 2010)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

More information about the Python-Dev mailing list