[Python-Dev] PEP 384: Defining a Stable ABI (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Mon May 25 23:04:58 CEST 2009

M.-A. Lemburg wrote:

Now, with the PEP, I have a feeling that the Python C-API will in effect be limited to what's in the PEP's idea of a usable ABI and open up the non-inluded public C-APIs to the same rate of change as the private APIs.

Not really - before this PEP it was already fairly easy to write an extension that was source-level compatible with multiple versions of Python (depending on exactly what you wanted to do, of course).

However, it is essentially impossible to make an extension that is binary level compatible with multiple versions.

With the defined stable ABI in place, each extension module author will be able to make a choice:

An optional cross-version ABI would certainly be a good thing.

Limiting the Python C-API would be counterproductive.

I don't think anyone would disagree with that. A discussion on C-API sig would certainly be a good idea.

During the compilation of applications, the preprocessor macro PyLIMITEDAPI must be defined. Doing so will hide all definitions that are not part of the ABI. So extensions wanting to use the full Python C-API as documented in the C-API docs will still be able to do this, right ?

Yep - they just wouldn't define the new macro.

Type Objects ------------

The structure of type objects is not available to applications; declaration of "static" type objects is not possible anymore (for applications using this ABI). Hmm, that's going to create big problems for extensions that want to expose a C-API for their types: Type checks are normally done by pointer comparison using those static type objects.

They would just have to expose "MyExtensionPrefix_MyType_Check" and "MyExtensionPrefix_MyType_CheckExact" functions the same way that types in the C API do.

Functions and function-like Macros ----------------------------------

Function-like macros (in particular, field access macros) remain available to applications, but get replaced by function calls (unless their definition only refers to features of the ABI, such as the various Check macros) Including PyINCREF()/PyDECREF() ?

I believe so - MvL deliberately left the fields that the ref counting relies on as part of the ABI.

Excluded Functions ------------------

Functions declared in the following header files are not part of the ABI: - cellobject.h - classobject.h - code.h - frameobject.h - funcobject.h - genobject.h - pyarena.h - pydebug.h - symtable.h - token.h - traceback.h I don't think that's feasable: you basically remove all introspection functions that way. This will need a more fine-grained approach.

I don't think it is reasonable to expect the introspection interfaces to remain stable at a binary level across versions.

Having "I want deep introspection support from C" and "I want to use a single binary for multiple Python versions" be mutually exclusive choices sounds like a perfectly sensible position to me.

Also, keep in mind that even an extension module that restricts itself to Py_LIMITED_API would still be able to call in to the Python equivalents via PyObject_Call and friends (e.g. by importing and using the inspect and traceback modules).

What if you mix extensions that use the full C-API with ones that restrict themselves to the limited version ?

Would creating a Python object in a full-API extension and free'ing it in a limited-API extension cause problems ?

Possibly, if you end up mixing C runtimes in the process. Specifically:

  1. Python linked with MSVCRT X
  2. Full extension module linked with MSVCRT Y
  3. Limited extension module linked with MSVCRT Z

The PyMem/PyObject APIs in the limited extension module will use the heap in MSVCRT X, since they will be redirected through the Python stable ABI as function calls. However, if the full extension module uses the macro forms and links with the wrong MSVCRT version, then you have the usual opportunities for conflicts between the two C runtimes.

This isn't a problem created by defining a stable ABI though - it's the main reason mixing C runtimes is a bad idea. (The two others we have noted so far being IO issues, especially attempting to share FILE* instances and the fact that changing the locale will only affect whichever runtime the extension module linked against).

Good point. As a separate issue, I would actually like to deprecate, then remove these APIs. I had originally hoped that this would happen for 3.0 already, alas, nobody worked on it.

In any case, I have removed them from the ABI now. How do you expect Python extensions to allocate memory and objects in a platform independent way without those APIs ? And as an aside: Which API families are you referring to ? PyMemMalloc, PyObjectMalloc, or PyObjectNew ?

The ones with a FILE* parameter in the signature. There's no problem with the PyMem/PyObject functions since those will be redirected to consistently use the version of the C runtime that Python was originally linked against (their macro counterparts are obviously off limits for the stable ABI).

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list