[Python-Dev] PEP 384: Defining a Stable ABI (original) (raw)

M.-A. Lemburg mal at egenix.com
Tue May 26 18:42:37 CEST 2009

Martin v. Löwis wrote:

Now, with the PEP, I have a feeling that the Python C-API will in effect be limited to what's in the PEP's idea of a usable ABI and open up the non-inluded public C-APIs to the same rate of change as the private APIs. That's certainly not the plan. Instead, the plan is to have a stable ABI. The policy on the API isn't affected, except for restricting changes to the API that would break the ABI.

Thanks for clarifying this.

During the compilation of applications, the preprocessor macro PyLIMITEDAPI must be defined. Doing so will hide all definitions that are not part of the ABI. So extensions wanting to use the full Python C-API as documented in the C-API docs will still be able to do this, right ? Correct. They would link to the version-specific DLL on Windows.

Good.

The structure of type objects is not available to applications; declaration of "static" type objects is not possible anymore (for applications using this ABI). Hmm, that's going to create big problems for extensions that want to expose a C-API for their types: Type checks are normally done by pointer comparison using those static type objects. I don't see the problem. During module initialization, you create the type object and store it in a global variable, and then both clients and the module compare against the stored pointer.

Ah, good point !

Function-like macros (in particular, field access macros) remain available to applications, but get replaced by function calls (unless their definition only refers to features of the ABI, such as the various Check macros) Including PyINCREF()/PyDECREF() ? Yes, although some people are requesting that these become functions.

I'd opt against that, simply because it creates a lot of overhead due to the function call and issues with cache locality.

Excluded Functions ------------------

Functions declared in the following header files are not part of the ABI: - cellobject.h - classobject.h - code.h - frameobject.h - funcobject.h - genobject.h - pyarena.h - pydebug.h - symtable.h - token.h - traceback.h I don't think that's feasable: you basically remove all introspection functions that way. This will need a more fine-grained approach. What specifically is it that you want to do in a module that you couldn't do anymore?

See my reply to Nick: some of the functions are needed even if you don't want to do introspection, such as Py_FatalError() or PyTraceBack_Print().

BTW: Given the headline, I take it that the various type checking macros in these header will still be available, right ?

On Windows, applications shall link with python3.dll; You mean: extensions that were compiled with PyLIMITEDAPI, right ? Correct, see "Terminology" in the PEP.

Good, thanks.

an import library python3.lib will be available. This DLL will redirect all of its API functions through /export linker options to the full interpreter DLL, i.e. python3y.dll. What if you mix extensions that use the full C-API with ones that restrict themselves to the limited version ? Some link against python3.dll, others against python32.dll (say). Would creating a Python object in a full-API extension and free'ing it in a limited-API extension cause problems ? No problem that I can see.

Can we be sure that the MSVCRT used by python35.dll stays compatible to the one used by say python32.dll ? What if the CRT memory management changes between MSVCRT versions ?

Another aspect to consider:

How will this work in the light of having multiple copies of Python installed on a Windows machine ?

They implementation section suggests that python3.dll would always redirect to the python3x.dll for which it was installed, ie. if I have Python 3.5 installed, but then need to run some app with Python 3.2, the installed python3.dll would then point back to the python32.dll.

Now, if I start a Python 3.5 application which uses a limited API extension, this would try to load python32.dll into the Python 3.5 process. AFAIK, that's not possible due to the naming conflicts.

This PEP will be implemented in a branch, allowing users to check whether their modules conform to the ABI. To simplify this testing, an additional macro PyLIMITEDAPIWITHTYPES will expose the existing type object layout, to let users postpone rewriting all types. When the this branch is merged into the 3.2 code base, this macro will be removed. Now I'm confused again: this sounds a lot like you do want all extension writers to only use the limited API. I certainly want to support as many modules as reasonable with the PEP. Whether or not developers then chose to build version-independent binaries is certainly outside the scope of the PEP - it only specifies action items for Python, not for application authors.

Thanks for the clarification.

Something I haven't seen explicitly mentioned as yet (in the PEP or the

python-dev list discussion) are the memory management APIs and the FILE* APIs which can cause the MSVCRT versioning issues on Windows.

Those would either need to be excluded from the stable ABI or else changed to use opaque pointers. Good point. As a separate issue, I would actually like to deprecate, then remove these APIs. I had originally hoped that this would happen for 3.0 already, alas, nobody worked on it. In any case, I have removed them from the ABI now. How do you expect Python extensions to allocate memory and objects in a platform independent way without those APIs ? I have only removed functions from the ABI that have FILE* in their signatures. And as an aside: Which API families are you referring to ? PyMemMalloc, PyObjectMalloc, or PyObjectNew ? Neither. PyRunAnyFileFlags and friends.

Good.

Thanks,

Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, May 26 2009)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2009-06-29: EuroPython 2009, Birmingham, UK 33 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

More information about the Python-Dev mailing list