[Python-Dev] Make the stable API-ABI usable (original) (raw)

Koos Zevenhoven k7hoven at gmail.com
Sat Nov 18 09:13:28 EST 2017

Your email didn't compile. The compiler says that it's a naming conflict, but actually I think you forgot a semicolon! (Don't worry, it happens to all of us, whether we be happy or not :-)

--Koos

On Sat, Nov 18, 2017 at 3:05 AM, Victor Stinner <victor.stinner at gmail.com> wrote:

Hi,

tl; dr I propose to extend the existing "stable API" to make it almost as complete as the current API. For example, add back PyTupleGETITEM() to be stable API, but it becomes a function call rather than a macro. The final question is if it's not too late to iterate on an implementation of this idea for Python 3.7? Knowing that the stable API doesn't affect the "current API" at all, since the "new C API" (extended stable API) would only be accessible using an opt-in flag.

Since I failed to find time to write a proper PEP (sorry about that), and the incoming deadline for Python 3.7 features is getting closer, I decided to dump my dump into this email. Sorry about the raw formatting. It's a third turn of feedback. The first one was done in python-ideas last July, the second was at the CPython sprint last September at Instagram. Both were positive. https://mail.python.org/pipermail/python-ideas/2017-July/046399.html The C API of CPython is amazing. It made CPython as powerful as we know it today. Without the C API, Python wouldn't have numpy. Without numpy, Python wouldn't be as popular as it is nowadays. Ok, enough for the introduction. The C API is awful. I hate it so much! The C API is old, error prone, too big, and expose every single CPython implementation detail... The C API is likely the first reason why faster implementation of Python (PyPy?) are not as popular as they should be. The fact that the C API is so widely used prevents many evolutions of CPython. CPython is stuck by its own (public) C API. PyPy developers spent a lot of time to make cffi great and advertize it all around the world. While it's a great project, the fact is that the C API is still used and the motivation to rewrite old code written with the C API to cffi, Cython or whateer else is too low. Maybe because the carrot is not big enough. Let's make a bigger carrot! (bigger than PyPy amazing performances? I'm not sure that it's doable :-( we will see) Using Cython, it's simpler to write C extensions and Cython code is more "portable" on different Python versions, since the C API evolved the last 10 years. For example, Python 3 renamed PyString with PyBytes and dropped the PyInt type. Writing a C extension working on Python 2 and Python 3 requires to "pollute" the code with many #ifdef. Cython helps to reduce them (or even avoid them? sorry, I don't know well Cython). The C API is tidely linked to the ABI. I tried to explain it with an example in my article: https://vstinner.github.io/new-python-c-api.html A known ABI issue is that it's not possible to load C extensions compiled in "release mode" on a Python compiled in "debug mode". The debug mode changes the API in a subtle way which changes the ABI and so makes C extensions incompatible. This issue prevents many people to use a Python debug build, whereas debug builds are very helpful to detect bugs earlier. Another issue is that C extensions must be recompiled for each Python release (3.5, 3.6, 3.7, etc.). Linux vendors like Red Hat cannot provide a single binary for multiple Python versions which prevents to upgade the "system" Python in the lifecycle of a distribution major version (or at least, it makes things much more complicated in term of packaging, it multiply packages for each Python version...). . . . Don't worry (be happy!), I'm not writing this email to complain, but to propose a solution. Aha! I propose to modify the API step by step to add more functions to the "stable ABI" (PEP 384) to be able to compile more and more C extensions using the "stable API" (please try to follow, first it was B, now it's P... seriously, the difference between the ABI and the API is subtle, to simplify let's say that it's the same thing, ok? :-)). I wrote a draft PEP, but I never found time to update it after the two rounds of feedbacks (python-ideas and the sprint) to write a proper PEP. So I will only give a link to my draft, sorry! https://github.com/vstinner/misc/blob/master/python/pepcapi.rst In short, the plan is to add back the PyTupleGETITEM() API to the "stable API" but change its implementation to a function call rather than the existing macro, so the compiled C extension will use a function call and so don't rely on the ABI anymore. My plan is to have two main milestones: (1) Python 3.7: Extend the existing opt-in "stable API" which requires to compile C extensions in a special mode. Add maybe an option in distutils to ease the compilation of a C extension with the "stable API"? (2) In Python 3.8, --if the project is successful and the performance overhead is acceptable compared the advantages of having C extensions working on multiple Python verisons--, make the "stable API (without implementation details)" the default, but add a new opt-in option to give access to the "full API (with implementation details)" for debuggers and other people who understand what they do (like Cython?). Note: currently, the "stable API" is accessible using PyLIMITEDAPI define, and the "full API" is accessible using PyBUILDCORE define. No define gives the current C API. My problem is more on the concrete implementation: * Need to provide two different API using the same filenames (like: #include "Python.h") * Need to extend distutils to have a flag to compile a C extension with one specific API (define PyLIMITEDAPI or PyBUILDCORE?) * Need to test many C extensions and check how many extensions are broken My plan for Python 3.7 is to not touch the current API at all. There is no risk of backward incompatibility. You should only get issues if you opt-in for the new API without implementation details. Final note: Nothing new under the sun: PyPy already implemented my "idea"! Where the idea is a C API without macros; PyTupleGETITEM() is already a function call in PyPy ;-) Final question: Is it acceptable to iterate on many small changes on the C API implement this idea in Python 3.7? Maybe only write partial implementation, and finish it in Python 3.8? Victor

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ k7hoven%40gmail.com

--

More information about the Python-Dev mailing list