C API: My plan to clarify private vs public functions in Python 3.13 (original) (raw)
Hi,
In issue #106320, I removed many private C API functions in Python 3.13. It does break many C extensions projects on purpose. Here is my plan for the Python 3.13 development cycle. In short, it should be mostly fine before Python 3.13 beta1, otherwise we just follow the backup plan (revert all removals).
Phase 1: now, before alpha 1
- Remove as many private API (functions, variables, macros) as possible, as early as possible: in practice, it only means to move these API to the internal C API. Not really “remove” them, only “remove” them from the public C API.
- Identify affected C extensions: code search (I already started to do that), build projects with Python 3.13 (I just tried to build numpy: I created an issue for Cython).
- At least report the issue to affected projects, or even propose a fix.
- Consider adding a new clean (tested, documented, maybe better API) functions replacing removed private functions.
Phase 2: Python 3.13 alpha 1 (October 2023)
- Target: Python 3.13 alpha 1 (October 2023)
- For most popular projects like Cython, numpy and pip: revert removals to unblock these projects if no release can be shipped with a fix in the meanwhile. If possible, I would prefer to only revert removals once these projects are fixed upstream (got a fix merged). The alpha1 should be usable a minimum!
Phase 3: Python 3.13 beta 1 (May 2024)
- Target: Python 3.13 beta 1 (May 2024), but honestly doing this way before will be better. It’s just that I don’t know how much I can do alone so I prefer being pessimistic
- For remaining broken C extensions: decide what to do with removals causing most troubles. Propose a tool to automate migration to the new API? Add new public API to replace the removed one? Revert removals to give more time to C extensions maintainers? Something else?
Why? No, really, why, Victor?
Well, maybe I want your life to be a pain 
No, seriously: there is an ongoing work on fixing C API issues (new C API working group!). We would like to fix issues one by one, but it will take time since we want to reduce the number of impacted projects (smooth migration). If the C API is too broad, it’s hard to do this work.
Private API has multiple issues: they are usually not documented, not tested, and so their behavior may change without any warning or anything. Also, they can be removed anytime without any notice. Core developers don’t know how these functions are used, and so it’s hard to enhance them and maintain them.
My idea is to proactively go through the long list of private functions: see how they are used, see if anyone uses it, clarify the difference between private and public functions. My long goal would be to only provide clean public functions (tested, documented, follow regular PEP 387 deprecation process) in the public API, and put anything else in the internal C API. An alternative is to put some functions in the new PyUnstable C API, PEP 689 – Unstable C API tier.
As I wrote, I would like to convert some private functions to well tested and well documented public functions. IMO this work is important. During this conversion, we have to follow the latest C API guidelines to provide an even better API than before.
To convert a project to a new better public API, this project cannot lose support for old Python versions! That’s the problem of the idea of starting a new C API and leave the current C API unchanged: new API are only supported by new Python version. The solution here is to provide the new functions on old Python versions: that’s exactly what the pythoncapi-compat project does, header file providing these functions as static inline functions for old Python versions, up to Python 2.7!
In previous Python releases, I already followed a similar migration plan for deprecated functions: remove all deprecated functions planned for removal as early as possible at the beginning of a new development cycle, and then slowly reconsider reverting some removals once affected projects are aware of the issue (report the issue, or better fix the issue, but maybe there is no release yet), to unblock these projects.
The plan is to get a beta1 release as functional as possible, but allow some bumpy road between now and beta1.
In the worst case, the backup plan is simple: revert removals. In fact, I didn’t remove the function implementation. I only moved their definition from the public C API to the internal C API (move from Include/ or Include/cpython/ to Include/internal/). A revert is just to do the opposite again.
Obviously, anyone is welcomed to help me in this work: identify affected projects, report issues to these projects, propose patches, propose converting private functions to public functions, etc.
So. What do you think of this plan? Is anyone interested to help me?