Design a new better C API for Python — pythoncapi 0.1 documentation (original) (raw)

Subtitle: “Make the stable API usable”.

PyPy and Gilectomy projects are using different approaches to optimize Python, but their performance are limited by the current C API on C extensions. The C API should be “fixed” to unlock raw performances!

To be able to introduce backward incompatible changes to the C API without breaking too many C extensions, this project proposes two things:

• design a helper layer providing removed functions;
• a new Python runtime which is only usable with C extensions compiled with the new stricter and smaller C API (and the new stable ABI) for Python 3.8 and newer, whereas the existing “regular python” becomes the “regular runtime” which provides maximum backward compatibility with Python 3.7 and older.

The current C API has multiple issues:

• The Python lifecycle is shorter than the lifecycle of some operating systems: how to get the latest Python on an “old” but stable operating system?
• Python debug build is currently mostly unusable in practice, making development of C extension harder, especially debugging.

Existing C extensions will still be supported and will not have to be modified. The old C API is not deprecated and there is no plan penalize users of the old C API.

Table of Contents¶

• Statistics on the Python C API
  • Functions
  • Variables
  • Line Numbers
  • File Numbers
  • Public functions defined as macros and static inline functions
  • Private functions defined as macros and static inline functions
  • Structures
• Status of the new Python C API
  • Status
• Rationale to change the Python C API
  • PyPy cpyext is slow
  • The stable ABI is not usable
  • Performance problem: How can we make Python 2x faster?
  • Keep backward compatibility
  • The Big Carrot
• Bad C API
  • Borrowed references
  • Duplicated functions
  • Only keep abstract functions?
  • Functions kept for backward compatibility
  • No public C functions if it can’t be done in Python
  • Array of pointers to Python objects (PyObject**)
  • PyDict_GetItem()
  • C structures
  • PyType_Ready() and setting directly PyTypeObject fields
  • Integer overflow
  • Functions stealing references
  • Open questions
  • For internal use only
• A new C API for Python
  • Design goals
  • Remove functions and macros
  • New functions
  • Non-goal
  • Hide implementation details
• Python runtimes
  • Regular Python: /usr/bin/python3
  • Debug runtime: /usr/bin/python3-dbg
  • New experimental runtime: python3-exp
  • Other Python implementations
  • Put your CPython fork here!
• Old C API
  • What is the Python C API?
  • Current Python C API
• Implement a PyTypeObject in C
  • Old C API
  • PyType_FromSpec()
  • Remove cross-version binary compatibility
  • PyStructSequence_InitType()
• Optimization ideas
  • Remove debug checks
  • Change the garbage collector and remove reference counting: unlikely
  • Remove the GIL: unlikely
  • Tagged pointers: doable
  • Copy-on-Write (CoW): doable
  • Transactional Memory: highly experimental
  • Specialized list for small integers
  • O(1) conversion of bytearray to bytes
  • And more!
• Fix the Python C API to optimize Python
  • Optimizations
  • Reasons why the C API prevents to optimize Python
• Backward compatibility
  • Backward compatibility with Python 3.7 and older
  • Forward compatibility with Python 3.8 and newer
  • Open question: support Python 2?
  • Open question: how to install the compatibility layers
• Supporting multiple Python versions per operating system release
• C API calling conventions
  • CPython 3.7 calling conventions
  • Argument Clinic
  • CPython PyArg_ParseTuple() and Py_BuildValue(), getargs.c
  • Summer 2018: 3 PEPs
  • New calling conventions?
• Python stable ABI
  • Links
  • Relationship between the C API and the ABI
  • CPython Py_LIMITED_API
  • Check for ABI changes
• Consumers of the Python C API
  • Popular C extensions using the C API
  • Popular modules using Cython
  • Debugging tools
• PyPy cpyext module
  • Source
  • Performance issue
  • Issue with borrowed references
  • Replace macros with functions
• Cython
• cffi
• Gilectomy
• Remove the C API
• Performance
  • Performance slowdown
• Split Include/ directory
• PyHandle
  • Email
  • POSIX and Windows API
• Deprecate old APIs
• Python Intermediate representation (IR)
• Reorganize Python “runtime”
  • Goal
  • What do we need?
  • Current state of the code (2019-05-24)
  • TODO
  • Out of the scope
  • Roots
  • Status (2019-05-24)
  • Links
• Opaque PyObject structure
• Implicit and explicit resource management
  • Pointer API
  • Array of objects
• Links
• Misc notes
  • Public structures of Python types in Python 3.12 (49)
  • Structures which don’t inherit from PyObject (50)