[C API] Remove token.h header file · Issue #92651 · python/cpython (original) (raw)

Symbols and functions exported by the <token.h> header file cannot be used outside Python internals. There is no public C API to tokenize a Python source file. I propose to simply remove this header file. It's not included by Python.h.

In Python 3.10, I removed the C API related to AST and the parser, since it was not possible to use it outside Python internals. See: https://docs.python.org/dev/whatsnew/3.10.html#id4