[Python-Dev] New Python Initialization API (original) (raw)

Ivan Pozdeev vano at mail.mipt.ru
Thu Mar 28 06:24:10 EDT 2019

On 27.03.2019 20:48, Victor Stinner wrote:

Hi,

I would like to add a new C API to initialize Python. I would like your opinion on the whole API before making it public. The code is already implemented. Doc of the new API: https://pythondev.readthedocs.io/initconfig.html

To make the API public, PyWstrList, PyInitError, PyPreConfig, PyCoreConfig and related functions should be made public. By the way, I would suggest to rename "PyCoreConfig" to just "PyConfig" :-) I don't think that "core init" vs "main init" is really relevant: more about that below. Let's start with two examples using the new API. Example of simple initialization to enable isolated mode: PyCoreConfig config = PyCoreConfigINIT; config.isolated = 1; PyInitError err = PyInitializeFromConfig(&config); By my outsider observation, the config' argument and return code are asking to be added to Py_Initialize instead, _Py_InitializeFromConfigand_Py_PreInitialize` look redundant. if (PyINITFAILED(err)) { PyExitInitError(err); } /* ... use Python API here ... */ PyFinalize(); Example using the pre-initialization to enable the UTF-8 Mode (and use the "legacy" PyInitialize() function): PyPreConfig preconfig = PyPreConfigINIT; preconfig.utf8mode = 1; PyInitError err = PyPreInitialize(&preconfig); if (PyINITFAILED(err)) { PyExitInitError(err); } /* at this point, Python will only speak UTF-8 */ PyInitialize(); /* ... use Python API here ... */ PyFinalize(); Since November 2017, I'm refactoring the Python Initialization code to cleanup the code and prepare a new ("better") API to configure Python Initialization. I just fixed the last issues that Nick Coghlan asked me to fix (add a pre-initialization step: done, fix mojibake: done). My work is inspired by Nick Coghlan's PEP 432, but it is not implementing it directly. I had other motivations than Nick even if we are somehow going towards the same direction. Nick wants to get a half-initialized Python ("core init"), configure Python using the Python API and Python objects, and then finish the implementation ("main init"). I chose a different approach: put everything into a single C structure (PyCoreConfig) using C types. Using the structure, you should be able to do what Nick wanted to do, but with C rather than Python. Nick: please tell me if I'm wrong :-) This work is also connected to Eric Snow's work on sub-interpreters (PEP 554) and moving global variables into structures. For example, I'm using his PyRuntime structure to store a new "preconfig" state (pre-initialization configuration, more about that below). In November 2017, when I started to work on the Python Initialization (bpo-32030), I identified the following problems: * Many parts of the code were interdependent * Code executed early in PyMain() used the Python API before the Python API was fully initialized. Like code parsing -W command line option which used PyUnicodeFromWideChar() and PyListAppend(). * Error handling used PyFatalError() which didn't let the caller to decide how to handle the error. Moreover, exit() was used to exit Python, whereas libpython shouldn't do that: a library should not exit the whole process! (imagine when Python is embedded inside an application) One year and a half later, I implemented the following solutions: * PyMain() and PyInitialize() code has been reorganized to respect priorities between global configuration variables (ex: PyIgnoreEnvironmentFlag), environment variables (ex: PYTHONPATH), command line arguments (ex: -X utf8), configuration files (ex: pyenv.cfg), and the new PyPreConfig and PyCoreConfig structures which store the whole configuration. * Python Initialization no longer uses the Python API but only C types like wchart* strings, a new PyWstrList structure and PyMemRawMalloc() memory allocator (PyMemMalloc() is no longer used during init). * The code has been modified to use a new PyInitError structure. The caller of the top function gets control to cleanup everything before handling the error (display a fatal error message or simply exit Python). The new PyCoreConfig structure has the top-priority and provides a single structure for all configuration parameters. It becomes possible to override the code computing the "path configuration" like sys.path to fully control where Python looks to import modules. It becomes possible to use an empty list of paths to only allow builtin modules. A new "pre-initialization" steps is responsible to configure the bare minimum before the Python initialization: memory allocators and encodings (LCCTYPE locale and the UTF-8 mode). The LCCTYPE is no longer coerced and the UTF-8 Mode is no longer enabled automatically depending on the user configuration to prevent mojibake. Previously, calling PyDecodeLocale() to get a Unicode wchart* string from a bytes wchar* string created mojibake when called before PyInitialize() if the LCCTYPE locale was coerced and/or if the UTF-8 Mode was enabled. The pre-initialization step ensures that the encodings and memory allocators are well defined before PyInitialize() is called. Since the new API is currently private, I didn't document it in Python. Moreover, the code changed a lot last year :-) But it should now be way more stable. I started to document it in a separated webpage: https://pythondev.readthedocs.io/initconfig.html The plan is to put it in the Python documentation once it becomes public. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- Regards, Ivan

More information about the Python-Dev mailing list