[Python-Dev] PEP 587: Python Initialization Configuration (original) (raw)

Victor Stinner vstinner at redhat.com
Wed Mar 27 21:30:17 EDT 2019

Hi,

Since Steve Dower asked me to write a formal PEP for my proposal of a new C API to initialize Python, here you have!

This new PEP will be shortly rendered as HTML at: https://www.python.org/dev/peps/pep-0587/

The Rationale might be a little bit weak at this point and there are still 3 small open questions, but the whole PEP should give you a better overview of my proposal than my previous email :-)

I chose to start with a new PEP rather than updating PEP 432. I didn't feel able to just "update" the PEP 432, my proposal is different. The main difference between my PEP 587 and Nick Coghlan's PEP 432 is that my PEP 587 only allows to configure Python before its initialization, whereas Nick wanted to give control between its "Initializing" and "Initialized" initialization phases. Moreover, my PEP only uses C types rather Nick wanted to use Python types.

By setting the private PyConfig._init_main field to 1, my PEP 587 allows to stop Python initialization early to get a bare minimum working Python only with builtin types and no importlib. If I understood correctly, with my PEP, it remains possible to extend Python C API later to implement what Nick wants. Nick: please correct me if I'm wrong :-)

Victor

PEP: 587 Title: Python Initialization Configuration Author: Victor Stinner <vstinner at redhat.com> Discussions-To: python-dev at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Mar-2018 Python-Version: 3.8

Abstract

Add a new C API to configure the Python Initialization providing finer control on the whole configuration and better error reporting.

Rationale

Python is highly configurable but its configuration evolved organically: configuration parameters is scattered all around the code using different ways to set them (mostly global configuration variables and functions). A straightforward and reliable way to configure Python is needed. Some configuration parameters are not accessible from the C API, or not easily.

The C API of Python 3.7 Initialization takes wchar_t* strings as input whereas the Python filesystem encoding is set during the initialization.

Python Initialization C API

This PEP proposes to add the following new structures, functions and macros.

New structures (4):

New functions (8):

New macros (6):

PyWideCharList

PyWideCharList is a list of wchar_t* strings.

Example to initialize a string from C static array::

static wchar_t* argv[2] = {
    L"-c",
    L"pass",
};
PyWideCharList config_argv = PyWideCharList_INIT;
config_argv.length = Py_ARRAY_LENGTH(argv);
config_argv.items = argv;

PyWideCharList structure fields:

If length is non-zero, items must be non-NULL and all strings must be non-NULL.

.. note:: The "WideChar" name comes from the existing Python C API. Example: PyUnicode_FromWideChar(const wchar_t *str, Py_ssize_t size).

PyInitError

PyInitError is a structure to store an error message or an exit code for the Python Initialization.

Example::

PyInitError alloc(void **ptr, size_t size)
{
    *ptr = PyMem_RawMalloc(size);
    if (*ptr == NULL) {
        return Py_INIT_NO_MEMORY();
    }
    return Py_INIT_OK();
}

int main(int argc, char **argv)
{
    void *ptr;
    PyInitError err = alloc(&ptr, 16);
    if (Py_INIT_FAILED(err)) {
        Py_ExitInitError(err);
    }
    PyMem_Free(ptr);
    return 0;
}

PyInitError fields:

Macro to create an error:

Other macros and functions:

Pre-Initialization with PyPreConfig

PyPreConfig structure is used to pre-initialize Python:

Example using the pre-initialization to enable the UTF-8 Mode::

PyPreConfig preconfig = PyPreConfig_INIT;
preconfig.utf8_mode = 1;

PyInitError err = Py_PreInitialize(&preconfig);
if (Py_INIT_FAILED(err)) {
    Py_ExitInitError(err);
}

/* at this point, Python will speak UTF-8 */

Py_Initialize();
/* ... use Python API here ... */
Py_Finalize();

Functions to pre-initialize Python:

These functions can be called with config set to NULL. The caller is responsible to handler error using Py_INIT_FAILED() and Py_ExitInitError().

PyPreConfig fields:

The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are disabled by default in PyPreConfig. Set coerce_c_locale, coerce_c_locale_warn and utf8_mode to -1 to let Python enable them depending on the user configuration.

Initialization with PyConfig

The PyConfig structure contains all parameters to configure Python.

Example of Python initialization enabling the isolated mode::

PyConfig config = PyConfig_INIT;
config.isolated = 1;

PyInitError err = Py_InitializeFromConfig(&config);
if (Py_INIT_FAILED(err)) {
    Py_ExitInitError(err);
}
/* ... use Python API here ... */
Py_Finalize();

Functions to initialize Python:

These functions can be called with config set to NULL. The caller is responsible to handler error using Py_INIT_FAILED() and Py_ExitInitError().

PyConfig fields:

There are also private fields which are for internal-usage only:

New Py_UnixMain() function

Python 3.7 provides a high-level Py_Main() function which requires to pass command line arguments as wchar_t* strings. It is non-trivial to use the correct encoding to decode bytes. Python has its own set of issues with C locale coercion and UTF-8 Mode.

This PEP adds a new Py_UnixMain() function which takes command line arguments as bytes::

int Py_UnixMain(int argc, char **argv)

Memory allocations and Py_DecodeLocale()

New pre-initialization and initialization APIs use constant PyPreConfig or PyConfig structures. If memory is allocated dynamically, the caller is responsible to release it. Using static strings is just fine.

Python memory allocation functions like PyMem_RawMalloc() must not be used before Python pre-initialization. Using malloc() and free() is always safe.

Py_DecodeLocale() must only be used after the pre-initialization.

XXX Open Questions

This PEP is still a draft with open questions which should be answered:

Backwards Compatibility

This PEP only adds a new API: it leaves the existing API unchanged and has no impact on the backwards compatibility.

Alternative: PEP 432

This PEP is inspired by Nick Coghlan's PEP 432 with a main difference: it only allows to configure Python before its initialization.

The PEP 432 uses three initialization phases: Pre-Initialization, Initializing, Initialized. It is possible to configure Python between Initializing and Initialized phases using Python objects.

This PEP only uses C types like int and wchar_t* (and PyWideCharList structure). All parameters must be configured at once before the Python initialization using the PyConfig structure.

Annex: Python Configuration

Priority and Rules

Priority of configuration parameters, highest to lowest:

Priority of warning options, highest to lowest:

Rules on PyConfig and PyPreConfig parameters:

Configuration Files

Python configuration files:

Global Configuration Variables

Global configuration variables mapped to PyPreConfig fields:

======================================== ================================ Variable Field ======================================== ================================ Py_LegacyWindowsFSEncodingFlag legacy_windows_fs_encoding Py_LegacyWindowsFSEncodingFlag legacy_windows_fs_encoding Py_UTF8Mode utf8_mode Py_UTF8Mode utf8_mode ======================================== ================================

Global configuration variables mapped to PyConfig fields:

======================================== ================================ Variable Field ======================================== ================================ Py_BytesWarningFlag bytes_warning Py_DebugFlag parser_debug Py_DontWriteBytecodeFlag write_bytecode Py_FileSystemDefaultEncodeErrors filesystem_errors Py_FileSystemDefaultEncoding filesystem_encoding Py_FrozenFlag _frozen Py_HasFileSystemDefaultEncoding filesystem_encoding Py_HashRandomizationFlag use_hash_seed, hash_seed Py_IgnoreEnvironmentFlag use_environment Py_InspectFlag inspect Py_InteractiveFlag interactive Py_IsolatedFlag isolated Py_LegacyWindowsStdioFlag legacy_windows_stdio Py_NoSiteFlag site_import Py_NoUserSiteDirectory user_site_directory Py_OptimizeFlag optimization_level Py_QuietFlag quiet Py_UnbufferedStdioFlag buffered_stdio Py_VerboseFlag verbose _Py_HasFileSystemDefaultEncodeErrors filesystem_errors Py_BytesWarningFlag bytes_warning Py_DebugFlag parser_debug Py_DontWriteBytecodeFlag write_bytecode Py_FileSystemDefaultEncodeErrors filesystem_errors Py_FileSystemDefaultEncoding filesystem_encoding Py_FrozenFlag _frozen Py_HasFileSystemDefaultEncoding filesystem_encoding Py_HashRandomizationFlag use_hash_seed, hash_seed Py_IgnoreEnvironmentFlag use_environment Py_InspectFlag inspect Py_InteractiveFlag interactive Py_IsolatedFlag isolated Py_LegacyWindowsStdioFlag legacy_windows_stdio Py_NoSiteFlag site_import Py_NoUserSiteDirectory user_site_directory Py_OptimizeFlag optimization_level Py_QuietFlag quiet Py_UnbufferedStdioFlag buffered_stdio Py_VerboseFlag verbose _Py_HasFileSystemDefaultEncodeErrors filesystem_errors ======================================== ================================

Py_LegacyWindowsFSEncodingFlag and Py_LegacyWindowsStdioFlag are only available on Windows.

Command Line Arguments

Usage::

python3 [options]
python3 [options] -c COMMAND
python3 [options] -m MODULE
python3 [options] SCRIPT

Command line options mapped to pseudo-action on PyConfig fields:

================================ ================================ Option PyConfig field ================================ ================================ -b bytes_warning++ -B write_bytecode = 0 -c COMMAND run_module = COMMAND --check-hash-based-pycs=MODE _check_hash_pycs_mode = MODE -d parser_debug++ -E use_environment = 0 -i inspect++ and interactive++ -I isolated = 1 -m MODULE run_module = MODULE -O optimization_level++ -q quiet++ -R use_hash_seed = 0 -s user_site_directory = 0 -S site_import -t ignored (kept for backwards compatibility) -u buffered_stdio = 0 -v verbose++ -W WARNING add WARNING to warnoptions -x skip_source_first_line = 1 -X XOPTION add XOPTION to xoptions ================================ ================================

-h, -? and -V options are handled outside PyConfig.

Environment Variables

Environment variables mapped to PyPreConfig fields:

================================= ============================================= Variable PyPreConfig field ================================= ============================================= PYTHONCOERCECLOCALE coerce_c_locale, coerce_c_locale_warn PYTHONDEVMODE dev_mode PYTHONLEGACYWINDOWSFSENCODING legacy_windows_fs_encoding PYTHONMALLOC allocator PYTHONUTF8 utf8_mode ================================= =============================================

Environment variables mapped to PyConfig fields:

================================= ==================================== Variable PyConfig field ================================= ==================================== PYTHONDEBUG parser_debug PYTHONDEVMODE dev_mode PYTHONDONTWRITEBYTECODE write_bytecode PYTHONDUMPREFS dump_refs PYTHONEXECUTABLE program_name PYTHONFAULTHANDLER faulthandler PYTHONHASHSEED use_hash_seed, hash_seed PYTHONHOME home PYTHONINSPECT inspect PYTHONIOENCODING stdio_encoding, stdio_errors PYTHONLEGACYWINDOWSSTDIO legacy_windows_stdio PYTHONMALLOCSTATS malloc_stats PYTHONNOUSERSITE user_site_directory PYTHONOPTIMIZE optimization_level PYTHONPATH module_search_path_env PYTHONPROFILEIMPORTTIME import_time PYTHONPYCACHEPREFIX, pycache_prefix PYTHONTRACEMALLOC tracemalloc PYTHONUNBUFFERED buffered_stdio PYTHONVERBOSE verbose PYTHONWARNINGS warnoptions ================================= ====================================

PYTHONLEGACYWINDOWSFSENCODING and PYTHONLEGACYWINDOWSSTDIO are specific to Windows.

PYTHONDEVMODE is mapped to PyPreConfig.dev_mode and PyConfig.dev_mode.

Annex: Python 3.7 API

Python 3.7 has 4 functions in its C API to initialize and finalize Python:

Python can be configured using scattered global configuration variables (like Py_IgnoreEnvironmentFlag) and using the following functions:

There is also a high-level Py_Main() function.

Copyright

This document has been placed in the public domain.

More information about the Python-Dev mailing list