[Python-Dev] PEP 432 progress: Python initalization (original) (raw)

Victor Stinner victor.stinner at gmail.com
Thu Dec 14 10:31:41 EST 2017

Currently, we have the following configuration options:

typedef struct { int ignore_environment; /* -E / int use_hash_seed; / PYTHONHASHSEED=x / unsigned long hash_seed; int _disable_importlib; / Needed by freeze_importlib */ const char allocator; / Memory allocator: _PyMem_SetupAllocators() / int dev_mode; / -X dev / int faulthandler; / -X faulthandler / int tracemalloc; / -X tracemalloc=N / int import_time; / -X importtime / int show_ref_count; / -X showrefcount / int show_alloc_count; / -X showalloccount / int dump_refs; / PYTHONDUMPREFS / int malloc_stats; / PYTHONMALLOCSTATS / int utf8_mode; / -X utf8 or PYTHONUTF8 environment variable */

wchar_t *module_search_path_env; /* PYTHONPATH environment variable */
wchar_t *home;          /* PYTHONHOME environment variable,
                           see also Py_SetPythonHome(). */
wchar_t *program_name;  /* Program name, see also Py_GetProgramName() */

} _PyCoreConfig;

and

typedef struct { int install_signal_handlers; PyObject argv; / sys.argv list, can be NULL */ PyObject module_search_path; / sys.path list */ PyObject warnoptions; / sys.warnoptions list, can be NULL */ PyObject xoptions; / sys._xoptions dict, can be NULL */ } _PyMainInterpreterConfig;

Victor

2017-12-14 16:16 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:

Hi,

Serhiy Storchaka seems to be worried by the high numbers of commits in https://bugs.python.org/issue32030 "PEP 432: Rewrite PyMain()", so let me explain the context of this work :-) To prepare CPython to implement my UTF-8 Mode PEP (PEP 540), I worked on the implementation of Nick Coghlan's PEP 432: PEP 432 -- Restructuring the CPython startup sequence https://www.python.org/dev/peps/pep-0432/ The startup sequence is a big pile of code made of multiple functions: main(), PyMain(), PyInitialize(), PyFinalize()... and a lot of tiny "configuration" functions like PySetPath(). Over the years, many configuration options were added in the middle of the code. The priority of configuration options is not always correct between command line options, envrionment variables, configuration files (like "pyenv.cfg"), etc. For technical reasons, it's hard to impement properly the -E option (ignore PYTHON* environment variables). For example, the new PYTHONCOERCECLOCALE environment variable (of PEP 538) doesn't handle properly -E (it ignores -E), because it was too complex to support -E. -- I'm working on fixing this. Last weeks, I mostly worked on the PyMain() function, Modules/getpath.c and PC/getpathp.c, to "refactor" the code: * Split big functions (300 to 500 lines) into multiple small functions (50 lines or less), to make it easily to follow the control flow and to allow to more easily move code * Replace static and global variables with memory allocated on the heap. * Reorganize how the configuration is read: populate a first temporary structure (PyMain using wchart*), then create Python objects (PyMainInterpreterConfig) to finish with the real configuration (like setting attributes of the sys module). The goal is to centralize all code reading configuration to fix the priority and to simplify the code. My motivation was to write a correct implementation of the UTF-8 Mode (PEP 540). Nick's motivation is to make CPython easily to embed. His plan for Python 3.8 is to give access to the new PyCoreConfig and PyMainInterpreterConfig structures to: * easily give access to most (if not all?) configuration options to "embedders" * allow to configure Python without environment variables, command line options, configuration files, but only using these structures * allow to configure Python using Python objects (PyObject*) rather than C types (like wchart*) (I'm not sure that I understood correctly, so please read the PEP 432 ;-)) IMHO the most visible change of the PEP 432 is to split Python initialization in two parts: * Core: strict minimum to use the Python C API * Main: everything else The goal is to introduce the opportunity to configure Python between Core and Main. The implementation is currently a work-in-progress. Nick will not have the bandwidth, neither do I, to update his PEP and finish the implementation, before Python 3.7. So this work remains private until at least Python 3.8. Another part of the work is to enhance the documentation. You can for example now find an explicit list of C functions which can be called before PyInitialize(): https://docs.python.org/dev/c-api/init.html#before-python-initialization And also a list of functions that must not be called before PyInitialize(), whereas you might want to call them :-) Victor

More information about the Python-Dev mailing list