[Python-Dev] Python initialization and embedded Python (original) (raw)

M.-A. Lemburg mal at egenix.com
Thu Nov 23 04:37:59 EST 2017

On 18.11.2017 01:01, Victor Stinner wrote:

Hi,

The CPython internals evolved during Python 3.7 cycle. I would like to know if we broke the C API or not. Nick Coghlan and Eric Snow are working on cleaning up the Python initialization with the "on going" PEP 432: https://www.python.org/dev/peps/pep-0432/ Many global variables used by the "Python runtime" were move to a new single "PyRuntime" variable (big structure made of sub-structures). See Include/internal/pystate.h. A side effect of moving variables from random files into header files is that it's not more possible to fully initialize PyRuntime at "compilation time". For example, previously, it was possible to refer to local C function (functions declared with "static", so only visible in the current file). Now a new "initialization function" is required to must be called. In short, it means that using the "Python runtime" before it's initialized by PyRuntimeInitialize() is now likely to crash. For example, calling PyMemRawMalloc(), before calling PyRuntimeInitialize(), now calls the function NULL: dereference a NULL pointer, and so immediately crash with a segmentation fault.

To prevent a complete crash, would it be possible to initialize the struct entries to a generic function (or set of such functions with the right signatures), which then issue a message to stderr hinting to the missing call to _PyRuntime_Initialize() before terminating ?

I'm writing this email to ask if this change is an issue or not to embedded Python and the Python C API. Is it still possible to call "all" functions of the C API before calling PyInitialize()?

I was bitten by the bug while reworking the PyMain() function to split it into subfunctions and cleanup the code to handle the command line arguments and environment variables. I fixed the issue in main() by calling PyRuntimeInitialize() as soon as possible: it's now the first instruction of main() :-) (See Programs/python.c) To give a more concrete example: PyDecodeLocale() is the recommanded function to decode bytes from the operating system, but this function calls PyMemRawMalloc() which does crash before PyRuntimeInitialize() is called. Is PyDecodeLocale() used to initialize Python? For example, "void PySetProgramName(wchart *);" expects a text string, whereas main() gives argv as bytes. Calling PySetProgramName() from argv requires to decode bytes... So use PyDecodeLocale()... Should we do something in PyDecodeLocale()? Maybe crash if PyRuntimeInitialize() wasn't called yet? Maybe, the minimum change is to expose PyRuntimeInitialize() in the public C API? Victor

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Nov 23 2017)

Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/

More information about the Python-Dev mailing list