[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode (original) (raw)

Ivan Pozdeev vano at mail.mipt.ru
Tue Apr 23 20:50:25 EDT 2019

On 24.04.2019 2:44, Victor Stinner wrote:

Hi,

Two weeks ago, I started a thread "No longer enable PyTRACEREFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal... Let me retry from scratch with a more explicit title: I would like to be able to run C extensions compiled in release mode on a Python compiled in debug mode ("pydebug").

This is going to be impossible because debug Python links against debug C runtime which is binary incompatible with the release one (at least, in Windows).

The use case is to debug bugs in C extensions thanks to additional runtime checks of a Python debug build, and more generally get a better debugging experiences on Python. Even for pure Python, a debug build is useful (to get the Pyhon traceback in gdb using "py-bt" command). That said, debug vs release extension compilation is currently bugged. It's impossible to make a debug build of an extension against a release Python (linked against release runtime, so not fully debug, just without optimizations) and vice versa. pip fails to build extensions for a debug Python for the same reason. I've no idea how (and if at all) people manage to diagnose problems in extensions. https://bugs.python.org/issue33637

Currently, using a Python compiled in debug mode means to have to recompile C extensions in debug mode. Compile a C extension requires a C compiler, header files, pull dependencies, etc. It can be very complicated in practical (and pollute your system with all these additional dependencies). On Linux, it's already hard, but on Windows it can be even harder. Just one concrete example: no debug build of numpy is provided at https://pypi.org/project/numpy/ Good luck to build numpy in debug mode manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.) :-) The above paragraph is probably the reason ;-) -- The first requirement for the use case is that a Python debug build supports the ABI of a release build. The current blocker issue is that the PyDEBUG define imply the PyTRACEREFS define: PyObject gets 2 extra fields (obprev and obnext) which change the offset of all attributes of all objects and makes the ABI completely incompatible. I propose to no longer imply PyTRACEREFS by default (but keep the code): https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615 (PyTRACEREFS would be a different ABI.) The second issue is that library filenames are different for a debug build: SOABI gets an additional "d" flag for PyDEBUG. A debug build should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not possible: a debug build contains many additional functions missing from a release build. For Windows, maybe we should provide a Python compiled in debug mode with the same C Runtime than a Python compiled in release mode. Otherwise, the debug C Runtime is causing another ABI issue. Maybe pip could be enhanced to support installing C extensions compiled in release mode when using a debug mode. But that's more for convenience, it's not really required, since it is easy to switch the Python runtime between release and debug build. Apart of PyTRACEREFS, I'm not aware of other ABI differences in structures. I know that the COUNTALLOCS define changes the ABI, but it's not implied by PyDEBUG: you have to opt-in for COUNTALLOCS. (I propose to do the same for PyTRACEREFS ;-)) Note: Refleaks buildbots don't use PyTRACEREFS to track memory leaks, only sys.gettotalrefcount(). -- Python debug build has many benefit. If you ignore C extensions, the debug build is usually compiled with compiler optimization disabled which makes debugging in gdb a much better experience. If you never tried: on a release build, most (if not all) variables are "" and it's really painful to basic debug functions like displaying the current Python frame. Assertions are removed in release modes, whereas they can detect a wide range of bugs way earlier: integer overflow, buffer under- and overflow, exceptions ignored silently, etc. Nobody likes to see a bug for the first time in production. For example, I modified Python 3.8 to now logs I/O errors when a file is closed implicitly, but only in debug or development mode. In release Python silently ignored EBADF error on such case, whereas it can lead to very nasty bugs causing Python to call abort() (which creates a coredump on Linux): see https://bugs.python.org/issue18748 ... DeprecationWarning and ResourceWarning are shown by default in debug mode :-) There are too many different additional checks done at runtime: I cannot list them all here. -- Being able to switch between Python in release mode and Python in debug mode is a first step. My long term plan would be to better separate "Python" from its "runtime". CPython in release mode would be one runtime, CPython in debug mode would be another runtime, PyPy can seeen as another runtime, etc. The more general idea is: "compile your C extension once and use any Python runtime". https://pythoncapi.readthedocs.io/runtimes.html#runtimes If you opt-in for the stable ABI, you can already switch between runtimes of different Python versions (ex: Python 3.6 or Python 3.8). Victor -- Night gathers, and now my watch begins. It shall not end until my death.

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- Regards, Ivan

More information about the Python-Dev mailing list