[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode (original) (raw)

Matthias Klose doko at ubuntu.com
Thu Apr 25 07:48:05 EDT 2019

On 25.04.19 13:14, Victor Stinner wrote:

Le jeu. 25 avr. 2019 à 09:34, Matthias Klose <doko at ubuntu.com> a écrit :

there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;) So depending on the package maintainer, you already have that available, but it is extra maintenance cost. Simplifying that would be a good idea. Fedora provides "debuginfo" for all binarry packages (like numpy), but that's different from a debug build. Usually, C code of packages are optimized by gcc -O2 or even gcc -O3 which makes the debugging experience very painful: gdb fails to read C local variables and just say "". To debug internals, you want a debug build compiled by gcc -Og or (better IMHO) gcc -O0. If you want to inspect Python internals but you don't need to inspect numpy internals, being able to run a release numpy on a debug Python is convenient.

yes, the Debian/Ubuntu packages contain both the debug build, and the debug info for they normal build, e.g.

/usr/lib/debug/.build-id/3a/8ea2ab6ee85ff68879a48170966873eb8da781.debug /usr/lib/debug/.build-id/78/5ff95f8d2d06c5990ae4e03cdff99452ca0de9.debug /usr/lib/debug/.build-id/92/e008cffa3f09106214bfb6b80b7fd02ceab74f.debug /usr/lib/debug/.build-id/ab/33160518c41acc0488bbc3af878995ef74e07f.debug /usr/lib/debug/.build-id/bd/65896626a4c6566e96ad008362922cf6a39cd6.debug /usr/lib/debug/.build-id/f1/e83b14a76dd9564e962dcdd2f70202e6fdb2b1.debug /usr/lib/debug/.build-id/ff/5eab5fd2d14f4bfa6a1ef2300358efdc7dd800.debug /usr/lib/python3/dist-packages/lxml/_elementpath.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/builder.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/etree.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/html/clean.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/html/diff.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/objectify.cpython-37dm-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/lxml/sax.cpython-37dm-x86_64-linux-gnu.so

With an additional change on SOABI (I will open a separated issue for that), my PR 12946 (no longer link C extensions to libpython) allows to load lxml built in release mode in a Python built in debug mode! That's very useful for debugging. I show an example of the gdb experience with a release Python vs debug Python:

https://bugs.python.org/issue21536#msg340821 With a release Python, the basic function "py-bt" works as expected, but inspecting Python internals doesn't work: most local C variables are "optimized out" :-( With a debug Python, the debugging experience is much better: it's possible to inspect Python internals!

However I still would like to be able to have "debug" and "non-debug" builds co-installable at the same time. One option is to keep "d" flag in the SOABI so C extensions get a different SO filename (no change compared to Python 3.7): "NAME.cpython-38-x8664-linux-gnu.so" for release vs "NAME.cpython-38d-x8664-linux-gnu.so" for debug, debug gets "d" suffix ("cpython-38" vs "cpython-38d"). But modify importlib when Python is compiled in debug mode to look also to SO without the "d" suffix: first try load "NAME.cpython-38d-x8664-linux-gnu.so" (debug: "d" suffix). If there is no match, look for "NAME.cpython-38-x8664-linux-gnu.so" (release: no suffix). Since the ABI is now compatible in Python 3.8, it should "just work" :-) From a Linux packager perspective, nothing changes ;-) We can still provide "apt install python3-numpy-dbg" (debug) which can is co-installable with "apt install python3-numpy" (release). The benefit is that it will be possible to load C extensions which are only available in the release flavor with a debug Python ;-)

yes, that sounds good. Are there use cases where you only want to load some debug extensions, even if more are installed?

More information about the Python-Dev mailing list