[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode (original) (raw)

Victor Stinner vstinner at redhat.com
Wed Apr 24 12:02:18 EDT 2019

Hum, I found issues with libpython: C extensions are explicitly linked to libpython built in release mode. So a debug python loading a C extension may load libpython in release mode, whereas libpython in debug mode is already loaded.

When Python is built with --enable-shared, the python3.7 program is linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly linked to libpython3.7m as well:

$ python3.7-config --ldflags ... -lpython3.7m ...

Example with numpy:

$ ldd /usr/lib64/python3.7/site-packages/numpy/core/umath.cpython-37m-x86_64-linux-gnu.so ... libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) ...

When Python 3.7 is compiled in debug mode, libpython gets a "d" flag for debug: libpython3.7dm.so.1.0.

I see 2 solutions:

(1) Use a different directory. If "libpython" gets the same filename in release and debug mode, at least, they must be installed in different directories. If libpython build in debug mode is installed in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug libpython.

(2) If "libpython" gets a different filename in debug mode, C extensions should not be linked to libpython explicitly but implicitly to avoid picking the wrong libpython. For example, remove "-lpython3.7m" from "python3.7-config --ldflags" output.

The option (1) rely on rpath which is discouraged by Linux vendors and may not be supported by all operating systems.

The option (2) is simpler and likely more portable.

Currently, C extensions of the standard library may or may not be linked to libpython depending on they are built. In practice, both work since python3.7 is already linked to libpython: so libpython is already loaded in memory before C extensions are loaded.

I opened https://bugs.python.org/issue34814 to discuss how C extensions of the standard library should be linked but I closed it because we failed to find a consensus and the initial use case became a non-issue. It seems like we should reopen the discussion :-)

Victor

More information about the Python-Dev mailing list