[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode (original) (raw)

Matthias Klose doko at ubuntu.com
Thu Apr 25 03:14:27 EDT 2019

On 25.04.19 08:31, Nathaniel Smith wrote:

You don't necessarily need rpath actually. The Linux loader has a bug/feature where once it has successfully loaded a library with a given soname, then any future requests for that soname within the same process will automatically return that same library, regardless of rpath settings etc. So as long as the main interpreter has loaded libpython.whatever from the correct directory, then extension modules will all get that same version. The rpath won't matter at all.

It is annoying in general that on Linux, we have these two different ways to build extension modules. It definitely violates TOOWTDI :-). It would be nice at some point to get rid of one of them. Note that we can't get rid of the two different ways entirely though – on Windows, extension modules must link to libpython.dll, and on macOS, extension modules can't link to libpython.dylib. So the best we can hope for is to make Linux consistently do one of these, instead of supporting both. In principle, having extension modules link to libpython.so is a good thing. Suppose that someone wants to dynamically load the python interpreter into their program as some kind of plugin. (Examples: Apache's modpython, LibreOffice's support for writing macros in Python.) It would be nice to be able to load python2 and python3 simultaneously into the same process as distinct plugins. And this is totally doable in theory, but it means that you can't assume that the interpreter's symbols will be automagically injected into extension modules, so it's only possible if extension modules link to libpython.so. In practice, extension modules have never consistently linked to libpython.so, so everybody who loads the interpreter as a plugin has already worked around this. Specifically, they use RTLDGLOBAL to dump all the interpreter's symbols into the global namespace. This is why you can't have python2 and python3 modpython at the same time in the same Apache. And since everyone is already working around this, linking to libpython.so currently has zero benefit... in fact manylinux wheels are actually forbidden to link to libpython.so, because this is the only way to get wheels that work on every interpreter.

extensions in Debian/Ubuntu packages are not linked against libpython.so, but the main reason here is that sometimes you have to extensions built in transition periods like for 3.6 and 3.7. And this is also the default when not configuring with --enable-shared.

More information about the Python-Dev mailing list