[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode (original) (raw)

Victor Stinner vstinner at redhat.com
Thu Apr 25 06:39:44 EDT 2019

Hi,

I'm now convinced that C extensions must not be linked to libpython on Unix.

I wrote PR 12946: https://github.com/python/cpython/pull/12946

bpo-21536: C extensions are no longer linked to libpython

On Unix, C extensions are no longer linked to libpython.

It is now possible to load a C extension built using a shared library
Python with a statically linked Python.

When Python is embedded, libpython must not be loaded with
RTLD_LOCAL, but RTDL_GLOBAL instead. Previously, using RTLD_LOCAL, it
was already not possible to load C extensions which were not linked
to libpython, like C extensions of the standard library built by the
"*shared*" section of Modules/Setup.

distutils, python-config and python-config.py have been modified.

This PR allows to load a C extension built by a shared libpython with a statically linked Python: https://bugs.python.org/issue21536#msg340819

It also allows to load C extension built in release mode with a Python built in debug mode: https://bugs.python.org/issue21536#msg340821

Le jeu. 25 avr. 2019 à 08:31, Nathaniel Smith <njs at pobox.com> a écrit :

In principle, having extension modules link to libpython.so is a good thing. Suppose that someone wants to dynamically load the python interpreter into their program as some kind of plugin. (Examples: Apache's modpython, LibreOffice's support for writing macros in Python.) It would be nice to be able to load python2 and python3 simultaneously into the same process as distinct plugins. And this is totally doable in theory, but it means that you can't assume that the interpreter's symbols will be automagically injected into extension modules, so it's only possible if extension modules link to libpython.so.

I'm aware of 2 special use cases of libpython:

(A) Embed Python using RTLD_LOCAL: dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW)

Example of issues describing this use case:

Python started to link C extensions to libpython in 2006 for this use case.

(B) Load "libpython2" (Python 2) and "libpython3" (Python 3).

I heard this idea... but I never saw anyone doing it in practice. I don't understand how it could work in a single address space.

Linking C extensions to libpython is causing different issues:

(1) C extension built by a shared library Python cannot be loaded with a statically linked Python: https://bugs.python.org/issue21536

(2) C extension built in release mode cannot be loaded with Python built in debug mode. That's the issue discussed in this thread ;-)

(3) C extension built by Python 3.6 cannot be loaded in Python 3.7, even if it has been compiled using the stable ABI (Py_LIMITED_API).

(4) C extensions of the standard library built by "shared" of Modules/Setup are not linked to libpython. For example, _struct.so on Fedora is not linked to libpython, whereas . If libpython is loaded with RTLD_LOCAL (use case A), import _struct fails.

The use case (A) (RTLD_LOCAL) is trivial to fix: replace RTLD_LOCAL with RTLD_GLOBAL.

The use case (B) (libpython2 + libpython3) is also easy to workaround: just use 2 separated processes. Python 2 will reach its end of life at the end of the year, I'm not sure that we should worry too much about this use case.

The issue (1) (statically/shared) is a very practical issue. Fedora/RHEL uses libpython whereas Debian/Ubuntu uses statically linked Python. C extension compiled on Fedora/RHEL is linked to libpython and so cannot be loaded on Debian/Ubuntu (their Python doesn't have libpython!). That's why manylinux forbid link to link to libpython: be able to load C extensions on all Linux distributions.

IMHO issues (1), (2), (3), (4) are more valuable to be fixed than supporting use cases (A) and (B).

Victor

Night gathers, and now my watch begins. It shall not end until my death.

More information about the Python-Dev mailing list