[Python-Dev] versioned .so files for Python 3.2 (original) (raw)

Barry Warsaw barry at python.org
Thu Jul 22 16:40:36 CEST 2010

On Jul 16, 2010, at 12:40 PM, Matthias Klose wrote:

I like the proposal, but IMO it is too unspecific about the abi tag. Assume that an extension is built with such a configured python and then tried to run with an abi compatible configured python, but with a slightly different version tag, the extension won't be found. Differing file names per configuration should be avoided. Proposing

1) Remove the configure option and use the new naming using the tag for all configurations unconditionally. Everybody can expect the same name on every installation, and PEP 384 most likely will require using a tag too. As Amaury did point out, there is a use case in that PyPy can use this tag too to build extensions only usable by PyPy. 2) As PEP 3147 defines a non-configurable name for .pyc files, this PEP should define a non-configurable way for the tag. The tag should include all information which currently makes an extension ABI incompatible: - the python implementation (cpython, PyPy, ...) - the python version (3,2, 3.3, ...) - unicode configure option (--with-wide-unicode, 16 or 32) - platform information (necessary?) If this list changes for coming python versions, then it can be extended. Barry pointed out on irc chat that people might want to build experimental ABI incompatible versions, which should have its own tag. If this is wanted, please provide a configure option which lets extend/append to the tag.

Okay, I'm convinced we need to hard code this tag, and I think it's okay not to provide a configure switch to control this. I've pushed a new version of the diff and PEP, the latter attached here for your convenience.

Thanks for all the feedback. -Barry

PEP: 3149 Title: ABI version tagged .so files Version: Revision:81577Revision: 81577 Revision:81577 Last-Modified: Date:2010−05−2719:54:25−0400(Thu,27May2010)Date: 2010-05-27 19:54:25 -0400 (Thu, 27 May 2010) Date:2010−05−2719:54:25−0400(Thu,27May2010) Author: Barry Warsaw <barry at python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2010-07-09 Python-Version: 3.2 Post-History: 2010-07-14, 2010-07-22 Resolution: TBD

Abstract

PEP 3147 [1]_ described an extension to Python's import machinery that improved the sharing of Python source code, by allowing more than one byte compilation file (.pyc) to be co-located with each source file.

This PEP defines an adjunct feature which allows the co-location of extension module files (.so) in a similar manner. This optional, build-time feature will enable downstream distributions of Python to more easily provide more than one Python major version at a time.

Background

PEP 3147 defined the file system layout for a pure-Python package, where multiple versions of Python are available on the system. For example, where the alpha package containing source modules one.py and two.py exist on a system with Python 3.2 and 3.3, the post-byte compilation file system layout would be::

alpha/
    __pycache__/
        __init__.cpython-32.pyc
        __init__.cpython-33.pyc
        one.cpython-32.pyc
        one.cpython-33.pyc
        two.cpython-32.pyc
        two.cpython-33.pyc
    __init__.py
    one.py
    two.py

For packages with extension modules, a similar differentiation is needed for the module's .so files. Extension modules compiled for different Python major versions are incompatible with each other due to changes in the ABI. Different configuration/compilation options for the same Python version can result in different ABIs (e.g. --with-wide-unicode).

While PEP 384 [2]_ defines a stable ABI, it will minimize, but not eliminate extension module incompatibilities between Python builds or major versions. Thus a mechanism for discriminating extension module file names is proposed.

Rationale

Linux distributions such as Ubuntu [3]_ and Debian [4]_ provide more than one Python version at the same time to their users. For example, Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, with Python 2.6 being the default.

In order to share as much as possible between the available Python versions, these distributions install third party (i.e. non-standard library) packages into /usr/share/pyshared and symlink to them from /usr/lib/pythonX.Y/dist-packages. The symlinks exist because in a pre-PEP 3147 world (i.e < Python 3.2), the .pyc files resulting from byte compilation by the various installed Pythons will name collide with each other. For Python versions >= 3.2, all pure-Python packages can be shared, because the .pyc files will no longer cause file system naming conflicts. Eliminating these symlinks makes for a simpler, more robust Python distribution.

A similar situation arises with shared library extensions. Because extension modules are typically named foo.so for a foo extension module, these would also name collide if foo was provided for more than one Python version.

In addition, because different configuration/compilation options for the same Python version can cause different ABIs to be presented to extension modules. On POSIX systems for example, the configure options --with-pydebug, --with-pymalloc, and --with-wide-unicode all change the ABI. This PEP proposes to encode build-time options in the file name of the .so extension module files.

PyPy [5]_ can also benefit from this PEP, allowing it to avoid name collisions in extension modules built for its API, but with a different .so tag.

Proposal

The configure/compilation options chosen at Python interpreter build-time will be encoded in the shared library file name for extension modules. This "tag" will appear between the module base name and the operation file system extension for shared libraries.

The following information MUST be included in the shared library file name:

These two fields are separated by a hyphen and no dots are to appear between the major and minor version numbers. E.g. cpython-32.

Python implementations MAY include additional flags in the file name tag as appropriate. For example, on POSIX systems these flags will also contribute to the file name:

By default in Python 3.2, configure enables --with-pymalloc so shared library file names would appear as foo.cpython-32m.so. When the other two flags are also enabled, the file names would be foo.cpython-32dmu.so.

(This PEP only addresses build issues on POSIX systems that use the configure script. While Windows or other platform support is not explicitly disallowed under this PEP, platform expertise is needed in order to evaluate, describe, and implement support on such platforms.)

The shared library file name tag is used unconditionally; it cannot be changed. The tag and extension module suffix are available through the sysconfig modules via the following variables::

>>> sysconfig.get_config_var('SO')
'.cpython-32mu.so'
>>> sysconfig.get_config_var('SOABI')
'cpython-32mu'

Note that $SOABI contains just the tag, while $SO includes the platform extension for shared library files, and is the exact suffix added to the extension module name.

For an arbitrary package foo, you would see these files when the distribution package was installed::

/usr/share/pyshared/foo.cpython-32m.so
/usr/share/pyshared/foo.cpython-33m.so

Python's dynamic module loader will recognize and import shared library extension modules with a tag that matches its build-time options. For backward compatibility, Python will also continue to import untagged extension modules, e.g. foo.so.

This shared library tag would be used globally for all distutils-based extension modules, regardless of where on the file system they are built. Extension modules built by means other than distutils would either have to calculate the tag manually, or fallback to the non-tagged .so file name.

Proven approach

The approach described here is already proven, in a sense, on Debian and Ubuntu system where different extensions are used for debug builds of Python and extension modules. Debug builds on Windows also already use a different file extension for dynamic libraries, and in fact encoded (in a different way than proposed in this PEP) the Python major and minor version in the .dll file name.

PEP 384

PEP 384 defines a stable ABI for extension modules. In theory, universal adoption of PEP 384 would eliminate the need for this PEP because all extension modules could be compatible with any Python version. In practice of course, it will be impossible to achieve universal adoption, and as described above, different built-time flags still affect the ABI. Thus even with a stable ABI, this PEP may still be necessary. While a complete specification is reserved for PEP 384, here is a discussion of the relevant issues.

PEP 384 describes a change to PyModule_Create() where 3 is passed as the API version if the extension was complied with Py_LIMITED_API. This should be formalized into an official macro called PYTHON_ABI_VERSION to mirror PYTHON_API_VERSION. If and when the ABI changes in an incompatible way, this version number would be bumped. To facilitate sharing, Python would be extended to search for extension modules with the PYTHON_ABI_VERSION number in its name. The prefix abi is reserved for Python's use.

Thus, an initial implementation of PEP 384, when Python is configured with the default set of flags, would search for the following file names when extension module foo is imported (in this order)::

foo.cpython-XYm.so
foo.abi3.so
foo.so

The distutils [6]_ build_ext command would also have to be extended to compile to shared library files with the abi3 tag, when the module author indicates that their extension supports that version of the ABI. This could be done in a backward compatible way by adding a keyword argument to the Extension class, such as::

Extension('foo', ['foo.c'], abi=3)

Alternatives

In the initial python-dev thread [7]_ where this idea was first introduced, several alternatives were suggested. For completeness they are listed here, along with the reasons for not adopting them.

Debian and Ubuntu could simply add a version-specific directory to sys.path that would contain just the extension modules for that version of Python. Or the symlink trick eliminated in PEP 3147 could be retained for just shared libraries. This approach is rejected because it propagates the essential complexity that PEP 3147 tries to avoid, and adds potentially several additional directories to search for all modules, even when the number of extension modules is much fewer than the total number of Python packages. For example, builds were made available both with and without wide unicode, with and without pydebug, and with and without pymalloc, the total number of directories search increases substantially.

Don't share packages with extension modules

It has been suggested that Python packages with extension modules not be shared among all supported Python versions on a distribution. Even with adoption of PEP 3149, extension modules will have to be compiled for every supported Python version, so perhaps sharing of such packages isn't useful anyway. Not sharing packages with extensions though is infeasible for several reasons.

If a pure-Python package is shared in one version, should it suddenly be not-shared if the next release adds an extension module for speed? Also, even though all extension shared libraries will be compiled and distributed once for every supported Python, there's a big difference between duplicating the .so files and duplicating all .py files. The extra size increases the download time for such packages, and more immediately, increases the space pressures on already constrained distribution CD-ROMs.

Reference implementation

Work on this code is tracked in a Bazaar branch on Launchpad [8]_ until it's ready for merge into Python 3.2. The work-in-progress diff can also be viewed [9]_ and is updated automatically as new changes are uploaded.

References

.. [1] PEP 3147

.. [2] PEP 384

.. [3] Ubuntu: <http://www.ubuntu.com>

.. [4] Debian: <http://www.debian.org>

.. [5] http://codespeak.net/pypy/dist/pypy/doc/

.. [6] http://docs.python.org/py3k/distutils/index.html

.. [7] http://mail.python.org/pipermail/python-dev/2010-June/100998.html

.. [8] https://code.edge.launchpad.net/~barry/python/sovers

.. [9] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411

Copyright

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20100722/8a639650/attachment.pgp>

More information about the Python-Dev mailing list