[Python-Dev] PEP 382: Namespace Packages (original) (raw)

"Martin v. Löwis" martin at v.loewis.de
Thu Apr 2 17:32:02 CEST 2009

I propose the following PEP for inclusion to Python 3.1. Please comment.

Regards, Martin

Abstract

Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages path must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package.

Terminology

Within this PEP, the term package refers to Python packages as defined by Python's import statement. The term distribution refers to separately installable sets of Python modules as stored in the Python package index, and installed by distutils or setuptools. The term vendor package refers to groups of files installed by an operating system's packaging mechanism (e.g. Debian or Redhat packages install on Linux systems).

The term portion refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package.

Namespace packages today

Python currently provides the pkgutil.extend_path to denote a package as a namespace package. The recommended way of using it is to put::

    from pkgutil import extend_path
    __path__ = extend_path(__path__, __name__)

int the package's __init__.py. Every distribution needs to provide the same contents in its __init__.py, so that extend_path is invoked independent of which portion of the package gets imported first. As a consequence, the package's __init__.py cannot practically define any names as it depends on the order of the package fragments on sys.path which portion is imported first. As a special feature, extend_path reads files named *.pkg which allow to declare additional portions.

setuptools provides a similar function pkg_resources.declare_namespace that is used in the form::

import pkg_resources
pkg_resources.declare_namespace(__name__)

In the portion's init.py, no assignment to path is necessary, as declare_namespace modifies the package path through sys.modules. As a special feature, declare_namespace also supports zip files, and registers the package name internally so that future additions to sys.path by setuptools can properly add additional portions to each package.

setuptools allows declaring namespace packages in a distribution's setup.py, so that distribution developers don't need to put the magic path modification into init.py themselves.

Rationale

The current imperative approach to namespace packages has lead to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports *.pkg files; setuptools doesn't. Likewise, setuptools supports inspecting zip files, and supports adding portions to its _namespace_packages variable, whereas pkgutil doesn't.

In addition, the current approach causes problems for system vendors. Vendor packages typically must not provide overlapping files, and an attempt to install a vendor package that has a file already on disk will fail or cause unpredictable behavior. As vendors might chose to package distributions such that they will end up all in a single directory for the namespace package, all portions would contribute conflicting init.py files.

Specification

Rather than using an imperative mechanism for importing packages, a declarative approach is proposed here, as an extension to the existing *.pkg mechanism.

The import statement is extended so that it directly considers *.pkg files during import; a directory is considered a package if it either contains a file named init.py, or a file whose name ends with ".pkg".

In addition, the format of the *.pkg file is extended: a line with the single character * indicates that the entire sys.path will be searched for portions of the namespace package at the time the namespace packages is imported.

Importing a package will immediately compute the package's path; the *.pkg files are not considered anymore after the initial import. If a *.pkg package contains an asterisk, this asterisk is prepended to the package's path to indicate that the package is a namespace package (and that thus further extensions to sys.path might also want to extend path). At most one such asterisk gets prepended to the path.

extend_path will be extended to recognize namespace packages according to this PEP, and avoid adding directories twice to path.

No other change to the importing mechanism is made; searching modules (including init.py) will continue to stop at the first module encountered.

Discussion

With the addition of *.pkg files to the import mechanism, namespace packages can stop filling out the namespace package's init.py. As a consequence, extend_path and declare_namespace become obsolete.

It is recommended that distributions put a file .pkg into their namespace packages, with a single asterisk. This allows vendor packages to install multiple portions of namespace package into a single directory, with no risk of overlapping files.

Namespace packages can start providing non-trivial init.py implementations; to do so, it is recommended that a single distribution provides a portion with just the namespace package's init.py (and potentially other modules that belong to the namespace package proper).

The mechanism is mostly compatible with the existing namespace mechanisms. extend_path will be adjusted to this specification; any other mechanism might cause portions to get added twice to path.

Copyright

This document has been placed in the public domain.

More information about the Python-Dev mailing list