[Python-Dev] PEP 382: Namespace Packages (original) (raw)

glyph at divmod.com glyph at divmod.com
Fri Apr 17 05:58:22 CEST 2009

On 16 Apr, 03:36 pm, pje at telecommunity.com wrote:

At 03:46 AM 4/16/2009 +0000, glyph at divmod.com wrote:

On 15 Apr, 09:11 pm, pje at telecommunity.com wrote:

Twisted has its own system for "namespace" packages, and I'm not really sure where we fall in this discussion. I haven't been able to follow the whole thread, but my original understanding was that the PEP supports "defining packages", which we now seem to be calling "base packages", just fine. Yes, it does. The discussion since the original proposal, however, has been dominated by MAL's counterproposal, which requires a defining package.

[snip clarifications]

Does that all make sense now?

Yes. Thank you very much for the detailed explanation. It was more than I was due :-).

MAL's proposal requires a defining package, which is counterproductive if you have a pure package with no base, since it now requires you to create an additional project on PyPI just to hold your defining package.

Just as a use-case: would the Java "com.*" namespace be an example of a "pure package with no base"? i.e. lots of projects are in it, but no project owns it?

I'd appreciate it if the PEP could also be extended cover Twisted's very similar mechanism for namespace packages,

"twisted.plugin.pluginPackagePaths". I know this is not quite as widely used as setuptools' namespace package support, but its existence belies a need for standardization.

The PEP also seems a bit vague with regard to the treatment of other directories containing init.py and *.pkg files. Do you have a clarification to suggest? My understanding (probably a projection) is that to be a nested namespace package, you have to have a parent namespace package.

Just to clarify things on my end: "namespace package" to me means "package with modules provided from multiple distributions (the distutils term)". The definition provided by the PEP, that a package is spread over multiple directories on disk, seems like an implementation detail.

Entries on path slow down import, so my understanding of the platonic ideal of a system python installation is one which has a single directory where all packages reside, and a set of metadata off to the side explaining which files belong to which distributions so they can be uninstalled by a package manager.

Of course, for a development installation, easy uninstallation and quick swapping between different versions of relevant dependencies is more important than good import performance. So in that case, you would want to optimize differently by having all of your distributions installed into separate directories, with a long PYTHONPATH or lots of .pth files to point at them.

And of course you may want a hybrid of the two.

So another clarification I'd like in the PEP is an explanation of motivation. For example, it comes as a complete surprise to me that the expectation of namespace packages was to provide only single-source namespaces like zope., peak., twisted.. As I mentioned above, I implicitly thought this was more for com., twisted.plugins.*.

Right now it just says that it's a package which resides in multiple directories, and it's not made clear why that's a desirable feature.

The concept of a "defining package" seems important to avoid conflicts like this one:

http://twistedmatrix.com/trac/ticket/2339

[snip some stuff about plugins and package layout]

Namespaces are not plugins and vice versa. The purpose of a namespace package is to allow projects managed by the same entity to share a namespace (ala Java "package" names) and avoid naming conflicts with other authors.

I think this is missing a key word: separate projects managed by the same entity.

Hmm. I thought I could illustrate that the same problem actually occurs without using a plugin system, but I can actually only come up with an example if an application implements multi-library-version compatibility by doing

try:
    from bad_old_name import bad_old_feature as feature
except ImportError:
    from good_new_name import good_new_feature as feature

rather than the other way around; and that's a terrible idea for other reasons. Other than that, you'd have to use pkg_resources.resource_listdir or somesuch, at which point you pretty much are implementing a plugin system.

So I started this reply disagreeing but I think I've convinced myself that you're right.

Precisely. Note, however, that neither is twisted.plugins a namespace package, and it should not contain any .pkg files. I don't think it's reasonable to abuse PEP 382 namespace packages as a plugin system. In setuptools' case, a different mechanism is provided for locating plugin code, and of course Twisted already has its own system for the same thing. It would be nice to have a standardized way of locating plugins in the stdlib, but that will need to be a different PEP.

Okay. So what I'm hearing is that Twisted should happily continue using our own wacky path-calculation logic for twisted.plugins, but that twisted should be a namespace package so that our separate distributions (TwistedCore, TwistedWeb, TwistedConch, et. al.) can be installed into separate directories.

More information about the Python-Dev mailing list