[Python-Dev] PEP 382: Namespace Packages (original) (raw)

P.J. Eby pje at telecommunity.com
Thu Apr 16 17:36:18 CEST 2009

At 03:46 AM 4/16/2009 +0000, glyph at divmod.com wrote:

On 15 Apr, 09:11 pm, pje at telecommunity.com wrote:

I think that there is some confusion here. A "main" package or buildout that assembles a larger project from components is not the same thing as having a "base" package for a namespace package. I'm certainly confused. Twisted has its own system for "namespace" packages, and I'm not really sure where we fall in this discussion. I haven't been able to follow the whole thread, but my original understanding was that the PEP supports "defining packages", which we now seem to be calling "base packages", just fine.

Yes, it does. The discussion since the original proposal, however, has been dominated by MAL's counterproposal, which requires a defining package.

There is a slight distinction between "base package" and "defining package", although I suppose I've been using them a bit interchangeably.

Base package describes a use case: you have a base package which is extended in the same namespace. In that use case, you may want to place your base package in the defining package.

In contrast, setuptools does not support a defining package, so if you have a base package, you must place it in a submodule or subpackage of the namespace.

Does that all make sense now?

MAL's proposal requires a defining package, which is counterproductive if you have a pure package with no base, since it now requires you to create an additional project on PyPI just to hold your defining package.

I'd appreciate it if the PEP could also be extended cover Twisted's very similar mechanism for namespace packages, "twisted.plugin.pluginPackagePaths". I know this is not quite as widely used as setuptools' namespace package support, but its existence belies a need for standardization.

The PEP also seems a bit vague with regard to the treatment of other directories containing init.py and *.pkg files.

Do you have a clarification to suggest? My understanding (probably a projection) is that to be a nested namespace package, you have to have a parent namespace package.

The concept of a "defining package" seems important to avoid conflicts like this one:

http://twistedmatrix.com/trac/ticket/2339 More specifically I don't quite understand the PEP's intentions towards hierarchical packages. It says that all of sys.path will be searched, but what about this case? In Twisted, the suggested idiom to structure a project which wants to provide Twisted plugins is to have a directory structure like this: MyProject/ myproject/ init.py twisted/ plugins/ myprojectplugin.py If you then put MyProject on PYTHONPATH, MyProject/twisted/plugins will be picked up automatically by the plugin machinery.

Namespaces are not plugins and vice versa. The purpose of a namespace package is to allow projects managed by the same entity to share a namespace (ala Java "package" names) and avoid naming conflicts with other authors.

A plugin system, by contrast, is explicitly intended for use by multiple authors, so the use case is rather different... and using namespace packages for plugins actually increases the possibility of naming conflicts, unless you add back in another level of hierarchy. (As apparently you are recommending via "myproject_plugin".)

However, as "twisted" is not a "namespace" package in the same way, .py files in MyProject/twisted/ would not be picked up - this is very much intentional, since the "twisted" namespace is intended to be reserved for packages that we actually produce. If either MyProject/twisted or MyProject/twisted/plugins/ had an init.py, then no modules in MyProject/twisted/plugins/ would be picked up, because it would be considered a conflicting package.

Precisely. Note, however, that neither is twisted.plugins a namespace package, and it should not contain any .pkg files. I don't think it's reasonable to abuse PEP 382 namespace packages as a plugin system. In setuptools' case, a different mechanism is provided for locating plugin code, and of course Twisted already has its own system for the same thing. It would be nice to have a standardized way of locating plugins in the stdlib, but that will need to be a different PEP.

I hope this all makes sense. As I understand it, both setuptools and the proposed standard would either still have the bug described by ticket 2339 above, or would ignore twisted/plugins/ as a namespace package because its parent isn't a namespace package.

If twisted/ lacked an init.py, then setuptools would ignore it. Under PEP 382, the same, unless it had .pkg files. (Again, setuptools explicitly does not support using namespace packages as a plugin mechanism.)

P.S.: vendor packaging systems ARE a major use case for just about any aspect of Python's package structure. I really liked MvL's coverage of "vendor packages", in the PEP, since this could equally well apply to MSIs, python libraries distributed in bundles on OS X, debs, or RPMs. If this use-case were to be ignored, as one particular fellow seems to be advocating, then the broken packages and user confusion that has been going on for the last 5 years or so is just going to get worse.

Indeed.

More information about the Python-Dev mailing list