[Python-Dev] Implementing PEP 382, Namespace Packages (original) (raw)

Brett Cannon brett at python.org
Mon May 31 22:10:45 CEST 2010

On Mon, May 31, 2010 at 00:53, "Martin v. Löwis" <martin at v.loewis.de> wrote:

For finders, their search algorithm is changed in a couple of ways. One is that modules are given priority over packages (is that intentional, Martin, or just an oversight?). That's an oversight. Notice, however, that it's really not the case that currently directories have precedence over modules, either: if a directory is later on the path than a module, it's still the module that gets imported. So the precedence takes place only when a module and a directory exist in the same directory. In any case, I have now fixed it.

Two, the package search requires checking for a .pth file on top of an init.py. This will change finders that could before simply do an existence check on an init "file" You are reading something into the PEP that isn't there yet. PEP 302 currently isn't considered, and the question of this discussion is precisely how the loaders API should be changed. (or whatever the storage back-end happened to be) and make it into a list-and-search which one would hope wasn't costly, but in same cases might be if the paths to files is not stored in a hierarchical fashion (e.g. zip files list entire files paths in their TOC or a sqlite3 DB which uses a path for keys will have to list all keys, sort them to just the relevant directory, and then look for .pth or some such approach). First, I think it's up to the specific loader mechanism whether PEP 382 should be supported at all. It should be possible to implement it if desired, but if it's not feasible (e.g. for URL loaders), pth files just may not get considered. The loader may well provide a different mechanism to support namespace packages. Are we worried about possible

performance implications of this search? For the specific case of zip files, I'm not. I don't think performance will suffer at all. And then the search for the init.py begins on the newly modified path, which I assume ends with the first init found on path, but if no file is found it's okay and essentially an empty module with just module-specific attributes is used? Correct. In other words, can a .pth file replace an init file in delineating a package? That's what it means by '''a directory is considered a package if it either contains a file named init.py, or a file whose name ends with ".pth".''' Or is it purely additive? I assume the latter for compatibility reasons, but the PEP says "a directory is considered a package if it either contains a file named init.py, or a file whose name ends with ".pth"" (emphasis mine). Why do you think this causes an incompatibility?

It's just if a project has no init older finders won't process it, that's all. But it looks like they are going to have to change somewhat anyway so that's not an issue.

Otherwise I assume that the search will be done simply with ``os.path.isdir(os.path.join(syspathentry, toplevelpackagename)`` and all existing paths will be added to path. Will they come before or after the directory where the *.pth was found? And will any subsequent *.pth files found in other directories also be executed? I may misremember, but from reading the text, it seems to say "no". It should work like the current pth mechanism (plus *, minus import). As for how "*" works, is this limited to top-level packages, or will sub-packages participate as well? I assume the former, but it is not directly stated in the PEP. And indeed, the latter is intended. You should be able to create namespace packages on all levels. If the latter, is a dotted package name changed to ``os.sep.join(sypathentry, packagename.replace('".", os.sep)``? No. Instead, the parent package's path is being searched for directories; sys.path is not considered anymore. I have fixed the text. For sys.pathhooks, I am assuming import will simply skip over passing that as it is a marker that path represents a namsepace package and not in any way functional. Although with sys.namespacepackages, is leaving the "*" in path truly necessary? It would help with efficiency, no?

Not sure how having "*" in path helps with efficiency. What are you thinking it will help with specifically?

For the search of paths to use to extend, are we limiting ourselves to actual file system entries on sys.path (as pkgutil does), or do we want to support other storage back-ends? To do the latter I would suggest having a successful path discovery be when a finder can be created for the hypothetical directory from sys.pathhooks. Again: PEP 302 isn't really considered yet. Proposals are welcome. The PEP (seems to) ask finders to look for a .pth file(s), calculate path, and then get a loader for the init. You could have finders grow a findnamespace method which returns the contents of the requisite .pth file(s). I must be misunderstanding the concept of finders. Why is it that it would be their function to process the pth files, and not the function of the loader?

I'm thinking from the perspective of finding an init module that exists somewhere else than where the .pth file was discovered. Someone has to find the .pth files and provide their contents. Someone else needs to find the init module for the package (if it exists). Then someone needs to load a namespace package, potentially from the init module. It's that second step -- find the init module -- that makes me think the finder is more involved. It doesn't have to be by definition, but seeing the word "find" just makes me think "finder".

More information about the Python-Dev mailing list