[Python-Dev] Implementing PEP 382, Namespace Packages (original) (raw)

Brett Cannon brett at python.org
Mon May 31 02:59:57 CEST 2010

On Sat, May 29, 2010 at 15:56, P.J. Eby <pje at telecommunity.com> wrote:

At 09:29 PM 5/29/2010 +0200, Martin v. Löwis wrote:

Am 29.05.2010 21:06, schrieb P.J. Eby:

At 08:45 PM 5/29/2010 +0200, Martin v. Löwis wrote:

In it he says that PEP 382 is being deferred until it can address PEP 302 loaders. I can't find any follow-up to this. I don't see any discussion in PEP 382 about PEP 302 loaders, so I assume this issue was never resolved. Does it need to be before PEP 382 is implemented? Are we wasting our time by designing and (eventually) coding before this issue is resolved? Yes, and yes. Is there anything we can do to help regarding that? You could comment on the proposal I made back then, or propose a different solution. Looking at that proposal, I don't follow how changing loaders (vs. importers) would help.  If an importer's findmodule doesn't natively support PEP 382, then there's no way to get a loader for the package in the first place.  Today, namespace packages work fine with PEP 302 loaders, because the namespace-ness is really only about setting up the path, and detecting that you need to do this in the first place. In the PEP 302 scheme, then, it's either importers that have to change, or the process that invokes them.  Being able to ask an importer the equivalents of os.path.join, listdir, and getdata would suffice to make an import process that could do the trick. Essentially, you'd ask each importer to first attempt to find the module, and then asking it (or the loader, if the find worked) whether packagename/*.pth exists, and then processing their contents. I don't think there's a need to have a special method for executing a package init, since what you'd do in the case where there are .pth but no init, is to simply continue the search to the end of sys.path (or the parent package path), and then create the module with an appropriate path. If at any point the findmodule() call succeeds, then subsequent importers will just be asked for .pth files, which can then be processed into the path of the now-loaded module. IOW, something like this (very rough draft):  pthcontents = []  module = None  for pathitem in syspathorparent_path:_  importer = pkgutil.getimporter(pathitem)  if importer is None:  continue  if module is None:  try:  loader = importer.findmodule(fullname)  except ImportError:  pass  else:  # errors here should propagate  module = loader.loadmodule(fullname)  if not hasattr(module, 'path'):  # found, but not a package  return module  pc = getpthcontents(importer)  if pc is not None:  subpath = os.path.join(pathitem, modulebasename)  pthcontents.append(subpath)  pthcontents.extend(pc)  if '*' not in pthcontents:  # got a package, but not a namespace  break  if pthcontents:  if module is None:  # No init, but we have paths, so make an empty package  module = # new module object w/empty path  modify_path(module, pthcontents)_  return module

Is it wise to modify path post-import? Today people can make sure that path is set to what they want before potentially reading it in their init module by making the pkgutil.extend_path() call first. This would actually defer to after the import and thus not allow any init code to rely on what path eventually becomes.

Obviously, the details are all in the 'getpthcontents()', and 'modify_path()' functions, and the above process would do extra work in_ the case where an individual importer implements PEP 382 on its own (although why would it?).

It's also the case that this algorithm will be slow to fail imports when implemented as a metapath hook, since it will be doing an extra pass over sys.path or the parent path, in addition to the one that's done by the normal import machinery.  (Though that's not an issue for Python 3.x, since this can be built into the core import). (Technically, the 3.x version should probably ask metapath hooks for their .pth files as well, but I'm not entirely sure that that's a meaningful thing to ask.) The PEP 302 questions all boil down to how getpthcontents() is implemented, and whether 'subpath' really should be created with os.path.join.  Simply adding a getpthcontents() method to the importer protocol (that returns None or a list of lines), and maybe a getsubpath(modulename) method that returns the path string that should be used for a subdirectory importer (i.e. path entry), or None if no such subpath exists.

Code already out there uses os.path.join() to extend path (e.g. Django), so I would stick with that unless we want to start transitioning to '/' only.

More information about the Python-Dev mailing list