[Python-Dev] Fwd: [Import-SIG] Where to discuss PEP 382 vs. PEP 402 (namespace packages)? (original) (raw)

PJ Eby pje at telecommunity.com
Mon Mar 12 16:43:28 CET 2012

Ugh; this was supposed to be sent to the list, not just Guido. (I wish Gmail defaulted to reply-all in the edit box.)

---------- Forwarded message ---------- From: PJ Eby <pje at telecommunity.com> Date: Mon, Mar 12, 2012 at 12:16 AM Subject: Re: [Import-SIG] Where to discuss PEP 382 vs. PEP 402 (namespace packages)? To: Guido van Rossum <guido at python.org>

On Sun, Mar 11, 2012 at 10:39 PM, Guido van Rossum <guido at python.org> wrote:

I'm leaning towards PEP 402 or some variant. Let's have a pow-wow at the sprint tomorrow (I'll arrive in Santa Clara between 10 and 10:30). I do want to understand Nick's argument better; I haven't studied PEP 395 yet.

Note that PEP 395 can stay compatible with PEP 402 by a fairly straightforward change: instead of implicitly and automagically guessing the needed sys.path[0] change, it could be made explicit by adding something like this to the top of script/modules that are inside a package:

import pkgutil
pkgutil.script_module(__name__, 'mypackage.thismodule')

Assuming name=='main', the API would set main.qualname, set sys.modules[qualname] = main, and fix up sys.path[0] if and only if it still is the parent directory of main.file. (If name!=='main' and it's not equal to the second argument either, it'd be an error.)

Then, in the event of broken relative imports or module aliasing, the error message can suggest adding a script_module() declaration to explicitly make the file a "dual citizen" -- i.e., script/module. (It's already possible for PEP 395 to be confused by stray init.py files or path manipulation; using error messages and explicit declaration instead of guessing seems like a better route for 395 to take.)

Of course, it's also possible to fix the 395/402 incompatibility by reintroducing some sort of marker, such as .pyp directory extensions or by including *.pyp marker files within package directories. The problem is that these markers work against the intuitive nature of PEP 402 if they are required, and they do not help 395 if nobody uses them due to their optionality. ;-)

(Last, but not least, the compromise approach: allow explicit script/module declaration as a workaround for virtual packages, AND support automagic qualname recognition for self-contained packages... but still give error messages for broken relative imports and aliasing that suggest the explicit declaration.)

Anyway, the other open issues for 402 are:

There was a Python-Dev discussion about the first, in which I realized that sys.path updates can actually be handled transparently by making virtual path objects be special iterables rather than lists; but the PEP hasn't been updated to reflect that. (I was actually waiting for some sign of BDFL interest before adding a potential complication like that to the PEP.) The relevant proposal was:

This seems to lean in favor of making a simple reiterable wrapper type for the path, that only allows you to take the length and iterate over it. With an appropriate design, it could actually update itself automatically, given a subname and a parent path/sys.path. That is, it could keep a tuple copy of the last-seen parent path, and before iteration, compare tuple(self.parentpath) to self.lastseenpath. If they're different, it rebuilds the value to be iterated over.

Voila: transparent updating of all virtual path values from sys.path changes (or modifications to self-contained path parents, btw), and trying to change it (or read an item from it positionally) will not create any silent failures.

Alright... if we support automatic updates to virtual paths, this is probably how we should do it. (It will require, though, that imp.findmodule be changed to use a different iteration method than PyListGetItem, as it's quite possible a virtual path will get passed into it.)

I actually drafted an implementation of this to work with importlib, so it seems pretty feasible to support automatically-updated virtual paths that change on the next import attempt if sys.path (or any parent path) has changed since the last time.

Iterating virtual packages is a somewhat harder problem, since it's not really practical to do an unbounded subdirectory search for importable files. Probably, the pkgutil module-walking APIs just need to grow some extra flags for virtual package searching, with some reasonable defaults. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20120312/2f4dab30/attachment.html>

More information about the Python-Dev mailing list