[Python-Dev] making the import machinery explicit (original) (raw)

Eric Snow ericsnowcurrently at gmail.com
Sat Apr 14 23:12:27 CEST 2012

On Sat, Apr 14, 2012 at 2:03 PM, Brett Cannon <brett at python.org> wrote:

To start off, what I am about to propose was brought up at the PyCon language summit and the whole room agreed with what I want to do here, so I honestly don't expect much of an argument (famous last words).

In the "ancient" import.c days, a lot of import's stuff was hidden deep in the C code and in no way exposed to the user. But with importlib finishing PEP 302's phase 2 plans of getting imoprt to be properly refactored to use importers, path hooks, etc., this need no longer be the case. So what I propose to do is stop having import have any kind of implicit machinery. This means sys.metapath gets a path finder that does the heavy lifting for import and sys.pathhooks gets a hook which provides a default finder. As of right now those two pieces of machinery are entirely implicit in importlib and can't be modified, stopped, etc. If this happens, what changes? First, more of importlib will get publicly exposed (e.g. the meta path finder would become public instead of private like it is along with everything else that is publicly exposed). Second, import itself technically becomes much simpler since it really then is about resolving module names, traversing sys.metapath, and then handling fromlist w/ everything else coming from how the path finder and path hook work. What also changes is that sys.metapath and sys.pathhooks cannot be blindly reset w/o blowing out import. I doubt anyone is even touching those attributes in the common case, and the few that do can easily just stop wiping out those two lists. If people really care we can do a warning in 3.3 if they are found to be empty and then fall back to old semantics, but I honestly don't see this being an issue as backwards-compatibility would just require being more careful of what you delete (which I have been warning people to do for years now) which is a minor code change which falls in line with what goes along with any new Python version. And lastly, sticking None in sys.pathimportercache would no longer mean "do the implicit thing" and instead would mean the same as NullImporter does now (which also means import can put None into sys.pathimportercache instead of NullImporter): no finder is available for an entry on sys.path when None is found. Once again, I don't see anyone explicitly sticking None into sys.pathimportercache, and if they are they can easily stick what will be the newly exposed finder in there instead. The more common case would be people wiping out all entries of NullImporter so as to have a new sys.pathhook entry take effect. That code would instead need to clear out None on top of NullImporter as well (in Python 3.2 and earlier this would just be a performance loss, not a semantic change). So this too could change in Python 3.3 as long as people update their code like they do with any other new version of Python. In summary, I want no more magic "behind the curtain" for Python 3.3 and import: sys.metapath and sys.pathhooks contain what they should and if they are emptied then imports will fail and None in sys.pathimportercache means "no finder" instead of "use magical, implicit stuff".

This is great, Brett. About sys.meta_path and sys.path_hooks, I see only one potential backwards-compatibility problem.

Those implicit hooks were fallbacks, effectively always at the end of the list, no matter how you manipulated the them. Code that appended onto those lists would now have to insert the importers/finders in the right way. Otherwise the default hooks would be tried first, which has a good chance of being the wrong thing.

That concern aside, I'm a big +1 on your proposal.

-eric

More information about the Python-Dev mailing list