[Python-Dev] Proposal: expose PEP 302 facilities via 'imp' and 'pkgutil' (original) (raw)

Phillip J. Eby pje at telecommunity.com
Tue Apr 11 23:04:24 CEST 2006

I just noticed that two stdlib modules (runpy and test.test_importhooks) contain reimplementations of the base PEP 302 algorithm, or loaders wrapping the standard (pre-302) import machinery.

Meanwhile, the 'imp' module exports an undocumented IMP_HOOK constant (since Python 2.3), that is used internally to Python/import.c but never actually returned from the imp API.

Further, the machinery called by imp.find_module() actually does the full PEP 302 search dance - but then skips any PEP 302 importers in the process, because the wrapper doesn't let it return a loader object.

What I'd like to do is make the necessary modifications to import.c that would allow you to access the loaders found by the C version of find_module.

I propose to create a new API, 'imp.find_loader()' and have it return a PEP 302-compatible loader object, even for cases that would normally be handled via 'imp.load_module()'. In such cases, the loader returned would be an instance of one of a loader class similar to those in runpy, test_importhooks, and setuptools (which also has similar code).

What I'm not sure of is where to put the loader class. It seems to me there should be a stdlib module, but it doesn't seem worth writing in C, especially with so many Python implementations floating around.

I could create a new Python module for them, but we already have so many import-related modules floating around. Would it be reasonable to add them to 'pkgutil', which until now has contained only one function? This would help cut down on some of the code duplication, without adding yet another module to the stdlib.

An additional issue: "pydoc" needs to be able to determine what submodules, if any, exist in a package, but the PEP 302 protocol does not provide for this ability. I'd like to add optional additional methods to the PEP 302 "importer" protocol (and to any stdlib importer objects) to support the type of filesystem-like queries performed by pydoc. This should allow pydoc to be compatible with zip files as well as regular files, and any future PEP 302 importers that provide the necessary features.

Having these features would also let me cut some code out of setuptools' "pkg_resources" module, that adds some of these features using adapter registries.

It may be too late for me to be able to implement all of this in time for alpha 2, but at minimum I think the 'find_loader()' addition and the move of the import wrapper classes to 'pkgutil' could be accomplished.

Replacing pydoc's myriad stat(), listdir() and other path hacking with clean importer protocols might be out of reach, but I think it's worth a try. For one thing, it would mean that shipping the Python stdlib in a zipfile would not inhibit pydoc's ability to display help for stdlib modules and packages. (Currently, pydoc does not work correctly with packages that are in zipfiles or which are spread across multiple directories, and it is unable to "discover" modules that are in zipfiles.)

More information about the Python-Dev mailing list