[Python-Dev] Writing importers and path hooks (original) (raw)

Brett Cannon brett at python.org
Thu Mar 28 17:08:29 CET 2013

On Thu, Mar 28, 2013 at 11:38 AM, Paul Moore <p.f.moore at gmail.com> wrote:

On 28 March 2013 13:42, Brett Cannon <brett at python.org> wrote: >> importer, as I wanted to try a proof of concept importer based on the >> new importlib stuff (which is intended to make writing custom >> importers easier), and I really struggled to get something working. > > Struggling how? With the finder? The loader? What exactly were you trying to > accomplish and how were you deviating from the standard import system?

What I was trying to do was to write a path hook that would allow me to add a string to sys.path which contained a base-64 encoded zipfile (plus some sort of marker so it could be distinguished from a normal path entry) and as a result the contents of that embedded zip file would be available as if I'd added an actual zip file with that content to sys.path. I got in a complete mess as I tried to strip out the (essentially non-interesting) zipfile handling to get to a dummy "do nothing, everything is valid" type of example. But I don't think I would have fared much better if I'd stuck to the original full requirement. >> It seems to me that the importlib documentation doesn't help much for >> people trying to import path hooks. > > There is a bug to clarify the docs to have more geared towards writing new > importers instead of just documenting what's available: > http://bugs.python.org/issue15867 Thanks. I'll keep an eye on that. >> But it might be just me. Does >> anyone have an example of a simple importlib-based finder/loader? > > Define simple. =) I would argue importlib itself is easy enough to read. :-) Fair point. I guess importlib is not that hard to read, but the only case that implements packages in the filesystem one, and that also deals with C extensions and other complexities that I don't have a need for. I'll try again to have a deeper look at it, but I didn't find it easy to extract the essentials when I looked before. >> That >> would be a huge help for me. In return for any pointers, I'll look at >> putting together a doc patch to clarify how to use importlib to build >> your own path hooks :-) > > Do you specifically mean the path hook aspect or the whole package of hook, > finder, and loader? OK, after some more digging, it looks like I misunderstood the process somewhat. Writing a loader that inherits from both FileLoader and SourceLoader,

You only need SourceLoader since you are dealing with Python source. You don't need FileLoader since you are not reading from disk but an in-memory zipfile.

and then implementing getdata (and modulerepr - why do I need that, couldn't the ABC offer a default implementation?)

http://bugs.python.org/issue17093 and http://bugs.python.org/issue17566

does the job for that.

You should be implementing get_data, get_filename, and path_stats for SourceLoader.

But the finder confuses me. I assume I want a PathEntryFinder and hence I should implement findloader().

Yes since you are pulling from sys.path.

The documentation on what I need to return from there is very sparse... In the end I worked out that for a package, I need to return (MyLoader(modulename, 'foo/init.py'), ['foo']) (here, "foo" is my dummy marker for my example).

The second argument should just be None: "An empty list can be used for portion to signify the loader is not part of a [namespace] package". Unfortunately a key word is missing in that sentence. http://bugs.python.org/issue17567

In essence, PathEntryFinder really has to implement some form of virtual filesystem mount point, and preserve the standard filesystem semantics of modules having a filename of .../init.py.

Well, if your zip file decided to create itself with a different file extension then it wouldn't be required, but then other people's code might break if they don't respect module abstractions (i.e. looking at package/name or path to see if something is a package).

So I managed to work out what was needed in the end, but it was a lot harder than I'd expected. On reflection, getting the finder semantics right (and in particular the path entry finder semantics) was the hard bit.

Yep, that bit has had the least API tweaks as most people don't muck with finders but with loaders.

I'm now 100% sure that some cookbook examples would help a lot. I'll see what I can do.

I plan on writing a pure Python zip importer for Python 3.4 which should be fairly minimal and work out as a good example chunk of code. And no one need bother writing it as I'm going to do it myself regardless to make sure I plug any missing holes in the API. If you really want something to try for fun go for a sqlite3-backed setup (don't see it going in the stdlib but it would be a project to have). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130328/7463eeea/attachment.html>

More information about the Python-Dev mailing list