[Python-Dev] zipimport.c broken with implicit namespace packages (original) (raw)

Guido van Rossum guido at python.org
Sat Jan 2 22:41:28 EST 2016

On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg at comcast.net> wrote:

-- >>>>> "Brett" == Brett Cannon <brett at python.org> writes: > I opened > https://bugs.python.org/issue25711 to specifically try to > fix this issue once and for all and along the way modernize > zipimport by rewriting it from scratch to be more > maintainable Every time I read about impementing a custom loader: https://docs.python.org/3/library/importlib.html I've wondered why python does not have some sort of virtual filesystem layer to deal with locating modules/packages/support files. Virtual file systems seem like a good way to store data on a wide range of storage devices.

Yeah, but most devices already implement a real filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.

A VFSLoader object would interface with importlib and deal with:

- implementing a finder and loader - Determine the actual type of file to load (.py, .pyc, .pyo, pycache, etc). - Do all of it's work by calling virtual functions such as: * listdir(path) * read(path) * stat(path) # for things like mtime, size, etc * write(path, data) # not all VFS implement this

Emulating a decent filesystem API requires you to implement functionality that would never be used by an import loader (write() is an example -- many of the stat() fields are another example). So it would just be overkill.

Then things like a ziploader would just inherit from VFSLoader implement the straight forward methods and everything should "just work" :). I see no reason why every type of loader (real filesystem, http, ssh, sql database, etc) would not do this as well.

All those examples except "real filesystem" are of very limited practical value.

Leave all the details such as implicit namespace packages, presence of init.py[oc] files, .pth files, etc in one single location and put the details on how to interact with the actual storage device in leaf classes which don't know or care about the high level details. They would not even know they are loading python modules. It is just blobs of data to them.

Actually the import loader API is much more suitable and less work to implement than a VFS.

I may try my hand at creating a prototype of this for just the zipimporter and see how it goes.

That would nevertheless be an interesting exercise -- I hope you do it and report back.

> At this point the best option might be, Mike, if you do a > code review for https://bugs.python.org/issue17633, even if > it is simply a LGTM. I will then personally make sure the > approved patch gets checked in for Python 3.6 in case the > rewrite of zipimport misses the release.

Cool. I'll see what I can do. I was having a bit of trouble with the register/login part of the bug tracker. Which is why I came here. I'll battle with it one more time and see if I can get it to log me in.

If you really can't manage to comment in the tracker (which sounds unlikely -- many people have succeeded :-) you can post your LGTM on the specific patch here.

The patch should be fairly simple. In a nutshell it just does a:

path.replace('.', '/') + '/' in two locations. One where it checks for the path being a directory entry in the zip file and the second to return an implicit namespace path (instead of not found) if it is a match. I'll check the patch on the tracker and see if it still works with 3.5.1. If not I'll attach mine.

Well, Brett would like to see your feedback on the specific patch. Does it work for you?

-- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20160102/64ba5d83/attachment.html>

More information about the Python-Dev mailing list