[Python-Dev] difficulty of implementing phase 2 of PEP 302 in Python source (original) (raw)

Brett Cannon brett at python.org
Thu Sep 28 20:25:25 CEST 2006

On 9/27/06, Phillip J. Eby <pje at telecommunity.com> wrote:

At 05:26 PM 9/27/2006 -0700, Brett Cannon wrote: >Ah, OK. So for importing 'email', the zipimporter would call the .pyc >importer and it would ask the zipimporter, "can you get me email.pyc?" and >if it said no it would move on to asking the .py importer for email.py, etc. Yes, exactly.

>That's fine. Just thinking about how the current situation sucks for NFS >but how caching just isn't done. But obvoiusly this could change. Well, with this design, you can have a CachingFilesystemImporter as your storage mechanism to speed things up. >> >>Of course, to fully support .pyc timestamp checking and writeback, you'd >> >>need some sort of "stat" or "getmtime" feature on the parent importer, as >> >>well as perhaps an optional "savedata" method. These would be extensions >> >>to PEP 302, but welcome ones. >> > >> >Could pass the string representing the location of where the string came >> >from. That would allow for the required stat calls for .pyc files as >> >needed without having to implement methods just for this one use case. >> >>Huh? In order to know if a .pyc is up to date, you need the stmtime of >>the .py file. That can't be done in the parent importer without giving it >>format knowledge, which goes against the point of the exercise. > >Sorry, thought .pyc files based whether they needed to be recompiled based >on the stat info on the .py and .pyc file, not on data stored from within >the .pyc . It's not just that (although I believe it's also the case that there is a timestamp inside .pyc), it's that to do the check in the parent importer, the parent importer would have to know that there is such a thing as .py-and-.pyc. The whole point of this design is that the parent importer doesn't have to know anything about filename extensions OR how those files are formatted internally. In this scheme, adding more child importers is sufficient to add all the special handling needed for .py/.pyc-style schemes. Of course, for maximum flexibility, you might want getstream() and getfile() methods optionally available, since a .so loader really needs a file, and .pyc might want to read in two stages. But the child importers can be defensively coded so as to be able to live with only a parent.getdata(), if necessary, and do the enhanced behaviors only if stat() or getstream() or writedata() etc. attributes are available on the parent.

Yeah, how to get the proper information to the data importers is going to be the trick.

If we get some standards for these additional attributes, we can document

them as standard PEP 302 extensions.

The format importer mechanism might want to have something like 'sys.importformats' as a list of importer classes (or factories). Parent (storage) importer classes would then create instances to use. If you add a new format importer to sys.importformats, you would of course need to clear sys.pathimportercache, so that the individual importers are rebuilt on the next import, and thus they will create new child importer chains. Yeah, that pretty much ought to do it.

I will think about it, but I am still trying to get the original question of how bad the C code is compared to rewriting import in Python from people. =)

-Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060928/979594a0/attachment.html

More information about the Python-Dev mailing list