[Python-Dev] Require loaders set package and loader (original) (raw)

Brett Cannon brett at python.org
Sun Apr 15 00:50:30 CEST 2012

On Sat, Apr 14, 2012 at 18:32, Guido van Rossum <guido at python.org> wrote:

On Sat, Apr 14, 2012 at 2:15 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote: > On Sat, Apr 14, 2012 at 2:56 PM, Brett Cannon <brett at python.org> wrote: >> An open issue in PEP 302 is whether to require loader attributes on >> modules. The claimed worry is memory consumption, but considering importlib >> and zipimport are already doing this that seems like a red herring. >> Requiring it, though, opens the door to people relying on its existence and >> thus starting to do things like loading assets with >> _loader_.getdata(pathtointernalpackagefile) which allows code to >> not care how modules are stored (e.g. zip file, sqlite database, etc.). >> >> What I would like to do is update the PEP to state that loaders are expected >> to set loader. Now importlib will get updated to do that implicitly so >> external code can expect it post-import, but requiring loaders to set it >> would mean that code executed during import can rely on it as well. >> >> As for package, PEP 366 states that modules should set it but it isn't >> referenced by PEP 302. What I want to do is add a reference and make it >> required like loader. Importlib already sets it implicitly post-import, >> but once again it would be nice to do this pre-import. >> >> To help facilitate both new requirements, I would update the >> importlib.util.moduleforloader decorator to set both on a module that >> doesn't have them before passing the module down to the decorated method. >> That way people already using the decorator don't have to worry about >> anything and it is one less detail to have to worry about. I would also >> update the docs on importlib.util.setpackage and importlib.util.setloader >> to suggest people use importlib.util.moduleforloader and only use the >> other two decorators for backwards-compatibility. > > +1

Funny, I was just thinking about having a simple standard API that will let you open files (and list directories) relative to a given module or package regardless of how the thing is loaded. If we guarantee that there's always a loader that's a first step, though I think we may need to do a little more to get people who currently do things like open(os.path.join(os.path.basename(file), 'somefilename') to switch. I was thinking of having a stdlib function that you give a module/package object, a relative filename, and optionally a mode ('b' or 't') and returns a stream -- and sibling functions that return a string or bytes object (depending on what API the user is using either the stream or the data can be more useful). What would we call thos functions and where would the live?

IOW go one level lower than get_data() and return the stream and then just have helper functions which I guess just exhaust the stream for you to return bytes or str? Or are you thinking that somehow providing a function that can get an explicit bytes or str object will be more optimized than doing something with the stream? Either way you will need new methods on loaders to make it work more efficiently since loaders only have get_data() which returns bytes and not a stream object. Plus there is currently no API for listing the contents of a directory.

As for what to call such functions, I really don't know since they are essentially abstract functions above the OS which work on whatever storage backend a module uses.

For where they should live, it depends if you are viewing this as more of a file abstraction or something that ties into modules. For the former it seems like shutil or something that dealt with higher order file manipulation. If it's the latter I would say importlib.util. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20120414/badfa7fb/attachment.html>

More information about the Python-Dev mailing list