[Python-Dev] Unique loader per module (original) (raw)

Barry Warsaw barry at python.org
Tue Jan 2 15:35:37 EST 2018

We have what I think is one last design question for importlib.resources.

https://gitlab.com/python-devs/importlib_resources/issues/49

The problem is that the ResourceReader ABC omits the package from the function signatures, so that on a compatible loader, you only need to specify the resource you are interested in.

This is fine for file loaders because every package will have a unique loader instance associated with it, so it will know which package the requested resource is homed to.

But AFAICT, there is no specification or requirement in Python that every module/package have a unique loader instance. In fact, it’s not unreasonable given some of the text in PEP 302 to think that loader instances can be shared. The PEP says "In many cases the finder and loader can be one and the same object: finder.find_module() would just return self” and you aren’t going to typically have a unique finder per module, so that would implied a shared loader per finder.

We even have an existence proof in the zip importer:

import test.testimportlib.zipdata02 import sys, os sys.path.insert(0, os.path.join(os.path.dirname(test.testimportlib.zipdata02.file), 'ziptestdata.zip')) import ziptestdata.two import ziptestdata.one ziptestdata.one.spec.loader == ziptestdata.two.spec.loader True

The issue above proposes two solutions. The first is to change the ABC so that it includes the package argument in the ABC method signatures. That way, a shared loader will know which package the requested resource is relative to.

Brett doesn’t like this, for several reasons (quoting):

  1. redundant API in all cases where the loader is unique to the module
  2. the memory savings of sharing a loader is small
  3. it's implementation complexity/overhead for an optimization case.

The second solution, and the one Brett prefers, is to reimplement zip importer to not use a shared loader. This may not be that difficult, if for example we were to use a delegate loader wrapping a shared loader.

The bigger problem IMHO is two-fold:

  1. It would be backward incompatible. If there’s any code out there expecting a shared loader in zipimport, it would break
  2. More problematic is that we’d have to impose an additional requirement on loaders - that they always be unique per module, contradicting the advice in PEP 302

The reason for this is third party finder/loaders. Sure, we can fix zipimport but any third party finder/loaders could have the same problem, and they’d be within their rights given the current specification. We’d have to prohibit that, or at least say that any third party finder/loaders that shared their loader can’t implement ResourceReader (which would be the practical effect anyway). I think that would be a shame.

So while I agree with Brett that it’s uglier, and that once we decide we’re essentially locked into the API, I don’t see a whole lot of options.

Thoughts, feedback, suggestions are welcome. -Barry

-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: <http://mail.python.org/pipermail/python-dev/attachments/20180102/a29cd876/attachment.sig>

More information about the Python-Dev mailing list