[Python-Dev] Unique loader per module (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Fri Jan 5 02:12:44 EST 2018

On 3 January 2018 at 06:35, Barry Warsaw <barry at python.org> wrote:

Brett doesn’t like this, for several reasons (quoting):

1. redundant API in all cases where the loader is unique to the module 2. the memory savings of sharing a loader is small 3. it's implementation complexity/overhead for an optimization case. The second solution, and the one Brett prefers, is to reimplement zip importer to not use a shared loader. This may not be that difficult, if for example we were to use a delegate loader wrapping a shared loader. The bigger problem IMHO is two-fold: 1. It would be backward incompatible. If there’s any code out there expecting a shared loader in zipimport, it would break 2. More problematic is that we’d have to impose an additional requirement on loaders - that they always be unique per module, contradicting the advice in PEP 302

We added module.spec.loader_state as part of PEP 451 precisely so shared loaders had a place to store per-module state without have to switch to a unique-loader-per-module model.

I think the main reason you're seeing a problem here is because ResourceReader has currently been designed to be implemented directly by loaders, rather than being a subcomponent that you can request from a loader.

If you instead had an indirection API (that could optionally return self in the case of non-shared loaders), you'd keep the current resource reader method signatures, but the way you'd access the itself would be:

resources = module.__spec__.loader.get_resource_reader(module)
# resources implements the ResourceReader ABC

For actual use, the loader protocol could be hidden behind a helper function:

resources = importlib_resources.get_resource_reader(module)

For a shared loader, get_resource_reader(module) would return a new non-shared resource reader (perhaps caching it in spec.loader_state).

For a non-shared loader, get_resource_reader(module) would just return self.

In both cases, we'd recommend that loaders ensure "self is module.spec.loader" as part of their get_resource_reader() implementation.

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list