[Python-Dev] PEP: Lazy module imports and post import hook (original) (raw)

Christian Heimes lists at cheimes.de
Wed Jan 9 06:01:01 CET 2008

glyph at divmod.com wrote:

and very recently, I implemented similar functionality myself (though it isn't in use in Twisted yet):

http://codebrowse.launchpad.net/~glyph/+junk/pyexport/files

I'm going to study your implementation tomorrow.

Something that I notice about every other implementation of this functionality is that they're all in Python. But the proposed implementation here is written in C, and therefore more prone to crashing bugs. Looking at the roundup log, I can see that several refcounting bugs have already been found and fixed. Perhaps the post- import hooks, being a modification to the import mechanism itself, needs to be in C, but given that lazy imports have been implemented before without using C code (and without post import hooks), I can't see why they need to be in the same PEP.

Correct, the first few patches were rough implementation sketches. I wanted to discuss implementation details with PJE and it was the easiest way to exchange code. A distributed VCS would have made the task easier but that's a different story. Eventually I focused on the reference counter and I straightened out some ref count bugs.

Deferred loading of modules can be implemented without post import hooks and vice versa. But PJE urged me to - at least - consider lazy modules / deferred importing in the post import hook implementation. 3rd party tools need a way to interact with the hooks and notify the system when their own lazy module implementation loads a real module.

I figured out I could write a PEP for both features in one PEP and implement them in a combined patch. I could separate them and create two PEPs.

Also, as is my custom, I'd like to suggest that, rather than designing something new, one of the third-party implementations be adopted (or at least its interface) since these have been demonstrated to work in real application code already. Happily, I can escape the charge of bias this time since I don't think my own implementation should be taken seriously in this case :).

:)

I did mention that my implementation of lazy imports is based on PJE's peak.util.import:

""" PJE's peak.util.imports [3] supports lazy modules an post load hooks. My implementation shares a lot with his and it's partly based on his ideas. """

Finally I like to say something about my choice to implement the lazy loading in C. Most implementation like zope.deferredimport, Hg's demandimport and bzr's lazy_import are using the proxy design pattern. They are storing a wrapper object in the module and sys.modules instead of the real module and they are loading the real module when an attribute is accessed. Some of some (brz) are even using some strange scope replacing. I've to study the implementation more carefully to understand what's going on.

peak.util.imports' way is slightly different. It return the instance of a module subclass. When the real module is loaded it redirect name lookup of the lazy module to the real module by replacing the getattribute and setattr methods of the lazy module with the methods of the real module.

My implementation uses a similar approach. But I've the luxury that I can modify the C implementation of the module class. It makes the load-real-module-when-attribute-is-accessed dance much easier. I don't have to write a proxy or deal with name spaces.

However the PEP covers only the basic infrastructure for lazy imports. For example imp.lazy_import("a.b.c") doesn't put "a" and "a.b" in sys.modules. It neither doesn't cover the replacement import hook provided by Hg nor the deprecation feature of zope.deferredimport. Such infrastructure functions can be implemented in Python.

Please remember, the PEP is a draft. I'm waiting for comments and suggestions from users and developers.

Christian

More information about the Python-Dev mailing list