[Python-Dev] Dynamic module namspaces (original) (raw)

Josiah Carlson jcarlson at uci.edu
Mon Jul 17 08:52:48 CEST 2006

Andrew Bennetts <andrew-pythondev at puzzling.org> wrote:

On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote: > In an effort to reduce the memory usage used by GTK+ applications > written in python I've recently added a feature that allows attributes > to be lazy loaded in a module namespace. The gtk python module contains > quite a few attributes (around 850) of which many are classes or > interfaces (150+)

Have you seen the "demandload" hack that Mercurial uses? You can find it here: http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py You can see an example use of it here: http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py

The problem with that particular method is that it requires that you use a string to describe the set of modules you would like to import, rather than a name.

In the py3k mailing list I recently described a mechanism (though not implementation) for a somewhat related method to support automatic reloading when a file has changed (for things like web frameworks), but by removing that file change check, you get the equivalent to what you describe, though you don't need to use strings, you can use names...

from dynamicreload import DR

and used like...

DR.os.path.join(...)

As long as you use the DR.-prefixed name, you get automatic reloading (if desired) on access.

The advantage for an interactive command line tool isn't so much memory consumption as speed. Why waste hundreds of milliseconds importing code that isn't used? There's an experimental branch to use the same demandload code in bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root' from 400ms => 200ms, etc." (according to http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967)

Over half the runtime wasted on importing unused code! There's a definite need for a nice solution to this, and it should be included in the standard batteries that come with Python.

Well, just starting up Python without loading any modules can be time consuming; perhaps even dwarfing the few hundred ms saved by dynamic loading.

If we can address related problems at the same time, like emitting deprecation warnings for accessing certain module attributes, then even better!

deprecated = ['name1', ...]

Make your dynamic load/reload aware of the deprecated attribute, and you are done.

More information about the Python-Dev mailing list