[Python-Dev] Dynamic module namspaces (original) (raw)

Andrew Bennetts andrew-pythondev at puzzling.org
Mon Jul 17 10:57:15 CEST 2006

On Sun, Jul 16, 2006 at 11:52:48PM -0700, Josiah Carlson wrote:

Andrew Bennetts <andrew-pythondev at puzzling.org> wrote: [...] > > Have you seen the "demandload" hack that Mercurial uses? You can find it here: > http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py > > You can see an example use of it here: > http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py

The problem with that particular method is that it requires that you use a string to describe the set of modules you would like to import, rather than a name.

I agree, it's ugly. I'd like there to be a nicer solution.

In the py3k mailing list I recently described a mechanism (though not implementation) for a somewhat related method to support automatic reloading when a file has changed (for things like web frameworks), but by removing that file change check, you get the equivalent to what you describe, though you don't need to use strings, you can use names...

from dynamicreload import DR and used like... DR.os.path.join(...) As long as you use the DR.-prefixed name, you get automatic reloading (if desired) on access.

Aside from also being ugly in its own way, a magic prefix like this would add a small performance penalty to places that use it. I believe demandload has the nice feature that once a demandloaded object is accessed, it is replaced with the actual object rather than the demandload gunk, so there's only a one-off performance penalty.

> The advantage for an interactive command line tool isn't so much memory > consumption as speed. Why waste hundreds of milliseconds importing code that > isn't used? There's an experimental branch to use the same demandload code in > bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root' > from 400ms => 200ms, etc." (according to > http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967) > > Over half the runtime wasted on importing unused code! There's a definite need > for a nice solution to this, and it should be included in the standard > batteries that come with Python.

Well, just starting up Python without loading any modules can be time consuming; perhaps even dwarfing the few hundred ms saved by dynamic loading.

Well, it's only about 10ms on my laptop running Ubuntu (it varies up to 90ms, but I expect that's just noise), and the "-S" switch makes no obvious difference (tested with "python -c ''"). 10ms overhead to start python I can live with. It takes about that long run "svn --version".

> If we can address related problems at the same time, like emitting deprecation > warnings for accessing certain module attributes, then even better!

deprecated = ['name1', ...] Make your dynamic load/reload aware of the deprecated attribute, and you are done.

I'm fine with that syntax. But regardless of how it's spelled, I'd really like the standard library or interpreter to have support for it, rather than having a non-standard 3rd-party system for it. I want there to "be one-- and preferably only one --obvious way to do it".

-Andrew.

More information about the Python-Dev mailing list