[Python-Dev] is this the fault of import_fresh_module or pickle? (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Wed Jan 9 02:28:21 CET 2013

On Wed, Jan 9, 2013 at 2:58 AM, Eli Bendersky <eliben at gmail.com> wrote:

Handling this case is why having a context-manager form of importfreshmodule was suggested earlier in this meta-thread. At least, I think that would solve it, I haven't tried it :)

Would you mind extracting just this idea into this discussion so we can focus on it here? I personally don't see how making importfreshmodule a context manager will solve things, unless you add some extra functionality to it? AFAIU it doesn't remove modules from sys.modules before importing, at this point.

Sure it does, that's how it works: the module being imported, as well as anything requested as a "fresh" module is removed from sys.modules, anything requested as a "blocked" module is replaced with None (or maybe 0 - whatever it is that will force ImportError). It then does the requested import and then reverts all those changes to sys.modules.

It's that last part which is giving you trouble: by the time you run the actual tests, sys.modules has been reverted to its original state, so pickle gets confused when it attempts to look things up by name.

Rather than a context manager form of import_fresh_module, what we really want is a "modules_replaced" context manager:

@contextmanager
def modules_replaced(replacements):
    _missing = object()
    saved = {}
    try:
        for name, mod in replacements.items():
            saved[name] = sys.modules.get(name, _missing)
            sys.modules[name] = mod
        yield
    finally:
        for name, mod in saved.items():
            if mod is _missing:
                del sys.modules[name]
            else:
                sys.modules[name] = mod

And a new import_fresh_modules function that is like import_fresh_module, but returns a 2-tuple of the requested module and a mapping of all the affected modules, rather than just the module object.

However, there will still be cases where this doesn't work (i.e. modules with import-time side effects that don't support repeated execution), and the pickle and copy global registries are a couple of the places that are notorious for not coping with repeated imports of a module. In that case, the test case will need to figure out what global state is being modified and deal with that specifically.

(FWIW, this kind of problem is why import_fresh_module is in test.support rather than importlib and the reload builtin became imp.reload in Python 3 - "module level code is executed once per process" is an assumption engrained in most Python developer's brains, and these functions deliberately violate it).

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list