[Python-Dev] Allowing to run certain regression tests in subprocesses (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Sun Aug 4 16:26:44 CEST 2013

On 4 August 2013 23:40, Eli Bendersky <eliben at gmail.com> wrote:

On Sat, Aug 3, 2013 at 7:08 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

Sure, it's just unusual to have a case where "importing is blocked by adding None to sys.modules" differs from "not actually available", so I'd like to understand the situation better. I must admit I'm confused by the behavior of importfreshmodule too. Snippet #1 raises the expected ImportError: sys.modules['pyexpat'] = None import elementtree However, snippet #2 succeeds importing: ET = importfreshmodule('elementtree', blocked=['pyexpat']) print(ET)

/me goes and looks

That function was much simpler when it was first created :P

Still, I'm fairly confident the complexity of that dance isn't relevant to the problem you're seeing.

I believe this happens because importfreshmodule does an import of the 'name' it's given before even looking at the blocked list. Then, it assigns None to sys.modules for the blocked names and re-imports the module. So in essence, this is somewhat equivalent to snippet #3:

modname = 'elementtree' import(modname) del sys.modules[modname] for m in sys.modules: if modname == m or m.startswith(modname + '.'): del sys.modules[m] sys.modules['pyexpat'] = None ET = importlib.importmodule(modname) print(ET) Which also succeeds. I fear I'm not familiar enough with the logic of importing to understand what's going on, but it has been my impression that this problem is occasionally encountered with importfreshmodule and C code that imports stuff (the import of pyexpat is done by C code in this case).

I had missed it was a C module doing the import. Looking into the _elementtree.c source, the problem in this case is the fact that a shared library that doesn't use PEP 3121 style per-module state is only loaded and initialised once, so reimporting it gets the same module back (from the extension loading cache), even if the Python level reference has been removed from sys.modules. Non PEP 3121 C extension modules thus don't work properly with test.support.import_fresh_module (as there's an extra level of caching involved that can't be cleared from Python, because it would break things).

To fix this, _elementree would need to move the pyexpat C API pointer to per-module state, rather than using a static variable (see http://docs.python.org/3/c-api/module.html#initializing-c-modules). With per-module state defined, the import machine should rerun the init function when the fresh import happens, thus creating a new copy of the module. However, this isn't an entirely trivial change for _elementree, since:

  1. Getting from the XMLParser instance back to the module where it was defined in order to retrieve the capsule pointer via PyModule_GetState() isn't entirely trivial in C. You'd likely do it once in the init method, store the result in an XMLParser attribute, and then tweak the EXPAT() using functions to include an appropriate local variable definition at the start of the method implementation.

  2. expat_set_error would need to be updated to accept the pyexpat capsule pointer as a function parameter

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list