[Python-Dev] Choosing a best practice solution for Python/extension modules (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Sat Feb 21 23:05:09 CET 2009

Brett Cannon wrote:

On Sat, Feb 21, 2009 at 11:43, <glyph at divmod.com_ _<mailto:glyph at divmod.com>> wrote: On 07:07 pm, brett at python.org <mailto:brett at python.org> wrote: On Sat, Feb 21, 2009 at 09:17, Jean-Paul Calderone <exarkun at divmod.com <mailto:exarkun at divmod.com>>wrote: But there is another issue with this: the pure Python code will never call the extension code because the globals will be bound to pypickle and not pickle. So if you have something like:: # pypickle def A(): return B() def B(): return -13 # pickle def B(): return 42 # pickle from pypickle import * try: from pickle import * except ImportError: pass This is really the same as any other high-level/low-level library split. It doesn't matter that in this case, one low-level implementation is provided as an extension module. Importing the low-level APIs from another module and then using them to implement high-level APIs is a pretty common, simple, well-understood technique which is quite applicable here. But that doesn't provide a clear way, short of screwing with sys.modules, to get at just the pure Python implementation for testing when the extensions are also present. The key point in trying to figure this out is to facilitate testing since the standard library already uses the import * trick in a couple of places. You don't have to screw with sys.modules. The way I would deal with testing this particular interaction would be a setUp that replaces pickle.A with pypickle.A, and a tearDown that restores the original one. Twisted's TestCase has specific support for this. You would spell it like this: import pypickle # ... testCase.patch(pickle, 'A', pypickle.A) You can read more about this method here: http://python.net/crew/mwh/apidocs/twisted.trial.unittest.TestCase.html#patch My worry with this approach is that while this works nicely if you are only overriding a single function, having to do this for all functions and classes in order to make sure you are testing the extension code with all the extension code instead of intermingled extension/Python code. So a function that did this automatically for the entire module would be needed, which is like what I proposed in my useextension function. I am seeing two approaches emerging. One is where pickle contains all Python code and then uses something like useextension to make sure the original Python objects are still reachable at some point. This has the drawback that you have to use some function to make the extensions happen and there is some extra object storage. The other approach is having pickle contain code known not to be overridden by anyone, import pypickle for stuff that may be overridden, and then import pickle for whatever is available. This approach has the perk of using a standard practice for how to pull in different implementation. But the drawback, thanks to how globals are bound, is that any code pulled in from pickle/pypickle will not be able to call into other optimized code; it's a take or leave it once the call chain enters one of those modules as they will always call the implementations in the module they originate from.

I'd actually say there's a third option which is still viable: continue with the current Foo/_Foo practice for optimised modules, and provide a function in test.support to get the original Python version's code out of Foo.

That actually has the virtue of directly testing that the ImportError for the missing module is being trapped correctly.

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list