[Python-Dev] Choosing a best practice solution for Python/extension modules (original) (raw)

Brett Cannon brett at python.org
Sat Feb 21 22:17:49 CET 2009

On Sat, Feb 21, 2009 at 11:43, <glyph at divmod.com> wrote:

On 07:07 pm, brett at python.org wrote:

On Sat, Feb 21, 2009 at 09:17, Jean-Paul Calderone <exarkun at divmod.com_ _>wrote:

But there is another issue with this: the pure Python code will never call the extension code because the globals will be bound to pypickle and not pickle. So if you have something like::

# pypickle def A(): return B() def B(): return -13 # pickle def B(): return 42 # pickle from pypickle import * try: from pickle import * except ImportError: pass This is really the same as any other high-level/low-level library split. It doesn't matter that in this case, one low-level implementation is provided as an extension module. Importing the low-level APIs from another module and then using them to implement high-level APIs is a pretty common, simple, well-understood technique which is quite applicable here. But that doesn't provide a clear way, short of screwing with sys.modules, to get at just the pure Python implementation for testing when the extensions are also present. The key point in trying to figure this out is to facilitate testing since the standard library already uses the import * trick in a couple of places. You don't have to screw with sys.modules. The way I would deal with testing this particular interaction would be a setUp that replaces pickle.A with pypickle.A, and a tearDown that restores the original one. Twisted's TestCase has specific support for this. You would spell it like this: import pypickle # ... testCase.patch(pickle, 'A', pypickle.A) You can read more about this method here: http://python.net/crew/mwh/apidocs/twisted.trial.unittest.TestCase.html#patch

My worry with this approach is that while this works nicely if you are only overriding a single function, having to do this for all functions and classes in order to make sure you are testing the extension code with all the extension code instead of intermingled extension/Python code. So a function that did this automatically for the entire module would be needed, which is like what I proposed in my use_extension function.

I am seeing two approaches emerging. One is where pickle contains all Python code and then uses something like use_extension to make sure the original Python objects are still reachable at some point. This has the drawback that you have to use some function to make the extensions happen and there is some extra object storage.

The other approach is having pickle contain code known not to be overridden by anyone, import _pypickle for stuff that may be overridden, and then import _pickle for whatever is available. This approach has the perk of using a standard practice for how to pull in different implementation. But the drawback, thanks to how globals are bound, is that any code pulled in from _pickle/_pypickle will not be able to call into other optimized code; it's a take or leave it once the call chain enters one of those modules as they will always call the implementations in the module they originate from.

-Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20090221/6ed52e48/attachment.htm>

More information about the Python-Dev mailing list