[Python-Dev] Strange artifacts with PEP 3121 and monkey-patching sys.modules (in csv, ElementTree and others) (original) (raw)

Eli Bendersky eliben at gmail.com
Sun Aug 11 03:06:02 CEST 2013

n Sat, Aug 10, 2013 at 5:47 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

In a similar vein, Antoine recently noted that the fact the per-module state isn't a real PyObject creates a variety of interesting lifecycle management challenges.

I'm not seeing an easy solution, either, except to automatically skip reinitialization when the module has already been imported. This solution has problems. For example, in the case of ET it would preclude testing what happens when pyexpat is disabled (remember we were discussing this...). This is because there would be no real way to create new instances of such modules (they would all cache themselves in the init function - similarly to what ET now does in trunk, because otherwise some of its global-dependent crazy tests fail).

A more radical solution would be to really have multiple instances of state per sub-interpreter. Well, they already exist -- it's PyState_FindModule which is the problematic one because it only remembers the last one. But I see that it's only being used by extension modules themselves, to efficiently find modules they belong to. It feels a bit like a hack that was made to avoid rewriting lots of code, because in general a module's objects can know which module instance they came from. E.g. it can be saved as a private field in classes exported by the module.

So a more radical approach would be:

PyState_FindModule can be deprecated, but still exist and be documented to return the state the last module created in this sub-interpreter. stdlib extension modules that actually use this mechanism can be rewritten to just remember the module for real, and not rely on PyState_FindModule to fetch it from a global cache. I don't think this would be hard, and it would make the good intention of PEP 3121 more real - actual intependent state per module instance.

Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20130810/5f3333a7/attachment.html>

More information about the Python-Dev mailing list