[Python-Dev] Pre-PEP: Redesigning extension modules (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Sun Sep 1 16:10:08 CEST 2013

On 1 September 2013 23:03, Antoine Pitrou <solipsis at pitrou.net> wrote:

On Sun, 1 Sep 2013 11:28:36 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote:

* PEP 3121 with a size of "0". As above, but avoids the module state APIs in order to support reloading. All module state (including type cross-references) is stored in hidden state (e.g. an instance of a custom type not exposed to Python, with a reference stored on each custom type object defined in the module, and any module level "functions" actually being methods of a hidden object). Still doesn't support loading a fresh copy due to the hidden PEP 3121 module cache. Not sure what you mean by that:

import atexit id(atexit) 140031896222680 import sys del sys.modules['atexit'] import atexit id(atexit) 140031896221400

Ah, you're right - I misremembered the exact problem that broke xml.etree.ElementTree testing. PyModule_GetState is actually fine (since that pointer is hidden state on the module object), it's only PyState_GetModule that is broken when you import a second copy. So, here, when the second import happens, it breaks the original atexit module's callbacks, even though the two callback registries are properly isolated:

$ ./python Python 3.4.0a1+ (default:575071257c92+, Aug 25 2013, 00:42:17) [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux Type "help", "copyright", "credits" or "license" for more information.

import atexit atexit.register(print, "Hello World!") import sys del sys.modules["atexit"] import atexit as atexit2 atexit2.register(print, "Goodbye World!") Goodbye World!

So I think PEP 3121 is actually as good as we can get on the hidden state front, but the important point is that it is the PyState_GetModule API that can't handle fresh imports - the second import will always replace the first one. So anyone affected needs to find some other way of passing the state, like using bound methods of a hidden type rather than ordinary callables. If you have to interoperate with a C API that only accepts a C callback without allowing additional state arguments, you're going to have trouble.

I think atexit serves as a good example, though - that _Py_PyAtExit call will always be destructive (even if you still have a reference to the original module), so there should be a way for the module to explicitly indicate to the import system "you can only create this module once, and then you're committed - unloading it and importing it again won't work properly due to side effects on the process state".

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list