[Python-Dev] PEP 573 -- Module State Access from C Extension Methods (original) (raw)

Thomas Wouters thomas at python.org
Thu Apr 26 09:32:38 EDT 2018

Thanks for working on this, Marcel (and Petr). This looks like an ambitious intern project :) Couple of questions and comments in-line.

On Mon, Apr 23, 2018 at 12:36 PM, Marcel Plch <gmarcel.plch at gmail.com> wrote:

Hello, I am an intern at Red Hat mentored by Petr Viktorin. As a part of my internship, I learned the CPython internals and how to contribute to the CPython interpreter.

As a result, I have prepared PEP 573, which solves some problems that PEP 489 (Multi-phase extension module initialization) has left open. Specifically, this PEP proposes a way to access per-module state from methods of built-in and extension types. Like PEP 489, it aims to make subinterpreter-friendly built-in/extension modules easier to create. A big problem found when converting many modules to PEP 489 multi-phase initialization is subinterpreter-friendly access to exception types defined in built-in/extension modules. This PEP solves this by introducing "immutable exception types". The current implementation requires one new type flag and two new pointers in the heap type structure. It should be possible to remove eiher the flag or one of the two pointers, if we agree on the other mechanics in the PEP .

=================== PEP: 573 Title: Module State Access from C Extension Methods Version: RevisionRevisionRevision Last-Modified: DateDateDate Author: Petr Viktorin <encukou at gmail.com>, Nick Coghlan <ncoghlan at gmail.com>, Eric Snow <ericsnowcurrently at gmail.com>, Marcel Plch <gmarcel.plch at gmail.com> Discussions-To: import-sig at python.org Status: Active Type: Process Content-Type: text/x-rst Created: 02-Jun-2016 Python-Version: 3.8 Post-History: Abstract ======== This PEP proposes to add a way for CPython extension methods to access context such as the state of the modules they are defined in. This will allow extension methods to use direct pointer dereferences rather than PyStateFindModule for looking up module state, reducing or eliminating the performance cost of using module-scoped state over process global state. This fixes one of the remaining roadblocks for adoption of PEP 3121 (Extension module initialization and finalization) and PEP 489 (Multi-phase extension module initialization). Additionaly, support for easier creation of immutable exception classes is added.

I'm not a fan of using 'immutable' here, or in the API function name. I understand the types are to some extent immutable (apart from their refcount, I assume), but I think it's going to be too easy to confuse it with types whose instances are immutable. (We do occasionally say things like "tuples are an immutable type".) Since the point is that they behave like statically defined ones, perhaps 'Static' would be a reasonable replacement.

This removes the need for keeping per-module state if it would only be used

for exception classes.

While this PEP takes an additional step towards fully solving the problems that PEP 3121 and PEP 489 started tackling, it does not attempt to resolve all remaining concerns. In particular, accessing the module state from slot methods (nbadd, etc) remains slower than accessing that state from other extension methods.

Terminology =========== Process-Global State -------------------- C-level static variables. Since this is very low-level memory storage, it must be managed carefully. Per-module State ---------------- State local to a module object, allocated dynamically as part of a module object's initialization. This isolates the state from other instances of the module (including those in other subinterpreters). Accessed by PyModuleGetState(). Static Type ----------- A type object defined as a C-level static variable, i.e. a compiled-in type object. A static type needs to be shared between module instances and has no information of what module it belongs to. Static types do not have _dict_ (although their instances might). Heap Type --------- A type object created at run time. Rationale ========= PEP 489 introduced a new way to initialize extension modules, which brings several advantages to extensions that implement it: * The extension modules behave more like their Python counterparts. * The extension modules can easily support loading into pre-existing module objects, which paves the way for extension module support for runpy or for systems that enable extension module reloading. * Loading multiple modules from the same extension is possible, which makes testing module isolation (a key feature for proper sub-interpreter support) possible from a single interpreter. The biggest hurdle for adoption of PEP 489 is allowing access to module state from methods of extension types. Currently, the way to access this state from extension methods is by looking up the module via PyStateFindModule (in contrast to module level functions in extension modules, which receive a module reference as an argument). However, PyStateFindModule queries the thread-local state, making it relatively costly compared to C level process global access and consequently deterring module authors from using it. Also, PyStateFindModule relies on the assumption that in each subinterpreter, there is at most one module corresponding to a given PyModuleDef. This does not align well with Python's import machinery. Since PEP 489 aimed to fix that, the assumption does not hold for modules that use multi-phase initialization, so PyStateFindModule is unavailable for these modules. A faster, safer way of accessing module-level state from extension methods is needed. Immutable Exception Types ------------------------- For isolated modules to work, any class whose methods touch module state must be a heap type, so that each instance of a module can have its own type object. With the changes proposed in this PEP, heap type instances will have access to module state without global registration. But, to create instances of heap types, one will need the module state in order to get the type object corresponding to the appropriate module. In short, heap types are "viral" – anything that “touches” them must itself be a heap type. Curently, most exception types, apart from the ones in builtins, are heap types. This is likely simply because there is a convenient way to create them: PyErrNewException. Heap types generally have a mutable _dict_. In most cases, this mutability is harmful. For example, exception types from the sqlite module are mutable and shared across subinterpreters. This allows "smuggling" values to other subinterpreters via attributes of sqlite3.Error. Moreover, since raising exceptions is a common operation, and heap types will be "viral", PyErrNewException will tend to "infect" the module with "heap type-ness" – at least if the module decides play well with subinterpreters/isolation. Many modules could go without module state entirely if the exception classes were immutable. To solve this problem, a new function for creating immutable exception types is proposed. Background =========== The implementation of a Python method may need access to one or more of the following pieces of information: * The instance it is called on (self) * The underlying function * The class the method was defined in * The corresponding module * The module state In Python code, the Python-level equivalents may be retrieved as:: import sys def meth(self): instance = self moduleglobals = globals() moduleobject = sys.modules[name] # (1) underlyingfunction = Foo.meth # (1) definingclass = Foo # (1) definingclass = class # (2) .. note:: The defining class is not type(self), since type(self) might be a subclass of Foo. The statements marked (1) implicitly rely on name-based lookup via the function's _globals_: either the Foo attribute to access the defining class and Python function object, or _name_ to find the module object in sys.modules. In Python code, this is feasible, as _globals_ is set appropriately when the function definition is executed, and even if the namespace has been manipulated to return a different object, at worst an exception will be raised. The _class_ closure, (2), is a safer way to get the defining class, but it still relies on _closure_ being set appropriately. By contrast, extension methods are typically implemented as normal C functions. This means that they only have access to their arguments and C level thread-local and process-global states. Traditionally, many extension modules have stored their shared state in C-level process globals, causing problems when: * running multiple initialize/finalize cycles in the same process * reloading modules (e.g. to test conditional imports) * loading extension modules in subinterpreters PEP 3121 attempted to resolve this by offering the PyStateFindModule API, but this still has significant problems when it comes to extension methods (rather than module level functions): * it is markedly slower than directly accessing C-level process-global state * there is still some inherent reliance on process global state that means it still doesn't reliably handle module reloading It's also the case that when looking up a C-level struct such as module state, supplying an unexpected object layout can crash the interpreter, so it's significantly more important to ensure that extension methods receive the kind of object they expect. Proposal ======== Currently, a bound extension method (PyCFunction or PyCFunctionWithKeywords) receives only self, and (if applicable) the supplied positional and keyword arguments. While module-level extension functions already receive access to the defining module object via their self argument, methods of extension types don't have that luxury: they receive the bound instance via self, and hence have no direct access to the defining class or the module level state. The additional module level context described above can be made available with two changes. Both additions are optional; extension authors need to opt in to start using them: * Add a pointer to the module to heap type objects. * Pass the defining class to the underlying C function. The defining class is readily available at the time built-in method object (PyCFunctionObject) is created, so it can be stored in a new struct that extends PyCFunctionObject. The module state can then be retrieved from the module object via PyModuleGetState. Note that this proposal implies that any type whose method needs to access per-module state must be a heap type, rather than a static type. This is necessary to support loading multiple module objects from a single extension: a static type, as a C-level global, has no information about which module it belongs to. Slot methods ------------ The above changes don't cover slot methods, such as tpiter or nbadd. The problem with slot methods is that their C API is fixed, so we can't simply add a new argument to pass in the defining class. Two possible solutions have been proposed to this problem: * Look up the class through walking the MRO. This is potentially expensive, but will be useful if performance is not a problem (such as when raising a module-level exception). * Storing a pointer to the defining class of each slot in a separate table, _typeslots_ [#typeslots-mail]. This is technically feasible and fast, but quite invasive. Due to the invasiveness of the latter approach, this PEP proposes adding an MRO walking helper for use in slot method implementations, deferring the more complex alternative as a potential future optimisation. Modules affected by this concern also have the option of using thread-local state or PEP 567 context variables, or else defining their own reload-friendly lookup caching scheme.

I do not believe walking the MRO is going to work without reworking the implementation of types, specifically how typeobject.c deals with slots of subclasses: in some cases copies the slots from the base class (see inherit_slots() and from where it's called). I believe this would cause problems if, for example, you define type X in module A, subclass it from type Y in module B without overriding the slot, and try to find the module object for A from the slot implementation. I don't think copying slots is a requirement for the desired semantics, but it's going to be fairly involved to rewrite it to do something else. There's also backward-compatibility to consider: third-party libraries can be inheriting from builtin types (e.g. numpy does this extensively) using the same copying-slot mechanism, which means those builtin types can't use the MRO walking to find their module without breaking compatibility with those third-party libraries.

Immutable Exception Types ------------------------- To faciliate creating static exception classes, a new function is proposed: PyErrPrepareImmutableException. It will work similarly to PyErrNewExceptionWithDoc but will take a PyTypeObject ** pointer, which points to a PyTypeObject * that is either NULL or an initialized PyTypeObject. This pointer may be declared in process-global state. The function will then allocate the object and will keep in mind that already existing exception should not be overwritten. The extra indirection makes it possible to make PyErrPrepareImmutableException part of the stable ABI by having the Python interpreter, rather than extension code, allocate the PyTypeObject. Specification ============= Adding module references to heap types -------------------------------------- The PyHeapTypeObject struct will get a new member, ``PyObject *htmodule``, that can store a pointer to the module object for which the type was defined. It will be NULL by default, and should not be modified after the type object is created. A new factory method will be added for creating modules:: PyObject* PyTypeFromModuleAndSpec(PyObject *module, PyTypeSpec *spec, PyObject *bases) This acts the same as PyTypeFromSpecWithBases, and additionally sets htmodule to the provided module object. Additionally, an accessor, PyObject * PyTypeGetModule(PyTypeObject *) will be provided. It will return the htmodule if a heap type with module pointer set is passed in, otherwise it will set a SystemError and return NULL. Usually, creating a class with htmodule set will create a reference cycle involving the class and the module. This is not a problem, as tearing down modules is not a performance-sensitive operation (and module-level functions typically also create reference cycles). The existing "set all module globals to None" code that breaks function cycles through fglobals will also break the new cycles through htmodule. Passing the defining class to extension methods ----------------------------------------------- A new style of C-level functions will be added to the current selection of PyCFunction and PyCFunctionWithKeywords:: PyObject *PyCMethod(PyObject *self, PyTypeObject *definingclass, PyObject *args, PyObject *kwargs) A new method object flag, METHMETHOD, will be added to signal that the underlying C function is PyCMethod. To hold the extra information, a new structure extending PyCFunctionObject will be added:: typedef struct { PyCFunctionObject func; PyTypeObject mmclass; / Passed as 'definingclass' arg to the C func */ } PyCMethodObject; To allow passing the defining class to the underlying C function, a change to private API is required, now PyMethodDefRawFastCallDict and PyMethodDefRawFastCallKeywords will receive PyTypeObject *cls as one of their arguments. A new macro PyCFunctionGETCLASS(cls) will be added for easier access to mmclass. Method construction and calling code and will be updated to honor METHMETHOD. Argument Clinic --------------- To support passing the defining class to methods using Argument Clinic, a new converter will be added to clinic.py: definingclass. Each method may only have one argument using this converter, and it must appear after self, or, if self is not used, as the first argument. The argument will be of type PyTypeObject *. When used, Argument Clinic will select METHMETHOD as the calling convention. The argument will not appear in _textsignature_. This will be compatible with _init_ and _new_ methods, where an MRO walker will be used to pass the defining class from clinic generated code to the user's function. Slot methods ------------ To allow access to per-module state from slot methods, an MRO walker will be implemented:: PyTypeObject *PyTypeDefiningTypeFromSlotFunc(PyTypeObject *type, int slot, void *func) The walker will go through bases of heap-allocated type and search for class that defines func at its slot. The func needs not to be inherited by type, only requirement for the walker to find the defining class is that the defining class must be heap-allocated. On failure, exception is set and NULL is returned. Static exceptions ----------------- A new function will be added:: int PyErrPrepareImmutableException(PyTypeObject **exc, const char *name, const char *doc, PyObject *base) Creates an immutable exception type which can be shared across multiple module objects.

How is this going to deal with type.subclasses()? Is re-using the static type object between reloads and sub-interpreters important enough to warrant the different behaviour? What if sub-interpreters end up wanting to disallow sharing objects between them?

If the type already exists (determined by a process-global pointer,

*exc), skip the initialization and only INCREF it.

If *exc is NULL, the function will allocate a new exception type and initialize it using given parameters the same way PyTypeFromSpecAndBases would. The doc and base arguments may be NULL, defaulting to a missing docstring and PyExcException base class, respectively. The exception type's tpflags will be set to values common to built-in exceptions and the PyTPFLAGSHEAPIMMUTABLE flag (see below) will be set. On failure, PyErrPrepareImmutableException will set an exception and return -1. If called with an initialized exception type (*exc is non-NULL), the function will do nothing but incref *exc. A new flag, PyTPFLAGSHEAPIMMUTABLE, will be added to prevent mutation of the type object. This makes it possible to share the object safely between multiple interpreters. This flag is checked in typesetattro and blocks setting of attributes when set, similar to built-in types. A new pointer, htmoduleptr, will be added to heap types to store exc. On deinitialization of the exception type, *exc will be set to NULL. This makes it safe for PyErrPrepareImmutableException to check if the exception was already initialized. PyTypeoffsets -------------- Some extension types are using instances with _dict_ or _weakref_ allocated. Currently, there is no way of passing offsets of these through PyTypeSpec. To allow this, a new structure and a spec slot are proposed. A new structure, PyTypeoffsets, will have two members containing the offsets of _dict_ and _weakref_:: typedef struct { Pyssizet dict; Pyssizet weaklist; } PyTypeoffsets; The new slot, Pyoffsets, will be used to pass a PyTypeoffsets * structure containing the mentioned data.

Helpers ------- Getting to per-module state from a heap type is a very common task. To make this easier, a helper will be added:: void *PyTypeGetModuleState(PyObject *type) This function takes a heap type and on success, it returns pointer to state of the module that the heap type belongs to. On failure, two scenarios may occure. When a type without a module is passed in, SystemError is set and NULL returned. If the module is found, pointer to the state, which may be NULL, is returned without setting any exception. Modules Converted in the Initial Implementation ----------------------------------------------- To validate the approach, several modules will be modified during the initial implementation: The zipimport, io, elementtree, and csv modules will be ported to PEP 489 multiphase initialization.

zipimport currently caches things in C globals. Changing it to use PEP 489 multi-phase initialisation is very likely going to change semantics in subtle ways... Is it really worth the risk?

Summary of API Changes and Additions ==================================== New functions: * PyTypeGetModule * PyTypeDefiningTypeFromSlotFunc * PyTypeGetModuleState * PyErrPrepareImmutableException New macros: * PyCFunctionGETCLASS New types: * PyCMethodObject New structures: * PyTypeoffsets Modified functions: * PyMethodDefRawFastCallDict now receives PyTypeObject *cls. * PyMethodDefRawFastCallKeywords now receives PyTypeObject *cls. Modified structures: * heaptypeobject - added htmodule and htmoduleptr Other changes: * METHMETHOD call flag * definingclass converter in clinic * PyTPFLAGSHEAPIMMUTABLE flag * Pyoffsets type spec slot Backwards Compatibility ======================= Two new pointers are added to all heap types. All other changes are adding new functions, structures and a type flag. The new PyErrPrepareImmutableException function changes encourages modules to switch from using heap type Exception classes to immutable ones, and a number of modules will be switched in the initial implementation. This change will prevent adding class attributes to such types. For example, the following will raise AttributeError:: sqlite.OperationalError.foo = None Instances and subclasses of such exceptions will not be affected. Implementation ============== An initial implementation is available in a Github repository [#gh-repo]; a patchset is at [#gh-patch]. Possible Future Extensions ========================== Easy creation of types with module references --------------------------------------------- It would be possible to add a PEP 489 execution slot type to make creating heap types significantly easier than calling PyTypeFromModuleAndSpec. This is left to a future PEP. Optimization ------------ CPython optimizes calls to methods that have restricted signatures, such as not allowing keyword arguments. As proposed here, methods defined with the METHMETHOD flag do not support these optimizations. Optimized calls still have the option of accessing per-module state the same way slot methods do. References ========== .. [#typeslots-mail] [Import-SIG] On singleton modules, heap types, and subinterpreters (https://mail.python.org/pipermail/import-sig/2015-July/001035.html) .. [#gh-repo] https://github.com/Traceur759/cpython/commits/pep-c .. [#gh-patch] https://github.com/Traceur759/cpython/compare/master... Traceur759:pep-c.patch Copyright ========= This document has been placed in the public domain.

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ thomas%40python.org

-- Thomas Wouters <thomas at python.org>

Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20180426/d065a6dd/attachment-0001.html>

More information about the Python-Dev mailing list