[Python-Dev] PEP: Post import hooks (original) (raw)

Christian Heimes lists at cheimes.de
Thu Jan 10 03:20:00 CET 2008

I've parted the former PEP as promised. Here is the post import hook. I'll tackle the other half of the PEP later.

PEP: 369 Title: Post import hooks Version: RevisionRevisionRevision Last-Modified: DateDateDate Author: Christian Heimes <christian(at)cheimes(dot)de> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 02-Jan-2008 Python-Version: 2.6, 3.0 Post-History:

Abstract

This PEP proposes enhancements for the import machinery to add post import hooks. It is intended primarily to support the wider use of abstract base classes that is expected in Python 3.0.

The PEP originally started as a combined PEP for lazy imports and post import hooks. After some discussion on the python-dev mailing list the PEP was parted in two separate PEPs. [1]_

Rationale

Python has no API to hook into the import machinery and execute code after a module is successfully loaded. The import hooks of PEP 302 are about finding modules and loading modules but they were not designed to as post import hooks.

Use cases

A use case for a post import hook is mentioned in Nick Coghlan's initial posting [2]_. about callbacks on module import. It was found during the development of Python 3.0 and its ABCs. We wanted to register classes like decimal.Decimal with an ABC but the module should not be imported on every interpreter startup. Nick came up with this example::

@imp.when_imported('decimal') def register(decimal): Inexact.register(decimal.Decimal)

The function register is registered as callback for the module named 'decimal'. When decimal is imported the function is called with the module object as argument.

While this particular example isn't necessary in practice, (as decimal.Decimal will inherit from the appropriate abstract Number base class in 2.6 and 3.0), it still illustrates the principle.

Existing implementations

PJE's peak.util.imports [3]_ implements post load hooks. My implementation shares a lot with his and it's partly based on his ideas.

Post import hook implementation

Post import hooks are called after a module has been loaded. The hooks are callable which take one argument, the module instance. They are registered by the dotted name of the module, e.g. 'os' or 'os.path'.

The callable are stored in the dict sys.post_import_hooks which is a mapping from names (as string) to a list of callables or None.

States

No hook was registered ''''''''''''''''''''''

sys.post_import_hooks contains no entry for the module

A hook is registered and the module is not loaded yet '''''''''''''''''''''''''''''''''''''''''''''''''''''

The import hook registry contains an entry sys.post_import_hooks["name"] = [hook1]

A module is successfully loaded '''''''''''''''''''''''''''''''

The import machinery checks if sys.post_import_hooks contains post import hooks for the newly loaded module. If hooks are found then the hooks are called in the order they were registered with the module instance as first argument. The processing of the hooks is stopped when a method raises an exception. At the end the entry for the module name is removed from sys.post_import_hooks, even when an error has occured.

A module can't be loaded ''''''''''''''''''''''''

The import hooks are neither called nor removed from the registry. It may be possible to load the module later.

A hook is registered but the module is already loaded '''''''''''''''''''''''''''''''''''''''''''''''''''''

The hook is fired immediately.

C API

New PyImport_* API functions ''''''''''''''''''''''''''''

PyObject* PyImport_GetPostImportHooks(void) Returns the dict sys.post_import_hooks or NULL

PyObject* PyImport_NotifyModuleLoaded(PyObject *module) Notify the post import system that a module was requested. Returns the module or NULL if an error has occured.

PyObject* PyImport_RegisterPostImportHook(PyObject *callable, PyObject *mod_name) Register a new hook callable for the module mod_name

The PyImport_PostImportNotify() method is called by PyImport_ImportModuleLevel()::

PyImport_ImportModuleLevel(...) { ... result = import_module_level(name, globals, locals, fromlist, level); result = PyImport_PostImportNotify(result); ... }

Python API

The import hook registry and two new API methods are exposed through the sys and imp module.

sys.post_import_hooks The dict contains the post import hooks: {"name" : [hook1, hook2], ...}

imp.register_post_import_hook(hook: "callable", name: str) Register a new hook hook for the module name

imp.notify_module_loaded(module: "module instance") -> module Notify the system that a module has been loaded. The method is provided for compatibility with existing lazy / deferred import extensions.

The when_imported function decorator is also in the imp module, which is equivalent to:

def when_imported(name): def register(hook): register_post_import_hook(hook, name) return register

imp.when_imported(name) -> decorator function for @when_imported(name) def hook(module): pass

Open issues

The when_imported decorator hasn't been written.

The code contains several XXX comments. They are mostly about error handling in edge cases.

Backwards Compatibility

The new features and API don't conflict with old import system of Python and don't cause any backward compatibility issues for most software. However systems like PEAK and Zope which implement their own lazy import magic need to follow some rules.

The post import hooks carefully designed to cooperate with existing deferred and lazy import systems. It's the suggestion of the PEP author to replace own on-load-hooks with the new hook API. The alternative lazy or deferred imports will still work but the implementations must call the imp.notify_module_loaded function.

Reference Implementation

A reference implementation is already written and is available in the py3k-importhook branch. [4]_ It still requires some cleanups, documentation updates and additional unit tests.

Acknowledgments

Nick Coghlan, for proof reading and the initial discussion Phillip J. Eby, for his implementation in PEAK and help with my own implementation

Copyright

This document has been placed in the public domain.

References

.. [1] PEP: Lazy module imports and post import hook http://permalink.gmane.org/gmane.comp.python.devel/90949

.. [2] Interest in PEP for callbacks on module import http://permalink.gmane.org/gmane.comp.python.python-3000.devel/11126

.. [3] peak.utils.imports http://svn.eby-sarna.com/Importing/peak/util/imports.py?view=markup

.. [4] py3k-importhook branch http://svn.python.org/view/python/branches/py3k-importhook/

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

More information about the Python-Dev mailing list