[Python-Dev] Inconsistency of PyModule_AddObject() (original) (raw)

Serhiy Storchaka storchaka at gmail.com
Wed Apr 27 03:14:41 EDT 2016

There are three functions (or at least three documented functions) in C API that "steals" references: PyList_SetItem(), PyTuple_SetItem() and PyModule_AddObject(). The first two "steals" references even on failure, and this is well known behaviour. But PyModule_AddObject() "steals" a reference only on success. There is nothing in the documentation that points on this. Most usages of PyModule_AddObject() in the stdlib don't decref the reference to the value on PyModule_AddObject() failure. The only exceptions are in _json, _io, and _tkinter modules. In many cases, including examples in the documentation, the successfulness of PyModule_AddObject() is not checked either, but this is different issue.

We can just fix the documentation but adding a note that PyModule_AddObject() doesn't steal a reference on failure. And add explicit decrefs after PyModule_AddObject() in hundreds of places in the code.

But I think it would be better to "fix" PyModule_AddObject() by making it decrefing a reference on failure as expected by most developers. But this is dangerous change, because if the author of third-party code read not only the documentation, but CPython code, and added explicit decref on PyModule_AddObject() failure, we will get a double decrefing.

I think that we can resolve this issue by following steps:

  1. Add a new function PyModule_AddObject2(), that steals a reference even on failure.

  2. Introduce a special macro like PY_SSIZE_T_CLEAN (any suggestions about a name?). If it is defined, define PyModule_AddObject as PyModule_AddObject2. Define this macro before including Python.h in all CPython modules except _json, _io, and _tkinter.

  3. Make old PyModule_AddObject to emit a warning about possible leak and a suggestion to define above macro.

More information about the Python-Dev mailing list