[Python-Dev] Inconsistency of PyModule_AddObject() (original) (raw)

Serhiy Storchaka storchaka at gmail.com
Wed Apr 27 13:55:47 EDT 2016

On 27.04.16 10:14, Serhiy Storchaka wrote:

There are three functions (or at least three documented functions) in C API that "steals" references: PyListSetItem(), PyTupleSetItem() and PyModuleAddObject(). The first two "steals" references even on failure, and this is well known behaviour. But PyModuleAddObject() "steals" a reference only on success. There is nothing in the documentation that points on this. Most usages of PyModuleAddObject() in the stdlib don't decref the reference to the value on PyModuleAddObject() failure. The only exceptions are in json, io, and tkinter modules. In many cases, including examples in the documentation, the successfulness of PyModuleAddObject() is not checked either, but this is different issue.

We can just fix the documentation but adding a note that PyModuleAddObject() doesn't steal a reference on failure. And add explicit decrefs after PyModuleAddObject() in hundreds of places in the code. But I think it would be better to "fix" PyModuleAddObject() by making it decrefing a reference on failure as expected by most developers. But this is dangerous change, because if the author of third-party code read not only the documentation, but CPython code, and added explicit decref on PyModuleAddObject() failure, we will get a double decrefing. I think that we can resolve this issue by following steps: 1. Add a new function PyModuleAddObject2(), that steals a reference even on failure. 2. Introduce a special macro like PYSSIZETCLEAN (any suggestions about a name?). If it is defined, define PyModuleAddObject as PyModuleAddObject2. Define this macro before including Python.h in all CPython modules except json, io, and tkinter. 3. Make old PyModuleAddObject to emit a warning about possible leak and a suggestion to define above macro.

Opened an issue: http://bugs.python.org/issue26871 .

Provided patch introduces new macros PY_MODULE_ADDOBJECT_CLEAN that controls the behavior of PyModule_AddObject() as PY_SSIZE_T_CLEAN controls the behavior of PyArg_Parse* functions. If the macro is defined before including "Python.h", PyModule_AddObject() steals a reference unconditionally. Otherwise it steals a reference only on success, and the caller is responsible for decref'ing it on error (current behavior).

More information about the Python-Dev mailing list