[Python-Dev] Problem with PyObject_DEL in pydebug mode (original) (raw)

Victor Stinner victor.stinner at haypocalc.com
Sat Jul 19 13:40:34 CEST 2008

Hi,

I filled an issue about the crash: "import re; re.finditer("a", {})" http://bugs.python.org/issue3299

It appears quickly that the bug is specific to Python compiled in pydebug mode, or to be exact: when Py_TRACE_REFS is defined.

== The Py_TRACE_REFS option ==

The problem is that PyObject_Del(obj) and PyObject_DEL(obj) don't remove obj from the "object list" (_ob_prev and _ob_next attributes of all objects when Py_TRACE_REFS is defined). And so obj will be reused later (maybe removed by garbage collector? or something like that) whereas it's invalid (memory freed).

PyObject_NEW(obj) and PyObject_NEW_VAR(obj) call PyObject_Init() which registers the obj to the object list. So a developer can expect that PyObject_DEL(obj) will remove the object from this list.

PyObject_Del(obj) and PyObject_DEL(obj) are used on object initialization failure, but also in "dealloc" callbacks.

Classic object destruction is done by Py_DECREF(): when reference count is zero, dealloc() is called. PyObject_Del/DEL are different because they just free memory but don't call dealloc() whereas some attributes may be set (and some other may be uninitialized or NULL).

== Solutions ==

(a) Replace PyObject_Del/PyObject_DEL by Py_DECREF to call dealloc callback which will call PyObject_Del/PyObject_DEL. I prefer this solution because dealloc is called and so we make that all attributes are deinitialized.

New problem: dealloc expects that the object is fully initialized (all attributes are set and are not NULL), which is wrong is initialization fails. Eg. with re module,scanner_dealloc() calls Py_DECREF(self->pattern); whereas pattern attribute is NULL. Fix: replace Py_DECREF by Py_XDECREF. But all dealloc have to be reviewed.

(b) Fix PyObject_Del/PyObject_DEL to remove object from the object list

(c) Create new macro which is PyObject_Del/PyObject_DEL + remove the object from the list

(d) Never use Py_TRACE_REFS :-)

I wrote many informations in http://bugs.python.org/issue3299.

-- Victor Stinner aka haypo http://fusil.hachoir.org/

More information about the Python-Dev mailing list