[Python-Dev] Prevalence of low-level memory abuse? (original) (raw)

Tim Peters tim.peters at gmail.com
Sun Mar 26 07:20:09 CEST 2006

When Python's small-object memory allocator was introduced, some horrid hacks came with it to map PyMem_{Del, DEL} and PyMem_{Free, FREE} to PyObject_{Free, FREE}. This was to cater to less than a handful of extension modules found at the time that obtained memory for an object via PyObject_{New, NEW}, but released that memory via the insanely mismatched PyMem_{Del, DEL} or PyMem_{Free, FREE}.

Since such combinations were found rarely in real life, have been officially forbidden for years, the hacks are ugly & hard to understand, and the hacks needlessly slow PyMem_{Del, DEL, Free, FREE}, I'm trying to get rid of them now. Alas, in a release(*) build, Python's test suite segfaulted all over the place.

So far I found one smoking gun: in _subprocess.c, sp_handle_new() gets memory via PyObject_NEW(), but sp_handle_dealloc() releases that memory via PyMem_DEL(). That's nuts, and after removing the now-ancient hacks obtains the memory from obmalloc but releases it directly to the system free(). That ends up corrupting both obmalloc's and the platform C library's ultra-low-level memory bookkeeping bytes.

Since this wasn't common before, has it become common since then :-)? I checked Zope and ZODB a long time ago, and there were no PyMem/PyObject mismatches there. See any in your code?

(*) Nothing failed in a debug build, since all PyObject_ and PyMem_ calls go thru obmalloc in a debug build. Note that, because of this, the buildbot test runs wouldn't have detected anything wrong.

More information about the Python-Dev mailing list