[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build (original) (raw)

Steve Dower steve.dower at python.org
Wed Apr 10 14:09:49 EDT 2019

On 10Apr2019 0401, Victor Stinner wrote:

Le mar. 9 avr. 2019 à 22:16, Steve Dower <steve.dower at python.org> a écrit :

What are the other changes that would be required? I don't know.

And is there another way to get the same functionality without ABI modifications? PyTRACEREFS is a double linked list of all Python objects. To get this functionality, you need to store the list somewhere. I don't know how to maintain such list outside the PyObject structure.

There's certainly no more convenient way to do it. Maybe if we had detached reference counts it would be easier, but it would likely still result in ABI compatibility issues between debug builds of extensions and release builds of Python (the most common scenario, in my experience).

One solution would be to enable PyTRACEREFS in release mode. Does anyone want to add 16 bytes to every PyObject? I don't want that :-)

Yeah, nobody suggested that anyway :)

I think it's worthwhile if we can really get to debug and non-debug builds being ABI compatible. Getting partway there in this case doesn't seem to offer any benefits. Disabling PyTRACEREFS by default in debug mode reduces the Python memory footprint. PyTRACEREFS costs 2 pointers per PyObject: 16 bytes on 64-bit platforms.

Right, except it's debug mode.

I don't think that I ever used sys.getobjects(), whereas many projects use gc.getobjects() which is also available in release builds (not only in debug builds).

I'm quite sure that almost nobody uses debug builds because the ABI is incompatible.

There were just over 250,000 downloads of the prebuilt debug binaries for Windows (which are optional in the installer and turned off by default) in March. Whether they are being used is another question, but I know for sure at least a few people who use them.

When you want to use a debug build of your extension module, using a debug build of CPython is the only way to do it. So unless we can get rid of all the ABI incompatibilities, a debug build of CPython is still going to be necessary and disabling/removing reference tracking doesn't provide any benefit.

The main question is if anyone ever used PyTRACEREFS? Does someone use sys.getobjects() or PYTHONDUMPREFS environment variable?

Using PYTHONDUMPREFS=1 on a debug build (with PyTRACEREFS) does simply crash Python 3.7 at exit. So I don't think that anyone use it :-)

How do we track reference leaks in the buildbots? Can/should we be using this?

It doesn't crash on Python 3.8, so I suspect fixing the bug is a better option than using it as an excuse to remove the feature. From a quick test, it seems that a tuple element is being freed but not removed from the tuple, so it's probably a double-decref bug somewhere in 3.7.

Cheers, Steve

More information about the Python-Dev mailing list