[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode (original) (raw)

Neil Schemenauer nas-python at arctrix.com
Mon Apr 29 20:01:41 EDT 2019

On 2019-04-27, Nathaniel Smith wrote:

For PyTRACEREFS specifically, IIUC the only goal is to be able to produce a list of all live objects on demand. If that's the goal, then static type objects aren't a huge deal. You can't add extra data into the type objects themselves, but since there's a fixed set of them and they're immortal, you can just build a static list of all of them in PyTypeReady.

As far as I understand, we have a similar problem already for gc.get_objects() because those static type objects don't have a PyGC_Head. My 2-cent proposal for fixing things in the long term would be to introduce a function like PyType_Ready that returns a pointer to the new type. The argument to it would be what is the current static type structure. The function would copy things from the static type structure into a newly allocated type structure.

We have a kind of solution already with PyType_FromSpec, etc. However, I think it is harder to convert existing extension module source code to use that API. We want to make it very easy for people to fix source code.

If we can remove static types, that would allow us to kill off Py_TYPE(o)->tp_is_gc(o). I understand why that exists but I think it is quite an ugly detail of the current GC implementation. I wonder about the performance impact of it given current memory latencies. When we do a full GC run, we call PyObject_IS_GC() on many objects. I fear having to lookup and call tp_is_gc could be quite expensive.

I've been playing with the idea of using memory bitmaps rather then the PyGC_Head. That idea seems to depend on removing static type objects. Initially I was thinking of it as reducing the memory overhead for GC types. Now I think the memory overhead doesn't matter too much but perhaps the bitmaps would be much faster due to memory latency. There is an interesting Youtube video that compares vector traversals vs linked list traversals in C++. Linked lists on modern machines are really terrible.

Regards,

Neil

More information about the Python-Dev mailing list