[Python-Dev] Const access to CPython objects outside of GIL? (original) (raw)

Victor Stinner vstinner at redhat.com
Tue Jul 17 07:34:27 EDT 2018

2018-07-17 6:18 GMT+02:00 Radim Řehůřek <radim at rare-technologies.com>:

one of our Python projects calls for pretty heavy, low-level optimizations.

We went down the rabbit hole and determined that having access to PyListGETITEM(list), PyIntASLONG(int) and PyDictGetItem(dict, unicode) on Python objects outside of GIL might be a good-enough solution. The Python objects in question are guaranteed to live and not be mutated externally in any way. They're "frozen" and read-only.

IMHO it's a great path to introduce very tricky race conditions in multithreaded applications! :-)

At the C level, even immutable Python objects are mutable: str, tuple, etc.

IMHO you need a different approach to implement optimizations. For example, use your objects which don't rely on the GIL to be consistent. Sadly, I have no experience with that and so cannot provide any example.

Python releases the GIL often and pass pointers to buffers to C functions. For example, os.stat() writes into a memory block allocated by Python. But it's a raw memory block, it's much simpler than a Python object like a tuple.

Victor

More information about the Python-Dev mailing list