[Python-Dev] Const access to CPython objects outside of GIL? (original) (raw)
Ivan Pozdeev vano at mail.mipt.ru
Tue Jul 17 11:22:00 EDT 2018
- Previous message (by thread): [Python-Dev] Const access to CPython objects outside of GIL?
- Next message (by thread): [Python-Dev] PEP 572 and assert
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On 17.07.2018 7:18, Radim Řehůřek wrote:
Hi all,
one of our Python projects calls for pretty heavy, low-level optimizations. We went down the rabbit hole and determined that having access to PyListGETITEM(list), PyIntASLONG(int) and PyDictGetItem(dict, unicode) on Python objects outside of GIL might be a good-enough solution. The Python objects in question are guaranteed to live and not be mutated externally in any way. They're "frozen" and read-only.
The standard practice if you need to access something outside of GIL is to get a private C object from it. For immutable types, you can get a pointer to the underlying data if the internal representation is compatible with some C type (e.g. char* |PyBytesAsString|(PyObject/ o/) and FILE |PyFileAsFile|(PyObject/ *p/) (Py2 only -- in Py3, PyFile no longer wraps stdio FILE) ); otherwise, the C API can produce a copy (e.g. wchar_t* |PyUnicode_AsWideCharString|(PyObject/ *unicode/, Py_ssize_t/ *size/) ).
Though you can call whatever you want outside of GIL (it's not like we can prevent you), anything that's not officially guaranteed you're doing on your own risk. Even if it happens to work now, it can break in unpredictable ways at any point in the future.
Under what conditions is it OK to call these 3 functions on such objects?
More generally, what is the CPython 2.7/3.5 contract regarding (lack of) object mutation, and the need for reference counting and synchronization via GIL? Which C API functions are safe to call on "const" objects? Obviously releasing GIL and then calling C API is hacky, but from initial experiments, it seems to work (see https://stackoverflow.com/questions/51351609/can-i-const-access-cpython-objects-without-gil). But I'm wondering if there's a more formal contract around this behaviour. Cheers, Radim
Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru
-- Regards, Ivan
-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20180717/e23f5860/attachment-0001.html>
- Previous message (by thread): [Python-Dev] Const access to CPython objects outside of GIL?
- Next message (by thread): [Python-Dev] PEP 572 and assert
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]