(original) (raw)
Thanks Tim. That's one unambiguous answer.
I did consider posting to python-list, but this seemed somewhat python-devish. Any appetite for documenting which foundational functions are const-safe in the sense I described? Perhaps we could help. What kind of analysis, demand or momentum is needed for the Python dev team to consider introducing such a contract?
To be honest, I did do some CPython source code staring already. And at least for the functions we care about, it wasn't all that painful (PyDict\_GetItem being the trickiest). Doing this wholesale might be a superhuman task, but I'm thinking a practical subset could be relatively straightforward while still useful, 80/20.
On Tue, Jul 17, 2018 at 7:15 AM, Tim Peters <tim.peters@gmail.com> wrote:
\[Radim Řehůřek <radim@rare-technologies.com>\]one of our Python projects calls for pretty heavy, low-level optimizations.We went down the rabbit hole and determined that having access to PyList\_GET\_ITEM(list), PyInt\_AS\_LONG(int) and PyDict\_GetItem(dict, unicode) on Python objects \*\*outside of GIL\*\* might be a good-enough solution. The Python objects in question are guaranteed to live and not be mutated externally in any way. They're "frozen" and read-only.Under what conditions is it OK to call these 3 functions on such objects?
From the "initialization, Finalization, and Threads" section of the Python/C API Reference Manual:Therefore, the rule exists that only the thread that has acquired the GIL may operate on Python objects or call Python/C API functions.
Under protection of the GIL is the only documented - or intended - way to do anything with Python objects, or to call virtually any Python/C API function.
More generally, what is the CPython 2.7/3.5 contract regarding (lack of) object mutation, and the need for reference counting and synchronization via GIL?
There is no intention to support GIL-free access to any Python objects. So that's the contract: "All warranties are null & void if you do just about anything while not holding the GIL".
Which C API functions are safe to call on "const" objects?
None. If you find some combinations of CPython versions, functions, and objects, that happen to work, that's fine, but there's no guarantee they'll continue to work, not even across maintenance releases. Although they'll \_probably\_ continue to work.Obviously releasing GIL and then calling C API is hacky, but from initial experiments, it seems to work (see https://stackoverflow.com/questions/51351609/can-i- const-access-cpython-objects- without-gil). But I'm wondering if there's a more formal contract around this behaviour.
Nope! You're on your own here.
Which doesn't mean you can't do it. Just that if things blow up, you're still on your own - since you're far outside what the documentation says is required (which I suspect you knew before you asked ;-) ), the project won't even accept a bug report.
If you want more help trying to guess which functions \_may\_ work outside the law, that's fine too, but "python-dev" (this mailing list) is for development \_of\_ Python itself, not for questions about \_using\_ Python. The more general python-list would be more appropriate. But, since you're trying to use the C API in unsupported ways it wasn't intended to be used, you'll likely have a hard time finding people with significant experience doing the same thing. Since it's not at all an intended use, there are no "general principles" at work to ease the pain of staring at the CPython source code to try to guess what might go wrong across all code paths.