Request: Can we get a c-api hook into PyContext_Enter and PyContext_Exit (original) (raw)

Some Background:
Ok so at Meta we have this large C++ codebase that we call into from python a ton. That C++ framework “folly” has a concept of a context like python’s context vars. Back in 3.6, someone internally to Meta/Facebook started modifying the asyncio.Task._step at runtime so they could check to see if the C++ contextvar system needed to be updated every time a task is run. Well Task._step became Task.__step in 3.7 and ever since we have been forward porting local changes so we can hook into Task switching.

The Request:
Now hooking into tasks is probably the wrong place to do this, and I think we could be better served integrating to the Python ContextVar concept. Now All I need is a way to hook into contextvar enter/exits so we can use that to switch the C++ level contextvar, right now I’m just thinking of a void* callback that we pass the ctx* object to if its not null.

I don’t think we need more than one callback since nobody has ever asked for this before. Maybe a second api for enter/exit to get the value of the current callback so they could be chained programmatically in the unlikely event there are two things wanting the hook into contextvar switching.

@itamaro is working on forwarding the _step functionality to our new internal 3.12 and its segfaulting and I was wondering is this the signal we need to do this right.

guido (Guido van Rossum) April 24, 2024, 1:18am 2

Hi Jason, it’s bad timing, as the feature freeze for 3.13 (aka beta 1) is May 7, and everyone on the core team is already busy getting their stuff in / polished.

If you’re okay with aiming for 3.14, we should talk. Have you looked at the code for PyContext_Enter() yet? Do you need separate enter/exit hooks? What would the scope of the hook be? Thread state, or per interpreter? Can the hook raise exceptions?

itamaro (Itamar Oren) April 24, 2024, 9:48pm 3

It came up in an offline conversation, so repeating what I (tried to) say here:

  1. I got our Task._step patch working with 3.12 - segfault is gone :slight_smile:
  2. No pressure to have something in 3.13. We are perfectly comfortable backporting things from 3.14 to our internal copy as needed.
  3. Since PyCon US this year is after 3.13 beta cut – shall we use that opportunity to flesh out the details of the desired hook?

guido (Guido van Rossum) April 24, 2024, 10:29pm 4

I’d totally be amenable to getting together e.g. at the post-conference sprints and doing a bit of a design brainstorm. @yselivanov (contextvars inventor) will also be at PyCon, though I don’t know if he’s planning to stay for the sprints.

brettlangdon (Brett Langdon) May 9, 2024, 3:02pm 5

This is similar to a topic I started in the Ideas section, and I was just about to try and start conversations again for it. Maybe the context I have is useful for this discussion?

(this is less about context enter/exit, but more for task enter/exit, so understand if we aren’t really in the same ballpark here)