[Python-Dev] PEP 567 -- Context Variables (original) (raw)

Yury Selivanov yselivanov.ml at gmail.com
Wed Dec 13 16:35:59 EST 2017

Hi Eric,

Thanks for a detailed review!

On Wed, Dec 13, 2017 at 3:59 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:

Overall, I like this PEP. It's definitely easier to follow conceptually than PEP 550. Thanks for taking the time to re-think the idea. I have a few comments in-line below.

-eric On Tue, Dec 12, 2017 at 10:33 AM, Yury Selivanov <yselivanov.ml at gmail.com> wrote: This is a new proposal to implement context storage in Python. +1 This is something I've had on my back burner for years. Getting this right is non-trivial, so having a stdlib implementation will help open up clean solutions in a number of use cases that are currently addressed in more error-prone ways.

Right!

It's a successor of PEP 550 and builds on some of its API ideas and datastructures. Contrary to PEP 550 though, this proposal only focuses on adding new APIs and implementing support for it in asyncio. There are no changes to the interpreter or to the behaviour of generator or coroutine objects. Do you have any plans to revisit extension of the concept to generators and coroutine objects? I agree they can be addressed separately, if necessary. TBH, I'd expect this PEP to provide an approach that allows such applications of the concept to effectively be implementation details that can be supported later.

Maybe we'll extend the concept to work for generators in Python 3.8, but that's a pretty remote topic to discuss (and we'll need a new PEP for that). In case we decide to do that, PEP 550 provides a good implementation plan, and PEP 567 are forward-compatible with it.

Abstract ========

This PEP proposes the new contextvars module and a set of new CPython C APIs to support context variables. This concept is similar to thread-local variables but, unlike TLS, it allows s/it allows/it also allows/

Will fix it.

[..]

A new standard library module contextvars is added Why not add this to contextlib instead of adding a new module? IIRC this was discussed relative to PEP 550, but I don't remember the reason. Regardless, it would be worth mentioning somewhere in the PEP.

The mechanism is generic and isn't directly related to context managers. Context managers can (and in many cases should) use the new APIs to store global state, but the contextvars APIs do not depend on context managers or require them.

I also feel that contextlib is a big module already, so having the new APIs in their separate module and having a separate documentation page makes it more approachable.

with the following APIs:

1. getcontext() -> Context function is used to get the current Context object for the current OS thread. 2. ContextVar class to declare and access context variables. It may be worth explaining somewhere in the PEP the reason why you've chosen to add ContextVar instead of adding a new keyword (e.g. "context", a la global and nonlocal) to do roughly the same thing. Consider that execution contexts are very much a language-level concept, a close sibling to scope. Driving that via a keyword would a reasonable approach, particularly since it introduces less coupling between a language-level feature and a stdlib module. (Making it a builtin would sort of help with that too, but a keyword would seem like a better fit.) A keyword would obviate the need for explicitly calling .get() and .set(). FWIW, I agree with not adding a new keyword. To me context variables are a low-level tool for library authors to implement their high-level APIs. ContextVar, with its explicit .get() and .set() methods is a good fit for that and better communicates the conceptual intent of the feature. However, it would still be worth explicitly mentioning the alternate keyword-based approach in the PEP.

Yeah, adding keywords is way harder than adding a new module. It would require a change in Grammar, new opcodes, changes to frameobject etc. I also don't think that ContextVars will be that popular to have their own syntax -- how many threadlocals do you see every day?

For PEP 567/550 a keyword isn't really needed, we can implement the concept with a ContextVar class.

3. Context class encapsulates context state. Every OS thread stores a reference to its current Context instance. It is not possible to control that reference manually. Instead, the Context.run(callable, *args) method is used to run Python code in another context. I'd call that "Context.call()" since its for callables. Did you have a specific reason for calling it "run" instead?

We have a bunch of run() methods in asyncio, and as I'm actively working on its codebase I might be biased here, but ".run()" reads better for me personally than ".call()".

FWIW, I think there are some helpers you could add that library authors would appreciate. However, they aren't critical so I'll hold off and maybe post about them later. :)

My goal with this PEP is to keep the API to its bare minimum, but if you have some ideas please share!

contextvars.ContextVar ----------------------

The ContextVar class has the following constructor signature: ContextVar(name, *, default=nodefault). The name parameter is used only for introspection and debug purposes. It doesn't need to be required then, right?

If it's not required then people won't use it. And then when you want to introspect the context, you'll see a bunch of anonymous variables.

So as with namedtuple(), I think there'd no harm in requiring the name parameter.

[snip]

ContextVar.set(value) -> Token is used to set a new value for the context variable in the current Context:: # Set the variable 'var' to 1 in the current context. var.set(1) contextvars.Token is an opaque object that should be used to restore the ContextVar to its previous value, or remove it from the context if it was not set before. The ContextVar.reset(Token) is used for that:: old = var.set(1) try: ... finally: var.reset(old) The Token API exists to make the current proposal forward compatible with :pep:550, in case there is demand to support context variables in generators and asynchronous generators in the future. The "restoring values" focus is valuable on its own, It emphasizes a specific usage pattern to users (though a context manager would achieve the same). The token + reset() approach means that users don't need to think about "not set" when restoring values. That said, is there otherwise any value to the "not set" concept? If so, "isset()" (not strictly necessary) and "unset()" methods may be warranted.

"unset()" would be incompatible with PEP 550, which has a chained execution context model. When you have a chain of contexts, unset() becomes ambiguous.

"is_set()" is trivially implemented via "var.get(default=marker) is marker", but I don't think people will need this to add this method now.

[..]

Any changes to the context will be contained and persisted in the Context object on which run() is called on. For me this would be more clear if it could be spelled like this: with ctx: function()

But we would still need "run()" to use in asyncio. Context managers are slower that a single method call.

Also, context management like this is a very low-level API intended to be used by framework/library authors in very few places. Again, I'd really prefer to keep the API to the minimum in 3.7.

Also, let's say I want to run a function under a custom context, whether a fresh one or an adaptation of an existing one. How can I compose such a Context? AFAICS, the only way to modify a context is by using ContextVar.set() (and reset()), which modifies the current context. It might be useful if there were a more direct way, like a "Context.add(*var) -> Context" and "Context.remove(*var) -> Context" and maybe even a "Context.set(var, value) -> Context" and "Context.unset(var) -> Context".

Again this would be a shortcut for a very limited number of use-cases. I just can't come up with a good real-world example where you want to add many context variables to the context and run something in it. But even if you want that, you can always just wrap your function:

def set_and_call(var, val, func):
      var.set(val)
      return func()

context.run(set_and_call, var, val, func)

Yury

More information about the Python-Dev mailing list