[Python-Dev] PEP 550 V5 (original) (raw)

Yury Selivanov yselivanov.ml at gmail.com
Sun Sep 3 07:09:19 EDT 2017

Hi Eric,

On Fri, Sep 1, 2017 at 11:29 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:

Nice working staying on top of this! Keeping up with discussion is arguably much harder than actually writing the PEP. :) I have some comments in-line below.

Thanks!

-eric

On Fri, Sep 1, 2017 at 5:02 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote: [snip]

Abstract ======== [snip] Rationale ========= [snip] Goals ===== I still think that the Abstract, Rationale, and Goals sections should be clear that a major component of this proposal is lookup via chained contexts. Without such clarity it may not be apparent that chained lookup is not strictly necessary to achieve the stated goals (i.e. an async-compatible TLS replacement). This matters because the chaining introduces extra non-trivial complexity.

Let me walk you through the history of PEP 550 to explain how we arrived to having "chained lookups".

If we didn't have generators in Python, the first version of PEP 550 ([1]) would have been a perfect solution. First let's recap the how all versions of PEP 550 reason about generators:

PEP 550 v1 had excellent performance characteristics, a specification that was easy to fully understand, and a relatively straightforward implementation. It also had no chained lookups, as there was only one thread-specific collection of context values.

PEP 550 states that v1 was rejected because of the following reason: "The fundamental limitation that caused a complete redesign of the first version was that it was not possible to implement an iterator that would interact with the EC in the same way as generators". I believe it's no longer true: I think that we could use the same trick [2] we use in the current version of PEP 550. But that doesn't really matter.

The fundamental problem of PEP 550 v1 was that generators could not see EC updates while being iterated. Let's look at the following:

 def gen():
      yield some_decimal_calculation()
      yield some_decimal_calculation()
      yield some_decimal_calculation()

 with decimal_context_0:
     g = gen()
 with decimal_context_1:
     next(g)
 with decimal_context_2:
     next(g)

In the above example, with PEP 550 v1, generator 'g' would snapshot the execution context at the point where it was created. So all calls to "some_decimal_calculation()" would be made with "decimal_context_0". We could easily change the semantics to snapshot the EC at the point of the first iteration, but that would only make "some_decimal_calculation()" to be always called with "decimal_context_1".

In this email [3], Nathaniel explained that there are use cases where seeing updates in the surrounding Execution Context matters in some situations. One case is his idea to implement special timeouts handling in his async framework Trio. Another one, is backwards compatibility: currently, if you use a "threading.local()", you should be able to see updates in it while the generator is being iterated. With PEP 550 v1, you couldn't.

To fix this problem, in later versions of PEP 550, we transformed the EC into a stack of Logical Contexts. Every generator has its own LC, which contains changes to the EC local to that generator. Having a stack naturally made it a requirement to have "chained" lookups. That's how we fixed the above example!

Therefore, while the current version of PEP 550 is more complex than v1, it has a proper support of generators. It provides strong guarantees w.r.t. context isolation, and at the same time maintains their ability to see the full outer context while being iterated.

[..]

Speaking of which, I have plans for the near-to-middle future that involve making use of the PEP 550 functionality in a way that is quite similar to decimal. However, it sounds like the implementation of such (namespace) contexts under PEP 550 is much more complex than it is with threading.local (where subclassing made it easy). It would be helpful to have some direction in the PEP on how to port to PEP 550 from threading.local. It would be even better if the PEP included the addition of a contextlib.Context or contextvars.Context class (or NamespaceContext or ContextNamespace or ...). :) However, I recognize that may be out of scope for this PEP.

Currently we are focused on refining the fundamental low-level APIs and data structures, and the scope we cover in the PEP is already huge. Adding high-level APIs is much easier, so yeah, I'd discuss them if/after the PEP is accepted.

Thanks! Yury

[1] https://github.com/python/peps/blob/e8a06c9a790f39451d9e99e203b13b3ad73a1d01/pep-0550.rst [2] https://www.python.org/dev/peps/pep-0550/#generators-transformed-into-iterators [3] https://mail.python.org/pipermail/python-ideas/2017-August/046736.html

More information about the Python-Dev mailing list