[Python-Dev] PEP 550 v4 (original) (raw)

Yury Selivanov yselivanov.ml at gmail.com
Mon Aug 28 23:45:03 EDT 2017

On Mon, Aug 28, 2017 at 9:50 PM, Guido van Rossum <guido at python.org> wrote:

On Mon, Aug 28, 2017 at 6:07 PM, Nathaniel Smith <njs at pobox.com> wrote:

The important difference between generators/coroutines and normal function calls is that with normal function calls, the link between the caller and callee is fixed for the entire lifetime of the inner frame, so there's no way for the context to shift under your feet. If all we had were normal function calls, then (green-) thread locals using the save/restore trick would be enough to handle all the use cases above -- it's only for generators/coroutines where the save/restore trick breaks down. This means that pushing/popping LCs when crossing into/out of a generator frame is the minimum needed to get the desired semantics, and it keeps the LC stack small (important since lookups can be O(n) in the worst case), and it minimizes the backcompat breakage for operations like decimal.setcontext() where people do expect to call it in a subroutine and have the effects be visible in the caller. I like this way of looking at things. Does this have any bearing on asyncio.Task? To me those look more like threads than like generators. Or possibly they should inherit the lookup chain from the point when the Task was created, [..]

We explain why tasks have to inherit the lookup chain from the point where they are created in the PEP (in the new High-level Specification section): https://www.python.org/dev/peps/pep-0550/#coroutines-and-asynchronous-tasks

In short, without inheriting the chain we can't wrap coroutines into tasks (like wrapping an await in wait_for() would break the code, if we don't inherit the chain).

In the latest version (v4) we made all coroutines to have their own Logical Context, which, as we discovered today, makes us unable to set context variables in aenter coroutines. This will be fixed in the next version.

FWIW we could have a policy that OS threads also inherit the lookup chain from their creator, but I doubt that's going to fly with backwards compatibility.

Backwards compatibility is indeed an issue. Inheriting the chain for threads would mean another difference between PEP 550 and 'threading.local()', that could cause backwards incompatible behaviour for decimal/numpy when they are updated to new APIs.

For decimal, for example, we could use the following pattern to fallback to use the default decimal context for ECs (threads) that don't have it set:

ctx = decimal_var.get(default=default_decimal_ctx)

We can also add an 'initializer' keyword-argument to 'new_context_var' to specify a callable that will be used to give a default value to the var.

Another issue, is that with the current C API, we can only inherit EC for threads started with 'threading.Thread'. There's no reliable way to inherit the chain if a thread was initialized by a C extension.

IMO, inheriting the lookup chain in threads makes sense when we use them for pools, like concurrent.futures.ThreadPoolExecutor. When threads are used as long-running subprograms, inheriting the chain should be an opt-in.

Yury

More information about the Python-Dev mailing list