[Python-Dev] On suppress()'s trail blazing (was Re: cpython: Rename contextlib.ignored() to contextlib.ignore()) (original) (raw)

Guido van Rossum guido at python.org
Fri Oct 18 01:02:28 CEST 2013

On Thu, Oct 17, 2013 at 3:51 PM, Oscar Benjamin <oscar.j.benjamin at gmail.com>wrote:

On 17 October 2013 20:01, Guido van Rossum <guido at python.org> wrote: > On Thu, Oct 17, 2013 at 11:55 AM, Oscar Benjamin > <oscar.j.benjamin at gmail.com> wrote: >> >> On 17 October 2013 19:40, Xavier Morel <python-dev at masklinn.net> wrote: >> > I think there's already a significant split between context managers >> > which handle the lifecycle of a local resource (file, transaction) and >> > those which purport to locally alter global-ish state (cwd, >> > decimal.localcontext, logging.captureWarnings, redirectstdout). >> > >> > And the latter worries me (much more than the very localized behavior of >> > suppress) because I don't see any way to implement them safely and >> > correctly when mixing it with coroutines in today's Python (some of them >> > aren't even thread-safe), all of that while I expect coroutines will see >> > significantly more use in the very near future with yield from and >> > tulip's promotion of coroutine-style async. >> >> I maybe misunderstanding how the coroutine-style async works but I >> would have thought that it would be as simple as: don't use >> global-state-restoring-context-managers around statements that yield >> control (it would be simpler if there was a good term for describing >> that kind of CM). That's simpler to implement and computationally >> cheaper than e.g. the thread-local state used by the decimal module. > > Context managers that actually save and restore global state are already > not thread-safe, so concluding they are also not coroutine-safe (or > task-safe?) seems a small step. > > I'd be more worried about context manager that use thread-local state -- > there is no similar concept in Tulip.

It's unnecessary in Tulip. The need for thread-local state in e.g. decimal contexts is driven by the fact that multi-threaded execution switches in an uncontrollable way. Tulip specifically makes it possible to control the points at which a switch occurs making this safe (even if localcontext() wasn't thread-safe): with decimal.localcontext() as ctx: ctx.prec = 100 c = a + b # more synchronous decimal calculations # State is restored before allowing other code to execute yield from saveindatabase(c) So it's fine to use global/thread-local state modifying/restoring context managers in Tulip as long as you don't yield control to other code within the with block. (unless I misunderstand - I lost track of Tulip some time ago).

You've got it exactly right.

The issue with decimal.localcontext() and yield arises when using generators as much as coroutines e.g.:

def exactsum(nums): start = Decimal(0) with decimal.localcontext() as ctx: ctx.traps[decimal.Inexact] = True for num in nums: try: total += Decimal(num) except decimal.Inexact: ctx.prec *= 2 return total The above is fine for computing the sum of a list of Decimals/ints/floats. However it fails if you pass in a generator that carelessly modifies the arithmetic context around yield calls: def compute(): with decimal.localcontext() as ctx: ctx.prec = 15 ctx.traps[decimal.Inexact] = False yield a + b yield b - c # etc. exactsum(compute()) There needs to be a convention that either functions like exactsum() mustn't assume continuity of the context between iterations or a function like compute() must restore before yielding. IMO the only sane approach for async coroutines is to say that if you yield or yield from then it is your responsibility to restore any temporarily altered global/thread-local state first.

Right again. The simplest rule to remember seems to be "don't use yield or yield-from inside a with-statement". You can relax it by limiting it to context managers that manage any kind of shared resource, but that is probably already too subtle: e.g. yielding inside "with open(file) as f" seems fine, but yielding inside "with lock" is problematic, since the other side might try to acquire the same lock, and deadlock.

-- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20131017/015918d2/attachment.html>

More information about the Python-Dev mailing list