Issue 18677: Enhanced context managers with ContextManagerExit and None (original) (raw)
Created on 2013-08-07 14:22 by kristjan.jonsson, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (17)
Author: Kristján Valur Jónsson (kristjan.jonsson) * 
Date: 2013-08-07 14:22
A proposed patch adds two features to context managers:
1)It has always irked me that it was impossible to assemble nested context managers in the python language. See issue #5251. The main problem, that exceptions in enter cannot be properly handled, is fixed by introducing a new core exception, ContextManagerExit. When raised by enter(), the body that the context manager protects is skipped. This exception is in the spirit of other semi-internal exceptions such as GeneratorExit and StopIteration. Using this exception, contextlib.nested can properly handle the case where the body isn't run because of an internal enter exception which is handled by an outer exit.
- The mechanism used in implementing ContextManagerExit above is easily extended to allowing a special context manager: None. This is useful for having optional context managers. E.g. code like this: with performance_timer(): do_work() def performance_timer(): if profiling: return accumulator return None
None becomes the trivial context manager and its enter and exit calls are skipped, along with their overhead.
This patch implements both features. In addition, it:
- reintroduces contextlib.nested, which is based on nested_delayed
- introduces contextlib.nested_delayed, which solves the other problem with previous versions of nested, that an inner context manager expression shouldn't be evaluated early. contextlib.nested evaluates callables returning context managers, rather than managers directly.
- Allows contextlib.contextmanager decorated functions to not yield, which amounts to skipping the protected body (implicitly raising ContextManagerExit)
- unittests for the whole thing.
I'll introduce this stuff on python-ideas as well.
Author: R. David Murray (r.david.murray) *
Date: 2013-08-07 14:44
Your use cases are either already addressed by contextlib.ExitStack, or should be addressed in the context of its existence. It is the replacement for contextlib.nested.
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2013-08-07 14:59
IMHO, exitstack is not a very nice construct. It's implementation is far longer than contextlib.nested.
And the chief problem still remains, which has not been addressed until this patch (as far as I know): In Python, it is impossible to combine existing context managers into a nested one. ExitStack may address a use case of nested context managers, but it doesn't address the basic problem.
ContextManagerExit comes with its own nice little features, too. Now you can write:
@contextlib.contextmanager: def if_ctxt(condition): if condition: yield
#hey look! an if statement as a with statement! with if_ctxt(condition): do_work
This can easily be extended, where a context manager can both manage context, and provide optional execution of its block.
Author: R. David Murray (r.david.murray) *
Date: 2013-08-07 15:04
Raising it on python-ideas sounds like a good idea, then.
I must admit that I don't understand what you mean by "combining existing context managers into a nested one" that isn't addressed by ExitStack.
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2013-08-07 16:29
Simply put, there is no way in the language to nest two context managers, even though we have full access to their implementation model, i.e. can call enter and exit manually. This reflects badly (pun intended) on Python's reflection and introspection capabilities.
If context managers are to be first class entities in the language, then you ought to be able to write absract code using them, and assemble complex ones out of simple ones. Hypothetical code here:
def nest(a, b): # currently not possible return c
def run_with_context(ctxt, callable): # abstract executor with ctxt: return callable()
run_with_context(nested(a,b), callable)
ExitStack address one use case that contextlib.nested was supposed to solve, namely the cleanup of a dynamic sequence of context managers. But it does this no by creating a new manager, but by providing a programming pattern to follow. In that sensse, the multiple context manager syntax (with (a, b, c): ) is also a hack because it provides language magic to perform what you ought to be able to do dynamically...
Does this makes sense?
Anyway, by providing the ContextManagerExit exception, then sufficient flexibility is added to the context manager mechanism that at least the use case of nested() becomes possible.
Context managers are really interesting things. I was inspired by Raymond Hettinger's talk last pycon to explore their capabilities and this is one of the things I came up with :)
Author: Alyssa Coghlan (ncoghlan) *
Date: 2013-08-07 23:36
I pitched the idea of making it possible to skip the with statement body quite some time ago, and Guido convinced me it was a bad idea for much the same reason he chose PEP 343 over his original PEP 340 design: allowing suppression of exceptions from enter hides local control flow by blurring the boundaries between with and if statements.
Regarding nested, we killed that because it was a bug magnet for context managers that acquire the resource in init (like file objects), not because it didn't work. It's trivial to recreate that API on top of ExitStack if you like it, though. The only thing that doesn't work (relative to actual nested with statements) is suppressing exceptions raised inside enter methods.
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2013-08-08 00:09
Hi there. "allowing suppression of exceptions from enter hides local control flow by blurring the boundaries between with and if statements. " I'm not sure what this means. To me, it is a serious language design flaw that you can write a context manager, and it has a well known interface that you can invoke manually, but that you cannot take two existing context managers and assemble them into a nested one, correctly, however much you wiggle. In my mind, allowing context managers to skip the managed body breaks new ground. Both, by allowing this "combination" to be possible. And also by opening up new and exciting applications for context managers. If you saw Raymond's talk last Pycon, you should feel inspired to do new and exciting things with them.
the bug-magnet you speak of I already addressed in my patch with nested-delayed, more as a measure of completeness (address both the problems that old "nested" had. The more serious bug (IMHO) is the suppression of enter exceptions.
Author: Eric Snow (eric.snow) *
Date: 2013-08-08 03:25
Nick was probably talking about what is further elaborated in PEP 343. I'd recommend taking a particular look at the "Motivation and Summary" section regarding flow control macros.
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2013-08-08 10:24
I've modified the patch. The problem that nested_delayed was trying to solve are "hybrid" context managers, ones that allocate resources during init and release them at exit. A proper context manager should allocate resources during enter, and thus a number of them can be created upfront with impunity.
Added contextlib.proper to turn a hybrid context manager into a proper one by instantiating the hybrid in a delayed fashion. added contextlib.opened() as a special case that does open() properly.
With this change, and the ability to nest error handling of exceptions stemming from enter(), nested now works as intended.
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2013-08-08 10:39
Thanks, Eric. I read that bit and I can't say that I disagree. And I'm not necessarily advocating that "skipping the body" become a standard feature of context managers. But it is a necessary functionality if you want to be able to dynamically nest one or more context managers, something I think Python should be able to do, for completeness, if not only for aesthetic beauty.
Having said that, optionally skipping the body is a far cry from the more esoteric constructs achievable with pep 340.
And python already silently skips the body of managed code, if you nest two managers:
@contextmanager errordude: 1 // 0 yield @contextmanager handler: try: yield except ZeroDivisionError: pass
with handler, errordude: do_stuff()
These context managers will skip the execution of f. It will be Python's internal decision to do so, of course. But the "with" statement already has the potential to have the body silently skipped.
What I'm adding here, the ContextManagerExit, is the ability for the context manager itself to make the decision, so that the two context managers above can be coalesced into one:
with nested(handler, errordude):
do_stuff()The fact that do_stuff can be silently skipped in the first case, where we explicitly have two nested calls, invalidates IMHO the argument that context managers should not affect control flow. why shouldn't it also be skippable in the case of a single context manager?
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2013-08-08 10:43
Using my latest patch, the ExitStack inline example can be rewritten:
with ExitStack() as stack: files = [stack.enter_context(open(fname)) for fname in filenames] # All opened files will automatically be closed at the end of # the with statement, even if attempts to open files later # in the list raise an exception
becomes: with nested(opened(fname) for fname in filenames) as files: do_stuff_with_files(files)
Author: Alyssa Coghlan (ncoghlan) *
Date: 2013-08-08 15:33
Allowing a context manager to skip the statement body isn't a new proposal, and I previously argued your side. However, with multiple context managers, there is no invisible flow control. Two context managers are locally visible, which means the outer one completely encloses the inner one and can suppress exceptions it throws. Guido explicitly made the decision to require two managers at the point of use to achieve that behaviour when I proposed making the change - he doesn't care about allowing a single context manager to provide that functionality.
For the other question, how does your version of nested keep people from doing "nested(open(fname) for name in names)"? That was the core problem with that style of API: it made it far too easy to introduce a latent defect when combined with file like objects that eagerly acquire their resource. It wasn't that it couldn't be used correctly, but that the natural and obvious way of combining it with open() is silently wrong.
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2013-08-14 20:08
"locally visible" is, I think a very misleading term. How is
with ignore_error, acquire_resource as r: doo_stuff_with_resource(r) #can be silently skipped
any more locally visible than with acquire_resource_ignore_error as r: doo_stuff_with resource(r) # can be silently skipped.
? does the "nested with" syntax immediatelly tell you "hey, the body can be silently skipped"?
Requiring that some context manager patterns must be done with a special syntax is odd. What is more, it prohibits us to abstract away context managers. For instance, you can write a function like this
def execute_with_context(ctxt, fn, args): with ctxt: return fn(*args)
but if your context manager is of the kind mentioned, i.e. requiring the double syntax, you are screwed.
Basically, what I'm proposing (and what the patch provides) is that you can write this code: @contextmanager def nestedc(ca, cb): with ca as a, cb as b: yield a, b
and have it work for all pair of ca, cb. This then, allows context managers to be used like abstract entities, like other objects in the language. It is not about flow control, but about completeness.
A similar pattern for functions is already possible: def nestedf(fa, fb): def helper(v): return fa(fb(v)) return helper
And so, we could write: execute_with_context(nestedc(ca, cb), nestedf(fa, fb), ('foo',))
Current python does not allow this for arbitrary pairs ca, cb. My version does. This is what I'm advocating. That programmers are given the tool to combine context managers if they want.
As for "contextlib.nested()". I'm not necessarily advocation its resuciation in the standardlib, but adding that to the patch here to demonstrate how it now works.
Here is a simpler version of contextlib.nested:
@contextmanager def nested_empty(): yield []
@contextmanager def nested_append(prev, next): with prev as a, next as b: a.append(b) yield a
def nested(*managers): total = nested_empty() for mgr in managers: total = nested_append(total, mgr) return total
Pretty nice, no?
Now we come to the argument with nested(open(a), open(b)). I see your point, but I think that the problem is not due to nested, but to open. Deprecating nested, even as a programming pattern demonstration is throwing out the baby with the bathwater.
I´ve coined the term "hybrid context manager" (at least I think I have)to mean resources that are their own context managers. They're hybrid because they are acquired explicitly, but can be released via a context manager. The context manager is a bolt on, an afterthought. Instead of adding exit() to files, and allowing with open(fn) as f: pass We should have encouraged the use of proper context managers: with opened(fn) as f: pass or with closing(f): pass
Now, we unfortunately have files being context managers and widely see the pattern with open(fn) as f, open(fn2) as f2: pass
But how is this bug here: with nested(open(fn), open(fn2)) as f, f2: pass
any more devuiys than f, f2 = open(fn), open(fn2) with f, f2: pass ?
The problem is that files aren't "real" context managers but "hybrids" and this is what we should warn people about. The fact that we do have those hybrids in our code base should not be cause to remove tools that are designed to work with "proper" context managers.
The decision to remove "nested" on these grounds sets the precedence that we cannot have any functions that operate on context managers. In fact, what this is really is saying is this:
"context managers should only be used with the "with" statement and only instantiated in-line. Anything else may introduce sublte bugs because some context managers are in fact not context managers, but the resource that they manage. "
In my opinion, it would have been better to deprecate the use of files as context managers, and instead urge people to use proper context managers for the: (the proposed) contextlib.opened and (the existing) contextlib.closing)
K
Author: Alyssa Coghlan (ncoghlan) *
Date: 2013-08-14 21:48
I think you make a good case, but I already tried and failed to convince Guido of this in PEP 377 (see http://www.python.org/dev/peps/pep-0377/#rationale-for-change)
More importantly, see his quoted concerns in http://mail.python.org/pipermail/python-dev/2009-March/087263.html
While you have come up with a much simpler implementation for PEP 377, which imposes no additional overhead in the typical case (unlike my implementation, which predated the SETUP_WITH opcode and avoided introducing one, which required wrapping every enter call in a separate try/except block), it still adds a new builtin exception type, and I thing needs a new builtin constant as well.
The latter comes in because I think the bound variable name still needs to be set to something, and rather than abusing any existing constant, I think a new SkipWith constant for both "don't call enter/exit" and "with statement body was skipped" would actually be clearer.
I actually think explaining a custom exception and constant is less of a burden than explaining why factoring out certain constructs with @contextmanager and yield doesn't work properly (that's why I wrote PEP 377 in the first place), but Guido is the one that ultimately needs to be convinced of the gain.
Author: Alex Waygood (AlexWaygood) * 
Date: 2021-12-27 10:41
Given that this issue has seen no activity for eight years, I am closing it as "rejected".
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2022-01-03 10:35
Great throwback.
As far as I know, context managers are still not first class citizens. You cannot compose two context managers into a new one programmatically in the language, in the same way that you can, for instance, compose two functions. Not even using "eval()" is this possible.
This means that the choice of context manager, or context managers, to be used, has to be known when writing the program. You cannot pass an assembled context manager in as an argument, or otherwise use a "dynamic" context manager at run time, unless you decide to use only a fixed number of nested ones. any composition of context managers becomes syntax at the point of invocation.
The restriction is similar to not allowing composition of functions, i.e. having to write
fa(fb(fc())) at the point of invocation and not have the capability of doing
def fd():
return fa(fb(fc))
...
fd()I think my "ContextManagerExit" exception provided an elegant solution to the problem and opened up new and exciting possibilities for context managers and how to use them.
But this here note is just a lament. I've stopped contributing to core python years ago, because it became more of an excercise in lobbying than anything else. Cheers!
Author: Kristján Valur Jónsson (kristjan.jonsson) *
Date: 2022-01-03 14:52
Having given this some thougt, years laters, I believe it is possible to write nested() (and nested_delayed()) in a correct way in python, without the ContextManagerExit function.
Behold!
import contextlib
@contextlib.contextmanager def nested_delayed(*callables): """ Instantiate and invoke context managers in a nested way. each argument is a callable which returns an instantiated context manager """ if len(callables) > 1: with nested_delayed(*callables[:-1]) as a, callables-1 as b: yield a + (b,) elif len(callables) == 1: with callables0 as a: yield (a,) else: yield ()
def nested(managers): """ Invoke preinstantiated context managers in a nested way """ def helper(m): """ A helper that returns the preinstantiated context manager when invoked """ def callable(): return m return callable return nested_delayed((helper(m) for m in managers))
@contextlib.contextmanager def ca(): print("a") yield 1
class cb: def init(self): print ("instantiating b") def enter(self): print ("b") return 2 def exit(*args): pass
@contextlib.contextmanager def cc(): print("c") yield 3
combo = nested(ca(), cb(), cc()) combo2 = nested_delayed(ca, cb, cc)
with combo as a: print("nested", a)
with combo2 as a: print("nested_delayed", a)
with ca() as a, cb() as b, cc() as c: print ("syntax", (a, b, c))
History
Date
User
Action
Args
2022-04-11 14:57:49
admin
set
github: 62877
2022-01-03 14:52:22
kristjan.jonsson
set
messages: +
2022-01-03 10:35:01
kristjan.jonsson
set
messages: +
2021-12-27 10:41:33
AlexWaygood
set
status: open -> closed
nosy: + AlexWaygood
messages: +
resolution: rejected
stage: resolved
2013-08-14 21:48:32
ncoghlan
set
nosy: + gvanrossum
messages: +
2013-08-14 20:08:26
kristjan.jonsson
set
messages: +
2013-08-12 20:15:50
barry
set
nosy: + barry
2013-08-08 15:33:28
ncoghlan
set
messages: +
2013-08-08 10:43:07
kristjan.jonsson
set
messages: +
2013-08-08 10:39:42
kristjan.jonsson
set
messages: +
2013-08-08 10:24:37
kristjan.jonsson
set
files: + contextmanagerexit.patch
messages: +
2013-08-08 03:25:30
eric.snow
set
nosy: + eric.snow
messages: +
2013-08-08 00:09:59
kristjan.jonsson
set
messages: +
2013-08-07 23:36:54
ncoghlan
set
messages: +
2013-08-07 16:29:21
kristjan.jonsson
set
messages: +
2013-08-07 15:04:51
r.david.murray
set
messages: +
2013-08-07 14:59:58
kristjan.jonsson
set
messages: +
2013-08-07 14:44:39
r.david.murray
set
nosy: + r.david.murray, ncoghlan
messages: +
2013-08-07 14:22:07
kristjan.jonsson
create