[Python-ideas] Possible PEP 380 tweak (original) (raw)

Guido van Rossum guido at python.org
Sat Oct 30 05:47:04 CEST 2010

On Fri, Oct 29, 2010 at 8:07 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

On Sat, Oct 30, 2010 at 11:15 AM, Guido van Rossum <guido at python.org> wrote:

BTW I don't think I like piggybacking a return value on GeneratorExit. Before you know it people will be writing except blocks catching GeneratorExit intending to catch one coming from inside but accidentally including a yield in the try block and catching one coming from the outside. The nice thing about how GeneratorExit works today is that you needn't worry about it coming from inside, since it always comes from the outside first. This means that if you catch a GeneratorExit, it is either one you threw into a generator yourself (it just bounced back, meaning the generator didn't handle it at all), or one that was thrown into you. But the pattern of catching GeneratorExit and responding by returning a value is a reasonable extension of the pattern of catching GeneratorExit and doing other cleanup. (TLDR version: I'm -1 on Guido's modified close() semantics if there is no way to get the result out of a yield from expression that is terminated by GeneratorExit, but I'm +1 if we tweak PEP 380 to make the result available on the reraised GeneratorExit instance, thus allowing framework authors to develop ways to correctly unwind a generator stack in response to close()) Stepping back a bit, let's look at the ways a framework may "close" a generator-based operation (or substep of a generator). 1. Send in a sentinel value (often None, but you could easily reuse the exception types as sentinel values  as well) 2. Throw in GeneratorExit explicitly 3. Throw in StopIteration explicitly

Throwing in StopIteration seems more unnatural than any other option.

4. Throw in a different specific exception 5. Call g.close()

Having close() return a value only helps with the last option, and only if the coroutine is set up to work that way. Yield from also isn't innately set up to unwind correctly in any of these cases, without some form of framework based signalling from the inner generator to indicate whether or not the outer generator should continue or bail out.

Yeah, there is definitely some kind of convention needed here. A framework or app can always choose not to use g.close() for this purpose (heck, several current frameworks use yield to return a value) and in some cases that's just the right thing. Just like in other flow control situations you can often choose between sentinel values, exceptions, or something else (e.g. flag variables that must be explicitly tested).

Now, if close() were set up to return a value, then that second point makes the idea less useful than it appears. To go back to the simple summing example (not my too-complicated-for-a-mailing-list-discussion version which I'm not going to try to rehabilitate):

def gtally():  count = tally = 0  try:  while 1:  tally += yield  count += 1  except GeneratorExit:  pass  return count, tally

I like this example.

Fairly straightforward, but one of the promises of PEP 380 is that it allows us to factor out some or all of a generator's internal logic without affecting the externally visible semantics. So, let's do that:

 def gtally2():  return (yield from gtally())

And I find this a good starting point.

Unless the PEP 380 yield from expansion is changed, Guido's proposed "close() returns the value on StopIteration" just broke this equivalence for gtally2() - since the yield from expansion turns the StopIteration back into a GeneratorExit, the return value of gtally2.close is always going to be None instead of the expected (count, tally) 2-tuple. Since the value of the internal call to close() is thrown away completely, there is absolute nothing the author of gtally2() can do to fix it (aside from not using yield from at all).

Right, they could do something based on the (imperfect) equivalency between "yield from f()" and "for x in f(): yield x".

To me, if Guido's idea is adopted, this outcome is as illogical and unacceptable as the following returning None:

 def sum2(seq):  return sum(seq)

Maybe.

We already thrashed out long ago that the yield from handling of GeneratorExit needs to work the way it does in order to serve its primary purpose of releasing resources, so allowing the inner StopIteration to propagate with the exception value attached is not an option.

The question is whether or not there is a way to implement the return-value-from-close() idiom in a way that doesn't completely break the equivalence between gtally() and gtally2() above. I think there is: store the prospective return-value on the GeneratorExit instance and have the yield from expansion provide the most recent return value as it unwinds the stack. To avoid giving false impressions as to which level of the stack return values are from, gtally2() would need to be implemented a bit differently in order to also convert GeneratorExit to StopIteration:  def gtally2():  # The PEP 380 equivalent of a "tail call" if g.close() returns a value  try:  yield from gtally()  except GeneratorExit as ex:  return ex.value

Unfortunately this misses the goal of equivalency between gtally() and your original gtally2() by a mile. Having to add extra except clauses around each yield-from IMO defeats the purpose.

Specific proposed additions/modifications to PEP 380:

1. The new "value" attribute is added to GeneratorExit as well as StopIteration and is explicitly read/write

I already posted an argument against this.

2. The semantics of the generator close method are modified to be:

 def close(self):  try:  self.throw(GeneratorExit)  except StopIteration as ex:  return ex.value  except GeneratorExit:  return None # Ignore the value, as it didn't come from the outermost generator  raise RuntimeError("Generator ignored GeneratorExit") 3.  The GeneratorExit handling semantics for the yield from expansion are modified to be:  except GeneratorExit as e:  try:  m = i.close  except AttributeError:  pass  else:  e.value = m() # Store close() result on the exception  raise e With these modifications, a framework could then quite easily provide a context manager to make the idiom a little more readable and hide the fact that GeneratorExit is being caught at all: class GenResult():  def init(self): self.value = None @contextmanager def generatorreturn():  result = GenResult()  try:  yield  except GeneratorExit as ex:  result.value = ex.value def gtally():  # The CM suppresses GeneratorExit, allowing us  # to convert it to StopIteration  count = tally = 0  with generatorreturn():  while 1:  tally += yield  count += 1  return count, tally def gtally2():  # The CM also collects the value of any inner  # yield from expression, allowing easier tail calls  with generatorreturn() as result:  yield from gtally()  return result.value

I agree that you've poked a hole in my proposal. If we can change the expansion of yield-from to restore the equivalency between gtally() and the simplest gtally2(), thereby restoring the original refactoring principle, we might be able to save it. Otherwise I declare defeat. Right now I am too tired to think of such an expansion, but I recall trying my hand at one a few nights ago and realizing that I'd introduced another problem. So this does not look too hopeful, especially since I really don't like extending GeneratorExit for the purpose.

-- --Guido van Rossum (python.org/~guido)

More information about the Python-ideas mailing list