[Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from] (original) (raw)

Guido van Rossum guido at python.org
Sat Apr 4 22:29:00 CEST 2009

[Answering somewhat out of order; new proposal developed at the end.]

On Fri, Apr 3, 2009 at 4:25 PM, Jacob Holm <jh at improva.dk> wrote:

I am still trying to get a clear picture of what kind of mistakes you are trying to protect against. If it is people accidently writing return in a generator when they really mean yield, that is what I thought the proposal for an alternate syntax was for. That sounds like a good idea to me, especially if we could also ban or discourage the use of normal return. But the alternate syntax doesn't have to mean a different exception.

I am leaning the other way now. New syntax for returning in a value is a high-cost proposition. Instead, I think we can guard against most of the same mistakes (mixing yield and return in a generator used as an iterator) by using a different exception to pass the value. This would delay the failure to runtime, but it would still fail loudly, which is good enough for me.

I want to name the new exception ReturnFromGenerator to minimize the similarity with GeneratorExit: if we had both GeneratorExit and GeneratorReturn there would be endless questions on the newbie forums about the differences between the two, and people might use the wrong one. Since ReturnFromGenerator goes out of the generator and GeneratorExit goes in, there really are no useful parallels, and similar names would cause confusion.

I could easily see "return value" as a separate PEP, except PEP 380 provides one of the better reasons for its inclusion. It might be good to figure out how this feature should work by itself before complicating things by integrating it in the yield-from semantics.

Here are my curent thoughts on this. When a generator returns, the return statement is treated normally (whether or not it has a value) until the frame is about to be left (i.e. after any finally-clauses have run). Then, it is converted to StopIteration if there was no value or ReturnFromGenerator if there was a value. I don't care which one is picked for an explicit "return None" -- that should be decided by implementation expediency. (E.g. if one requires adding new opcodes and one doesn't, I'd pick the one that doesn't.)

Normal loops (for-loops, list comprehensions, other implied loops) only catch StopIteration, so that returning a value is still wrong here. But some other contexts treat ReturnFromGenerator similar as StopIteration except the latter conveys None and the former conveys an explicit value. This applies to yield-from as well as to explicit or implied closing of the generator (close() or deallocation).

So g.close() returns the value (I think I earlier said I didn't like that -- I turned around on this one). It's pseudo-code is roughly:

def close(it): try: it.throw(GeneratorExit) except (GeneratorExit, StopIteration): return None except ReturnFromGenerator as e: # This block is really the only new thing return e.value

Other exceptions are passed out unchanged

else: # throw() yielded a value -- unchanged raise RuntimeError(.....)

Deleting a generator is like closing and printing (!) a traceback (to stderr) if close() raises an exception. A returned value it is just ignored. Explicit pseudo-code without falling back to close():

def del(it): try: it.throw(GeneratorExit) except (GeneratorExit, StopIteration, ReturnFromGenerator): pass except:

Some other exception happened

<print traceback>

else: # throw() yielded another value

I have also worked out what I want yield-from to do, see end of this message.

[Guido]

Oh, and "yield from" competes with @couroutine over when the initial next() call is made, which again suggests the two styles (yield-from and coroutines) are incompatible. It is a serious problem, because one of the major points of the PEP is that it should be useful for refactoring coroutines. As a matter of fact, I started another thread on this specific issue earlier today which only Nick has so far responded to. I think it is solvable, but requires some more work.

I think that's the thread where I asked you and Nick to stop making more proposals.I a worried that a solution would become too complex, and I want to keep the "naive" interpretation of "yield from EXPR" to be as close as possible to "for x in EXPR: yield x". I think the @coroutine generator (whether built-in or not) or explicit "priming" by a next() call is fine.

So now let me develop my full thoughts on yield-from. This is unfortunately long, because I want to show some intermediate stages. I am using a green font for new code. I am using stages, where each stage provides a better approximation of the desired semantics. Note that each stage adds some semantics for corner cases that weren't handled the same way in the previous stage. Each stage proposes an expansion for "RETVAL = yield from EXPR". I am using Py3k syntax.

  1. Stage one uses the for-loop equivalence:

for x in EXPR: yield x RETVAL = None

  1. Stage two expands the for-loop into an explicit while-loop that has the same meaning. It also sets RETVAL when breaking out of the loop. This prepares for the subsequent stages. Note that we have an explicit iter(EXPR) call here, since that is what a for-loop does:

it = iter(EXPR) while True: try: x = next(it) except StopIteration: RETVAL = None; break yield x

  1. Stage three further rearranges stage 2 without making semantic changes, Again this prepares for later stages:

it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = e.value else: while True: yield x try: x = next(x) except StopIteration: RETVAL = None; break

  1. Stage four adds handling for ReturnFromGenerator, in both places where next() is called:

it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = e.value except ReturnFromGenerator as e: RETVAL = e.value; break else: while True: yield x try: x = next(it) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break yield x

  1. Stage five shows what should happen if "yield x" above returns a value: it is passed into the subgenerator using send(). I am ignoring for now what happens if it is not a generator; this will be cleared up later. Note that the initial next() call does not change into a send() call, because there is no value to send before before we have yielded:

it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = None except ReturnFromGenerator as e: RETVAL = e.value else: while True: v = yield x try: x = it.send(v) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break

  1. Stage six adds more refined semantics for when "yield x" raises an exception: it is thrown into the generator, except if it is GeneratorExit, in which case we close() the generator and re-raise it (in this case the loop cannot continue so we do not set RETVAL):

it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = None except ReturnFromGenerator as e: RETVAL = e.value else: while True: try: v = yield x except GeneratorExit: it.close() raise except: try: x = it.throw(*sys.exc_info()) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break else: try: x = it.send(v) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break

  1. In stage 7 we finally ask ourselves what should happen if it is not a generator (but some other iterator). The best answer seems subtle: send() should degenerator to next(), and all exceptions should simply be re-raised. We can conceptually specify this by simply re-using the for-loop expansion:

it = iter(EXPR) if : for x in it: yield next(x) RETVAL = None else: try: x = next(it) except StopIteration: RETVAL = None except ReturnFromGenerator as e: RETVAL = e.value else: while True: try: v = yield x except GeneratorExit: it.close() raise except: try: x = it.throw(*sys.exc_info()) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break else: try: x = it.send(v) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break

Note: I don't mean that we literally should have a separate code path for non-generators. But writing it this way adds the generator test to one place in the spec, which helps understanding why I am choosing these semantics. The entire code of stage 6 degenerates to stage 1 if we make the following substitutions:

it.send(v) -> next(v) it.throw(sys.exc_info()) -> raise it.close() -> pass

(Except for some edge cases if the incoming exception is StopIteration or ReturnFromgenerator, so we'd have to do the test before entering the try/except block around the throw() or send() call.)

We could do this based on the presence or absence of the send/throw/close attributes: this would be duck typing. Or we could use isinstance(it, types.GeneratorType). I'm not sure there are strong arguments for either interpretation. The type check might be a little faster. We could even check for an exact type, since GeneratorType is final. Perhaps the most important consideration is that if EXPR produces a file stream object (which has a close() method), it would not consistently be closed: it would be closed if the outer generator was closed before reaching the end, but not if the loop was allowed to run until the end of the file. So I'm leaning towards only making the generator-specific method calls if it is really a generator.

-- --Guido van Rossum (home page: http://www.python.org/~guido/<http://www.python.org/%7Eguido/> ) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-ideas/attachments/20090404/0dad23ab/attachment.html>

More information about the Python-ideas mailing list