[Python-Dev] PEP 380 - return value question and prototype implementation (was Thoughts fresh after EuroPython) (original) (raw)

P.J. Eby pje at telecommunity.com
Sun Jul 25 03:51:37 CEST 2010

At 07:08 AM 7/24/2010 -0700, Guido van Rossum wrote:

- After seeing Raymond's talk about monocle (search for it on PyPI) I am getting excited again about PEP 380 (yield from, return values from generators). Having read the PEP on the plane back home I didn't see anything wrong with it, so it could just be accepted in its current form.

I would like to reiterate (no pun intended) the suggestion of a special syntactic form for the return, such as "yield return x", or "return with x" or something similar, to distinguish it from a normal generator return.

I think that when people are getting used to the idea of generators, it's important for them to get the idea that the function's "return value" isn't really a value, it's an iterator object. Allowing a return value, but then having that value silently disappear, seems like it would delay that learning, so, a special form might help to make it clear that the generator in question is intended for use with a corresponding "yield from", and help avoid confusion on this.

(I could of course be wrong, and would defer to anyone who sees a better way to explain/teach around this issue. In any event, I'm +1 on the PEP otherwise.)

By the way, the PEP's "optimized" implementation could probably be done just by making generator functions containing yield-from statements return an object of a different type than the standard geniter. Here's a Python implementation sketch, using a helper class and a decorator -- translation to a C version is likely straightforward, as it'll basically be this plus a light sprinkling of syntactic sugar.

So, in the pure-Python prototype (without syntax sugaring), usage would look like this:

 @From.container
 def some_generator(...):
     ...
     yield From(other_generator(...))  # equivalent to 'yield from'
     ...

 def other_generator(...):
     ...
     raise StopIteration(value)  # equivalent to 'return value'

We mark some_generator() with @From.container to indicate that it uses 'yield from' internally (which would happen automatically in the C/syntax sugar version). We don't mark other_generator(), though, because it doesn't contain a 'yield from'.

Now, the implementation code (a slightly altered/watered-down version of a trampoline I've used before in 2.x, hopefully altered correctly for Python 3.x syntax/semantics):

class From: @classmethod def container(cls, func): def decorated(*args, **kw): return cls(func(*args, **kw)) # wrap generator in a From() instance return decorated

 def __new__(cls, geniter):
     if isinstance(geniter, cls):
         # It's already a 'From' instance, just return it
         return geniter
     self = object.__new__(cls, geniter)
     self.stack = [geniter]
     return self

 def __iter__(self):
     return self

 def __next__(self):
     return self._step()

 def send(self, value):
     return self._step(value)

 def throw(self, *exc_info):
     return self._step(None, exc_info)

 def _step(self, value=None, exc_info=()):
     if not self.stack:
         raise RuntimeError("Can't resume completed generator")
     try:
         while self.stack:
             try:
                 it = self.stack[-1]
                 if exc_info:
                     try:
                         rv = it.throw(*exc_info)
                     finally:
                         exc_info = ()
                 elif value is not None:
                     rv = it.send(value)
                 else:
                     rv = it.next()
             except:
                 value = None
                 exc_info = sys.exc_info()
                 if exc_info[0] is StopIteration:
                     # pass return value up the stack
                     value, = exc_info[1].args or (None,)
                     exc_info = ()   # but not the error
                 self.stack.pop()
             else:
                 if isinstance(rv, From):
                     stack.extend(rv.stack)  # Call subgenerator
                     value, exc_info, rv = None, (), None
                 else:
                     return rv   # it's a value to yield/return
         else:
             # Stack's empty, so exit w/current return value or error
             if exc_info:
                 raise exc_info[1]
             else:
                 return value
     finally:
         exc_info = ()   # don't let this create garbage

 def close(self):
     if self.stack:
         try:
             # There's probably a cleaner way to do this in Py 3, I just
             # don't know what it is off the top of my head...
             raise GeneratorExit
         except GeneratorExit as e:
             try:
                 self.throw(*sys.exc_info())
             except (StopIteration, GeneratorExit):
                 pass
             else:
                 raise RuntimeError("Generator(s) failed to close()")
                 # XXX probably needs some more code here to clean up stack

 def __del__(self):
     try:
         self.close()
     except:
         pass

As you can (hopefully) see, the main code path simply ends up delegating next/send/close etc. directly to the iterator on the top of the stack, so there isn't any multi-layer passthru going on, as in the "non-optimized" version of the spec in the PEP.

More information about the Python-Dev mailing list