[Python-Dev] Tricky way of of creating a generator via a comprehension expression (original) (raw)

Guido van Rossum guido at python.org
Sat Nov 25 10:57:37 EST 2017

On Sat, Nov 25, 2017 at 6:55 AM, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:

On 25 November 2017 at 04:30, Guido van Rossum <guido at python.org> wrote:

On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum <guido at python.org> wrote:

The more I hear about this topic, the more I think that await, yield and yield from should all be banned from occurring in all comprehensions and generator expressions. That's not much different from disallowing return or break.

From the responses it seems that I tried to simplify things too far. Let's say that await in comprehensions is fine, as long as that comprehension is contained in an async def. While we could save yield_ _[from] in comprehensions, I still see it as mostly a source of confusion, and the fact that the presence of yield [from] implicitly makes the surrounding def a generator makes things worse. It just requires too many mental contortions to figure out what it does. There were some arguments that await is like a function call, while yield is like return. TBH, I don't really like these arguments since to me they are to vague. Continuing this logic one can say that return is just a fancy function call (calling continuation with the result). To me there is one clear distinction: return and break are statements, while yield, yield from, and await are expressions.

Indeed. However, yield from as an expression is mostly a deprecated way to write await, and yield as an expression is mostly an alternative way of writing coroutines (it's the only way that exists in Python 2). Another big difference is that the use of yield [from] affects the surrounding function, making it a generator.

Continuing the topic of the ban, what exactly should be banned? For example

will this still be valid?

def packtwo(): return [(yield), (yield)] # Just a list display

It's not a comprehension so it's still valid.

I don't see how this is controversial. It is clear that packtwo is a generator. If this is going to be prohibited, then one may be surprised by lack of referential transparency, since this will be valid:

def packtwo(): first = (yield) second = (yield) return [first, second] If the first example will be allowed, then one will be surprised why it can't be rewritten as def packtwo(): return [(yield) for in range(2)]

And yet Nick's example shows that that is not equivalent!

def example(): comp1 = yield from [(yield x) for x in ('1st', '2nd')] comp2 = yield from [(yield x) for x in ('3rd', '4th')] return comp1, comp2

In this example each thing that looks syntactically like a list comprehension becomes actually a generator expression at at runtime! And so does your example, so instead of a list of two items, it returns a generator that will produce two values when iterated over.

That's not referential transparency to me, it feels more like a bug in the code generator.

I want to ban this because apparently nobody besides Nick knows about this behavior (I certainly didn't, and from the above it seems you don't either).

I have found several other examples where it is not clear whether they should be prohibited with yield or not.

Such as?

I still propose to rule out all of the above from generator expressions,

because those can escape from the surrounding scope.

Here I agree. Also note that the above problem does not apply to generator expressions since (x, x) and (x for in range(2)) are two very different expressions.

PS. A more radical proposal (not for 3.7) would be to deprecate yield as an expression. It once was only a statement, but PEP 342 introduced yield as an expression in order to do coroutines. We now have async def and await as a superior coroutine mechanism. But we must continue to support yield expressions because there is a lot of Python 2/3 compatible code that depends on it. (E.g. Tornado.)

-- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20171125/26118b52/attachment-0001.html>

More information about the Python-Dev mailing list