[Python-Dev] Tricky way of of creating a generator via a comprehension expression (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Sat Nov 25 00:04:40 EST 2017

On 25 November 2017 at 13:30, Guido van Rossum <guido at python.org> wrote:

On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum <guido at python.org> wrote:

The more I hear about this topic, the more I think that await, yield and yield from should all be banned from occurring in all comprehensions and generator expressions. That's not much different from disallowing return or break. From the responses it seems that I tried to simplify things too far. Let's say that await in comprehensions is fine, as long as that comprehension is contained in an async def. While we could save yield [from] in comprehensions, I still see it as mostly a source of confusion, and the fact that the presence of yield [from] implicitly makes the surrounding def a generator makes things worse. It just requires too many mental contortions to figure out what it does.

While I'm OK with ruling the interaction of all of these subexpressions with generator expression too confusingly weird to be supportable, in https://bugs.python.org/issue10544?#msg306940 I came up with an example for the existing comprehension behaviour that I'm hesitant to break without a clear replacement: code that wraps the current comprehension behaviour in an explicit "yield from" expression.

That is, code like:

def example():
    comp1 = yield from [(yield x) for x in ('1st', '2nd')]
    comp2 = yield from [(yield x) for x in ('3rd', '4th')]
    return comp1, comp2

Creates a generator that returns a 2-tuple containing two 2-item lists, but the exact contents of those lists depend on the values you send into the generator while it is running.

The basic principle behind the construct is that a comprehension containing a yield expression essentially defines an inline subgenerator, so you then have to yield from that subgenerator in order to produce the desired list.

If the implicit "yield from" idea seems too magical, then the other direction we could go is to make the immediate "yield from" mandatory, such that leaving it out produces a SyntaxWarning in 3.7, and then a SyntaxError in 3.8+. (Something like "'yield from' clause missing before subgenerator comprehension")

While we don't generally like bolting two pieces of syntax together like that, I think it's defensible in this case based on the fact that allowing "[expr for var in iterable]" to ever return anything other than a list (and similarly for the other comprehension types) is genuinely confusing.

The nicest aspect of the "an explicit and immediate 'yield from' is required when using subgenerator comprehensions" approach is that the above syntax already works today, and has actually worked since 3.3 - we'd just never put the pieces together properly to establish this as a potential pattern for comprehension use in coroutines.

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list