[Python-Dev] Tricky way of of creating a generator via a comprehension expression (original) (raw)

Paul Moore p.f.moore at gmail.com
Wed Nov 22 09:47:06 EST 2017

On 22 November 2017 at 14:15, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:

On 22 November 2017 at 15:09, Paul Moore <p.f.moore at gmail.com> wrote:

On 22 November 2017 at 13:53, Ivan Levkivskyi <levkivskyi at gmail.com> wrote: > On 22 November 2017 at 14:38, Antoine Pitrou <solipsis at pitrou.net> > wrote: >> >> On Wed, 22 Nov 2017 15:03:09 +0200 >> Serhiy Storchaka <storchaka at gmail.com> wrote: >> > From >> > >> > >> > https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions. >> > >> > g = [(yield i) for i in range(3)] >> > >> > Syntactically this looks like a list comprehension, and g should be a >> > list, right? But actually it is a generator. This code is equivalent >> > to >> > the following code: >> > >> > def makelist(it): >> > result = [] >> > for i in it: >> > result.append(yield i) >> > return result >> > g = makelist(iter(range(3))) >> > >> > Due to "yield" in the expression makelist() is not a function >> > returning a list, but a generator function returning a generator. >> > >> > This change in semantic looks unintentional to me. It looks like >> > leaking >> > an implementation detail. >> >> Perhaps we can deprecate the use of "yield" in comprehensions and make >> it a syntax error in a couple versions? >> >> I don't see a reason for writing such code rather than the more >> explicit variants. It looks really obscure, regardless of the actual >> semantics. > > > People actually try this (probably simply because they like > comprehensions) > see two mentioned Stackoverflow questions, plus there are two b.p.o. > issues. > So this will be a breaking change. Second, recent PEP 530 allowed > writing a > similar comprehensions with await: > > async def process(funcs): > result = [await fun() for fun in funcs] # OK > ... > > Moreover, it has the semantics very similar to the proposed by Serhiy > for > yield (i.e. equivalent to for-loop without name leaking into outer > scope). > Taking into account that the actual fix is not so hard, I don't think it > makes sense to have all the hassles of deprecation period. I agree with Antoine. This (yield in a comprehension) seems far too tricky, and I'd rather it just be rejected. If I saw it in a code review, I'd certainly insist it be rewritten as an explicit loop. There are many things that I would reject in code review, but they are still allowed in Python, this is one of the reasons why code reviews exist. Also I am not sure how yield in a comprehension is more tricky than await in a comprehension. Anyway, this looks more like a taste question.

I generally don't understand "await" in any context, so I deferred judgement on that :-) Based on your comment that they are equally tricky, I'd suggest we prohibit them both ;-)

Less facetiously, comprehensions are defined in the language reference in terms of a source translation to nested loops. That description isn't 100% precise, but nevertheless, if yield/async in a comprehension doesn't behave like that, I'd consider it a bug. So current behaviour (for both yield and await) is a bug, and your proposed semantics for yield is correct. Await in a comprehension should work similarly - the docs for await expressions just say "Can only be used inside a coroutine function", so based on that they are legal within a comprehension inside a coroutine function (if the semantics in that case isn't obvious, it should be clarified - the await expression docs are terse to the point that I can't tell myself).

So I'm +1 on your suggestion.

I remain concerned that yield/await expressions are adding a lot of complexity to the language, that isn't explained in the manuals in a way that's accessible to non-specialists. Maybe "ban (or reject at code review) all the complex stuff" isn't a reasonable approach, but nor is expecting everyone encountering async code to have read and understood all the async PEPs and docs. We need to consider maintenance programmers as well (who are often looking at code with only a relatively high-level understanding).

Paul

More information about the Python-Dev mailing list