[Python-Dev] Tricky way of of creating a generator via a comprehension expression (original) (raw)

Stephen J. Turnbull turnbull.stephen.fw at u.tsukuba.ac.jp
Thu Nov 23 00:33:58 EST 2017

• Previous message (by thread): [Python-Dev] Tricky way of of creating a generator via a comprehension expression
• Next message (by thread): [Python-Dev] Tricky way of of creating a generator via a comprehension expression
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Paul Moore writes:

1. List comprehensions expand into nested for/if statements in the "obvious" way - with an empty list created to start and append used to add items to it. 1a. Variables introduced in the comprehension don't "leak" (see below).
2. Generator expressions expand into generators with the same "nested loop" behaviour, and a yield of the generated value.
3. List comprehensions are the same as list(the equivalent generator expression).

I'm a little late to this discussion, but I don't see how 3 can be true if 1 and 2 are.

Because I believe my model of generator expressions is pretty simple, I'll present it here:

• the generator expression in 2 and 3 implicitly creates a generator function containing the usual loop
  • the point being that it will "capture" all of the yields (implicit and explicit) in the generator expression
• then implicitly invokes it,
• passing the (iterable) generator object returned to the containing expression.

"Look Ma, no yields (or generator functions) left!"

However, my model of comprehensions is exactly a for loop that appends to an empty list repeatedly, but doesn't leak iteration variables. So a yield in a comprehension turns the function containing the comprehension (which may or may not exist) into a generator function. In other words, I don't agree with Ethan that a yield inside a list comprehension should not affect the generator-ness of the containing function. What would this mean under that condition:

[f(x) for x in (yield iterable)]

then? IOW,

x = ((yield i) for i in iterable)

IS valid syntax at top level, while

x = [(yield i) for i in iterable]

IS NOT valid syntax at top level given those semantics. The latter "works" in Python 3.6;

>>> for i in [(yield i) for i in (1, 2, 3)]:
...  i
... 
1
2
3

though I think it should be a syntax error, and "bare" yield does not "work":

>>> i = 1
>>> yield i
  File "<stdin>", line 1
SyntaxError: 'yield' outside function
>>> (yield i)
  File "<stdin>", line 1
SyntaxError: 'yield' outside function

FWIW, that's the way I'd want it, and the way I've always understood comprehensions and generator expressions. I think this is consistent with Yuri's, Serhiy's, and Ivan's claims, but I admit I'm not sure. Of course the compiler need not create a generator function and invoke it, but the resulting bytecode should be the same as if it did.

This means the semantics of [FOR-EXPRESSION] and [(FOR-EXPRESSION)] should differ in the same way.

Steve

• Previous message (by thread): [Python-Dev] Tricky way of of creating a generator via a comprehension expression
• Next message (by thread): [Python-Dev] Tricky way of of creating a generator via a comprehension expression
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the Python-Dev mailing list