[Python-Dev] [issue6673] Py3.1 hangs in coroutine and eats up all memory (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Wed Aug 12 15:00:45 CEST 2009

Stefan Behnel wrote:

This is also an important issue for other Python implementations. Cython simply transforms comprehensions into the equivalent for-loop, so when we implement PEP 342 in Cython, we will have to find a way to emulate CPython's behaviour here (unless we decide to stick with Py2.x sematics, which would not be my preferred solution).

How do you do that without leaking the iteration variable into the current namespace?

Avoiding that leakage is where the semantic change between 2.x and 3.x came from here: 2.x just creates the for loop inline (thus leaking the iteration variable into the current scope), while 3.x creates an inner function that does the iteration so that the iteration variables exist in their own scope without polluting the namespace of the containing function.

The translation of your example isn't quite as Alexandre describes it - we do at least avoid the overhead of creating a generator function in the list comprehension case. It's more like:

while True:
    def f():
        result = []
        for i in range(chunk_size):
            result.append((yield))
        return result
    target.send(f())

So what you end up with is a generator that has managed to bypass the syntactic restriction that disallows returning non-None values from generators. In CPython it appears that happens to end up being executed as if the return was just another yield expression (most likely due to a quirk in the implementation of RETURN_VALUE inside generators):

while True:
    def f():
        result = []
        for i in range(chunk_size):
            result.append((yield))
        yield result
    target.send(f())

It seems to me that CPython should be raising a SyntaxError for yield expressions inside comprehensions (in line with the "no returning values other than None from generator functions" rule), and probably for generator expressions as well.

Cheers, Nick.

P.S. Experimentation at a 3.x interpreter prompt:

def f(): ... return [(yield) for i in range(10)] ... x = f() next(x) for i in range(8): ... x.send(i) ... x.send(8) next(x) [0, 1, 2, 3, 4, 5, 6, 7, 8, None] x = f() next(x) for i in range(10): # A statement with a return value! ... x.send(i) ... [0, 1, 2, 3, 4, 5, 6, 7, 8, None] dis(f) 2 0 LOAD_CONST 1 (<code object at 0xb7c53bf0, file "", line 2>) 3 MAKE_FUNCTION 0 6 LOAD_GLOBAL 0 (range) 9 LOAD_CONST 2 (10) 12 CALL_FUNCTION 1 15 GET_ITER 16 CALL_FUNCTION 1 19 RETURN_VALUE dis(f.code.coconsts[1]) 2 0 BUILD_LIST 0 3 LOAD_FAST 0 (.0) 6 FOR_ITER 13 (to 22) 9 STORE_FAST 1 (i) 12 LOAD_CONST 0 (None) 15 YIELD_VALUE 16 LIST_APPEND 2 19 JUMP_ABSOLUTE 6 22 RETURN_VALUE

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list