[Python-Dev] [issue6673] Py3.1 hangs in coroutine and eats up all memory (original) (raw)

Stefan Behnel stefan_ml at behnel.de
Wed Aug 12 16:33:54 CEST 2009

Nick Coghlan wrote:

Stefan Behnel wrote:

This is also an important issue for other Python implementations. Cython simply transforms comprehensions into the equivalent for-loop, so when we implement PEP 342 in Cython, we will have to find a way to emulate CPython's behaviour here (unless we decide to stick with Py2.x sematics, which would not be my preferred solution). How do you do that without leaking the iteration variable into the current namespace?

We currently have 2.x sematics for comprehensions anyway, but the (long-standing) idea is to move comprehensions into their own scope (not a function, just a new type of scope), so that all names defined inside the expressions end up inside of the inner scope. This is completely orthogonal to the loop transformation itself, though, which would simply happen inside of the inner scope.

However, having to emulate the other Py3 semantics for comprehensions that this thread is about, would pretty much kill such a simple solution.

The translation of your example isn't quite as Alexandre describes it - we do at least avoid the overhead of creating a generator function in the list comprehension case. It's more like:

while True: def f(): result = [] for i in range(chunksize): result.append((yield)) return result target.send(f())

So the problem is that f(), i.e. the function-wrapped comprehension itself, swallows the "(yield)" expression (which redundantly makes it a generator). That means that the outer function in my example, which was

def chunker(chunk_size, target):
    while True:
        target.send([ (yield) for i in range(chunk_size) ])

doesn't become a generator itself, so the above simply ends up as an infinite loop.

IMHO, that's pretty far from obvious when you look at the code.

Also, the target receives a "generator object " instead of a list. That sounds weird.

It seems to me that CPython should be raising a SyntaxError for yield expressions inside comprehensions (in line with the "no returning values other than None from generator functions" rule), and probably for generator expressions as well.

Yes, that's what I was suggesting. Disallowing it in genexps is a more open question, though. I wouldn't mind being able to send() values into a generator expression, or to throw() exceptions during their execution.

Anyway, I have no idea about a use case, so it might just as well be disallowed for symmetry reasons.

Stefan

More information about the Python-Dev mailing list