Issue 6673: Uncaught comprehension SyntaxError eats up all memory (original) (raw)

Issue6673

Created on 2009-08-09 14:09 by scoder, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (11)
msg91428 - (view)	Author: Stefan Behnel (scoder) *	Date: 2009-08-09 14:09
Here's a simple coroutine that works perfectly in Python 2.6 but seems to let Py3.1 enter an infinite loop that ends up eating all memory. ----------------- def printing_sink(): "A simple sink that prints the received values." while True: print( (yield) ) def chunker(chunk_size, target): """Receives single items and forwards chunks of a fixed size. Usage example: >>> sink = printing_sink() >>> next(sink) >>> cr = chunker(4, sink) >>> next(cr) >>> for i in range(8): ... cr.send(i) [0, 1, 2, 3] [4, 5, 6, 7] >>> cr.close() """ while True: target.send([ (yield) for i in range(chunk_size) ]) if __name__ == '__main__': import doctest doctest.testmod() ----------------- Fails on: Python 3.1 (r31:73572, Jun 28 2009, 21:07:35) [GCC 4.3.2] on linux2 Works on: Python 2.6.2 (r262:71600, Apr 17 2009, 11:29:30) [GCC 4.3.2] on linux2 The problem seems to be the list comprehension. When I replace it with a normal for-loop, it works perfectly.
msg91432 - (view)	Author: Georg Brandl (georg.brandl) *	Date: 2009-08-09 21:22
Try list(genexp) instead of [listcomp] in 2.x and see what happens...
msg91435 - (view)	Author: Stefan Behnel (scoder) *	Date: 2009-08-09 22:06
Hmm, ok, so this is actually an anticipated bug? And I assume this has been discussed before and was decided to get solved by doing... what? Is it documented somewhere why this happens and what one must avoid to not run into this kind of pitfall?
msg91436 - (view)	Author: Georg Brandl (georg.brandl) *	Date: 2009-08-09 22:09
No idea, actually. I just wanted to point out that it is nothing specific to Python 3.
msg91488 - (view)	Author: Alexandre Vassalotti (alexandre.vassalotti) *	Date: 2009-08-11 21:58
Not a bug. The list comprehension in your chunker: while True: target.send([ (yield) for i in range(chunk_size) ]) is equivalent to the following generator in Python 3: while True: def g(): for i in range(chunk_size): yield (yield) target.send(list(g())) This clearly needs not what you want. So, just rewrite your code using for-loop: while True: result = [] for i in range(chunk_size): result.append((yield)) target.send(result)
msg91582 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2009-08-14 23:53
Reopening - this should be rejected by the compiler as a SyntaxError, since it is the comprehension equivalent of writing "return Value" inside a generator function. Specifically, in 3.x, the equivalent written out code is: while True: def _listcomp(): result = [] for i in range(chunk_size): result.append((yield)) return result # Would trigger SyntaxError! target.send(_listcomp()) As noted in the comment, the compiler would disallow that expansion with a SyntaxError on the marked line, so the comprehension equivalent should be rejected as well. This also applies to dict and set comprehensions. Generator expressions are slightly less clear, since their expanded equivalent would produce legal code: >>> g = ((yield)*(yield) for i in range(1)) >>> next(g) >>> g.send(2) >>> g.send(3) 6 >>> next(g) Traceback (most recent call last): File "", line 1, in StopIteration That first line is roughly equivalent to: def _genexp(): for i in range(1): yield (yield)*(yield) g = _genexp() So the compiler should probably be keeping track of whether it is inside a comprehension or not to decide whether or not to allow yield expressions.
msg91656 - (view)	Author: Stefan Behnel (scoder) *	Date: 2009-08-17 06:32
Very good argumentation, thanks Nick! I think this is worth being fixed in the 3.1 series.
msg139227 - (view)	Author: Terry J. Reedy (terry.reedy) *	Date: 2011-06-26 21:32
With 32bit winxp Windows Task Manager open to Processes, sorted by Men Usage, I verified expanding memory usage in 3.2.0 (and stopped at 200 mb versus normal <20Mb). While we do not expect compiler to catch while True: pass, it would be nice if this was, with a reasonable error message. No one marked for 2.7 because behavior only occurs with genexp and Nick says not so clearly a SyntaxError.
msg139234 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2011-06-27 01:47
The acceptance of PEP 380 actually muddies the waters on this one - the expansion I cited as a syntax error now has a defined meaning. Specifically, list comprehensions containing yield expressions would become generators that return a list (set and dict comprehensions would become generators that return a set or dict, respectively). Once it is confirmed those semantics are an acceptable outcome of PEP 380's acceptance, this behaviour would be documented in the appropriate locations (likely under the yield expression, with pointers from other parts of the documentation).
msg221003 - (view)	Author: Mark Lawrence (BreamoreBoy) *	Date: 2014-06-19 16:10
Assuming that documentation changes are needed, who's best placed to do them?
msg316315 - (view)	Author: Serhiy Storchaka (serhiy.storchaka) *	Date: 2018-05-09 08:40
"yield" in comprehensions is deprecated in 3.7: ../issue6673.py:22: DeprecationWarning: 'yield' inside list comprehension target.send([ (yield) for i in range(chunk_size) ]) and an error in 3.8: File "../issue6673.py", line 22 target.send([ (yield) for i in range(chunk_size) ]) ^ SyntaxError: 'yield' inside list comprehension