| msg51005 - (view) |
Author: Georg Brandl (georg.brandl) *  |
Date: 2006-08-29 08:33 |
| This is a big one: * cleanup grammar; unifies listcomp/genexp grammar which means that [x for x in 1, 2] is no longer valid * cleanup comprehension compiling code (unifies all AST code for the three comprehensions and most of the compile.c code) * add set comprehensions This patch modifies list comprehensions to be implemented more like generator expressions: in a separate function, which means that the loop variables will not leak any more. |
|
|
| msg51006 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2006-08-29 08:34 |
| Logged In: YES user_id=849994 The previously attached patch contains only the important files. The FULL patch (attached now) also contains syntax fixes in python files so that the test suite is mostly passing. Note that the compiler package isn't ready yet. |
|
|
| msg51007 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2006-08-29 17:59 |
| Logged In: YES user_id=6380 Nice! I see failures in 4 tests: test_compiler test_dis test_transformer test_univnewlines test_univnewlines is trivial (it's deleting a variable leaked out of a list comprehension); haven't looked at the rest in detail |
|
|
| msg51008 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2006-08-29 19:09 |
| Logged In: YES user_id=849994 test_compiler and test_transformer fail because the compiler package hasn't been updated yet. test_dis fails because list comprehensions now generate different bytecode. |
|
|
| msg51009 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2006-08-29 22:30 |
| Logged In: YES user_id=80475 Can you post a before and disassembly of some list and set comprehensions. |
|
|
| msg51010 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2006-08-31 19:55 |
| Logged In: YES user_id=849994 Attaching slightly revised patch and bytecode comparison. |
|
|
| msg51011 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2006-08-31 23:15 |
| Logged In: YES user_id=80475 Would it be an oversimplfication for list and set comps to keep everything in one code block and just hide the list loop variables by renaming them: x --> __[x] That approach would only require a minimal patch, and it would make for a cleaner disassembly. |
|
|
| msg51012 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2006-08-31 23:40 |
| Logged In: YES user_id=6380 +1. Would this cause problems for abominations like this though? >>> a=[1] >>> list(tuple(a) for a[0] in "abc") [('a',), ('b',), ('c',)] >>> a ['c'] >>> |
|
|
| msg51013 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2006-09-01 09:38 |
| Logged In: YES user_id=849994 Since you can put anything usable as an assignment target after the "for" of a listcomp, just renaming might be complicated. |
|
|
| msg51014 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2006-09-06 06:48 |
| Logged In: YES user_id=6380 Do you think this is ready to be checked in, or are you still working on it? |
|
|
| msg51015 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2006-09-06 07:03 |
| Logged In: YES user_id=849994 It is complete, it works and it does not leak the loop variable(s). The question is whether it is okay for listcomps and setcomps to be in their own anonymous function, which slows listcomps down compared to the 2.x branch. I don't know why the function approach was taken for genexps, but I suspect it was because who implemented it then saw this as the best way to hide the loop variable. Perhaps somebody else more familiar with the internals and the previous discussions can look over it. |
|
|
| msg51016 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2006-09-06 15:36 |
| Logged In: YES user_id=6380 I always assumed that the genexps *require* being a function because that's the only way to create a generator. But that argument doesn't apply to listcomps. That's about all I know of the implementation of these.. :-( Have you asked python-dev? |
|
|
| msg51017 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2006-09-08 17:13 |
| Logged In: YES user_id=80475 Genexps necessarily need a separate stack frame to achieve saved execution state (including the instruction pointer and local variable). Also, it was simplest to implement genexps in terms of the existing and proven code for regular generators. For list and set comps, I think you can take a simpler approach and just rename the inner loop variable to something invisible. That will make it faster, make the disassemby readable, and make it easier to follow in pdb. Also, we get to capitalize on proven code -- they only difference is that the induction variable won't be visible to surrounding code. Since what you have works, I would say just check it in; however, it would probably never get touched again and an early, arbitrary design choice would get set in stone. My bet is that the renaming approach will result in a much simpler patch. |
|
|
| msg51018 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2007-01-04 04:57 |
| There was some discussion on the py3k list about Raymond's suggestion. Are you thinking of doing that? I'd really like to see the syntactic changes and additions from this patch, but I agree that for list/set comps we can do without the extra stack frame. |
|
|
| msg51019 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2007-03-07 09:23 |
| This patch is now superseded by #1660500. |
|
|