[Python-3000] List & set comprehensions patch (original) (raw)

Georg Brandl g.brandl at gmx.net
Tue Mar 6 18:38:27 CET 2007

Nick Coghlan schrieb:

Georg and I have been working on the implementation of list comprehensions which don't leak their iteration variables, along with the implementation of set comprehensions. The latest patch can be found as SF patch #1660500 [1]. The file new-set-comps.diff is the combined patch which implements both features, and unifies handling of the different kinds of comprehension. In an effort to improve readability, the patch also converts the sets in symtable.c to be actual PySet objects, rather than PyDict objects with None keys and tries to reduce the number of different meanings assigned to the term 'scope'.

One of the comments made on Georg's initial attempt at implementing these features was that it would be nice to avoid the function call overhead in the listcomp & setcomp case (as it appears at first glance that the internal scope can be temporary). I tried to do that and essentially failed outright - working through symtable.c and compile.c, I found that dealing with the scoping issues created by the possibility of nested genexps, lambdas and list or set comprehensions would pretty much require reimplementing all of the scoping rules that functions already provide.

I have to thank you for digging through all that code and cases, which I always dreaded ;)

There are also a couple of tests we had to disable - one in testdis, one in testgrammar. Suggestions on how to reinstate those (or agreement that it is OK to get rid of them) would be appreciated.

The one in test_grammar is certainly okay, since it uses syntax which was agreed upon to be invalid in Py3k.

The PySet update code in symtable.c currently uses PyNumberInplaceOr with a subsequent call to PyDECREF to counter the implicit call to PyINCREF. Should this be changed to use PyObjectCallMethod to invoke the Python level update method?

Why not add a C-level PySet_Update API?

There are also two backwards compatibility problems which came up:

- code which explicitly deleted the listcomp variable started throwing NameErrors. Several tweaks were needed in the standard library to fix this.

This may be hard to catch with the 2to3 tool, but perhaps a certain heuristics (del statement after a statement involving a list comprehension) could apply. Furthermore, I don't think that "trick" is mostly used on module-level to avoid cluttering up the namespace, so a codebase which really uses its modules should see early failures.

- only the outermost iterator expression is evaluated in the scope containing the comprehension (just like generator expressions). This means that the inner expressions can no longer see class variables and values in explicit locals() dictionaries provided to exec & friends. This didn't actually cause any problems in the standard library - I only note it because my initial implementation mistakenly evaluated the outermost iterator in the new scope, which did cause severe problems along these lines.

It must be noted that this was always the case for generator expressions, so if a LC should only be syntactic sugar for list(genexp) as written in the PEP, this is even correct behavior.

cheers, Georg

More information about the Python-3000 mailing list