[Python-Dev] PEP 289 - Generator Expressions (original) (raw)

[Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward

Guido van Rossum guido at python.org
Fri Apr 30 08:16:32 EDT 2004

I did a quick review of the stdlib, including the tests, to see which list comprehensions could be replaced with generator expressions. Thought I admit I am biased towards early binding, I ended up looking for cases with the following properties:

* doesn't use the result immediately * contains free variables This is less that 10% of the cases. However, for each of them, I had to check if the free variables had a chance of being modified after the genexpr. This is pretty rare, admittedly, but there is an example in test/testrandom.py (testavgstd). There are a number of other examples that just don't happen to modify the free variables, by chance; e.g. in optparse.py: metavar = option.metavar or option.dest.upper() shortopts = [sopt + metavar for sopt in option.shortopts] longopts = [lopt + "=" + metavar for lopt in option.longopts] If we find out later that longopts actually needs a slightly different value for metavar, it would be tempting to do: metavar = option.metavar or option.dest.upper() shortopts = [sopt + metavar for sopt in option.shortopts] metavar = option.metavar or option.dest.capitalize() longopts = [lopt + "=" + metavar for lopt in option.longopts] Replace these with genexprs and it doesn't work any more: the 2nd metavar is unexpectedly used in the first genexpr as well. In general I find it strange to have to look if a given variable could be modified much later in the same function to be sure of what a genexpr really means. Early binding is closer to the idea that turning a listcomp into a genexprs should just work if you only iterate once on the result.

Thanks for the analysis. My comment on this: why on earth would you want to replace the perfectly sensible list comprehension with a generator comprehension in this particular case? (Note that the remainder of that function proceeds to do list concatenation of the two lists.)

A more general question would be: have you found any examples of list comprehensions that weren't immediately used where there would be a compelling reason to use a generator expression instead? In this example, I can't see any of the touted advantages -- the set of command line options will never be long enough to have to worry about the memory consumption. How about the other examples you found?

--Guido van Rossum (home page: http://www.python.org/~guido/)

More information about the Python-Dev mailing list