[Python-Dev] RE: list comprehensions (was parsers and compilers for 2.0) (original) (raw)

Tim Peters tim_one@email.msn.com
Sun, 13 Aug 2000 20:08:45 -0400

[Tim]

List comprehensions are one of the best-loved features of Haskell (really!), and Greg/Skip/Ping's patch implements as an exact a parallel to Haskell's syntax and semantics as is possible in Python.

[Thomas Wouters]

I don't see "it's cool in language X" as a particular good reason to include a feature... We don't add special syntax for regular expressions, support for continuations or direct access to hardware because of that, do we ?

As Guido (overly!) modestly says, the only language idea he ever invented was an "else" clause on loops. He decided listcomps "were Pythonic" before knowing anything about Haskell (or SETL, from which Haskell took the idea). Given that he already liked them, the value in looking at Haskell is for its actual experience with them. It would be pretty stupid not to look at experience with other languages that already have it! And that's whether you're pro or con.

So it's not "cool in language X" that drives it at all, it's "cool in language X" that serves to confirm or refute the prior judgment that "it's Pythonic, period". When, e.g., Eric predicts it will bite us hard someday, I can point to Haskell and legitimately ask "why here and not there?".

There was once a great push for adding some notion of "protected" class members to Python. Guido was initially opposed, but tempted to waffle because proponents kept pointing to C++. Luckily, Stroustrup had something to say about this in his "Design and Evolution of C++" book, including that he thought adding "protected" was a mistake, driven by relentless "good arguments" that opposed his own initial intuition. So in that case, really looking at C++ may have saved Guido from making the same mistake.

As another example, few arguments are made more frequently than that Python should add embedded assignments in conditionals. Lots of other languages have that -- but they mostly serve to tell us it's a bug factory in practice! The languages that avoid the bugs point to ways to get the effect safely (although none yet Pythonically enough for Guido).

So this is a fact: language design is very little about wholesale invention, and that's especially true of Python. It's a mystically difficult blending of borrowed ideas, and it's rare as readable Perl that an idea will get borrowed if it didn't even work well in its native home. listcomps work great where they came from, and that plus "hey, Guido likes 'em!" makes it 99% a done deal.

My main beef with the syntax is that it is, in my eyes, unpythonic. It has an alien, forced feel to it, much more so than the 'evil' map/filter/reduce. It doesn't 'fit' into Python the way most of the other features do;

Guido feels exactly the opposite: the business about "alien, forced feel, not fitting" is exactly what he's said about map/filter/reduce/lambda on many occasions. listcomps strike him (me too, for that matter) as much more Pythonic than those.

it's simply syntactic sugar for a specific kind of for-loop. It doesn't add any extra functionality,

All overwhelmingly true of augmented assignments, and range literals, and three-argument getattr, and list.pop, etc etc etc too. Python has lots of syntactic sugar -- making life pleasant is not a bad thing.

and for that large a syntactic change, I guess that scares me.

The only syntactic change is to add a new form of list constructor. It's isolated and self-contained, and so "small" in that sense.

Those doubts were why I was glad you were going to write the PEP. I was looking forward to you explaining why I had those doubts and giving sound arguments against them :-)

There is no convincing argument to made either way on whether "it's Pythonic", which I think is your primary worry. People never reach consensus on whether a given feature X "is Pythonic". That's why it's always Guido's job. You've been here long enough to see that -1 and +1 are about evenly balanced, except on (in recent memory) "import x as y" -- which I conviently neglected to mention had been dismissed as unPythonic by Guido just a couple weeks ago <wink -- but he didn't really mean it then, according to me>.

... but that's mostly because I was expecting, like ESR, a huge debate on its syntax.

Won't happen, because it already did. This was debated to death long ago, and more than once, and Guido likes what he likes now. Greg Wilson made the only new point on listcomps I've seen since two weeks after they were first proposed by Greg Ewing (i.e., that the ";" notation really sucked).

Lets say that most my doubts arose after playing with it for a while. I fear people will start using it in odd construct, and in odd ways,

Unlike, say, filter, map and reduce ?

expecting other aspects of for-loops to be included in list comprehensions (break, else, continue, etc.)

Those ideas were rejected long ago too (and that Haskell and SETL also rejected them independently shows that, whether we can explain it or not, they're simply bad ideas).

And then there's the way it's hard to parse because of the lack of punctuation in it:

[((a,b)*c, (spam(d)%34)^e) for a in [(x, y) for x in L for y in S] for b in [b for b in B if mean(b)] for b,c in C for a,d in D for e in [Egg(a, b, c, d, e) for e in E]]

That isn't a serious argument, to my eyes. Write that as a Lisp one-liner and see what it looks like then -- nuts is nuts, and a "scare example" could just as easily be concocted out of insane nesting of subscripts and/or function calls and/or parenthesized arithmetic. Idiotic nesting is a possibility for any construct that nests! BTW, you're missing the possibility to nest listcomps in "the expression" position too, a la

[[1 for i in range(n)] for n in range(10)] [[], [1], [1, 1], [1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1]]

I know you missed that possibility above because, despite your claim of being hard to parse, it's dead easy to spot where your listcomps begin: "[" is easy for the eye to find.

I hope anyone writing something like that (notice the shadowing of some of the outer vrbls in the inner loops)

You can find the same in nested lambdas littering map/reduce/etc today.

will either add some newlines and indentation by themselves, or will be hunted down and shot (or at least winged) by the PSU.

Nope! We just shun them. Natural selection will rid the Earth of them without violence .

I'm not arguing to remove list comprehensions. I think they are cool features that can replace map/filter, I just don't think they're that much better than the use of map/filter.

Haskell programmers have map/filter too, and Haskell programmers routinely favor using listcomps. This says something about what people who have both prefer. I predict that once you're used to them, you'll find them much more expressive: "[" tells you immediately you're getting a list, then the next thing you see is what the list is built out of, and then there's a bunch of lower-level detail. It's great.

Write-that-PEP-Tim-it-will-look-good-on-your-resume-ly y'rs,

except-i'm-too-old-to-need-a-resume-anymore-ly y'rs - tim