[Python-Dev] Re: new bytecode results (original) (raw)

Tim Peters tim.one@comcast.net
Thu, 27 Feb 2003 22:21:20 -0500

[Terry Reedy]

... The problems of teaching to the test and programming to the benchmark are well-known. ... Your contest, my questions and comments.

Not to worry -- I made my living for 15 years writing optimizing compilers in the highly competitive (at the time -- it competed itself to death) supercomputer biz. I've forgotten more dirty tricks than even a Perl guy can pick up in a lifetime .

One of my favorites: one of the rules in a particular competitive benchmark was that editing the source code was not allowed. Everyone had to run the source file verbatim from the prospective customer. One of Cray's competitors (and, no, I'm not making this up) implemented a simple sed-like subsystem driven by text on the compiler command line, which effectively allowed them to edit the source code without physically changing the file. They used this to dump all sorts of compiler directives into the benchmark code, and even to replace a particular critical loop nest with a call to a highly optimized library subroutine.

Their results were quite impressive. But ours were even better: general global analysis deduced that the computed results weren't used for anything, so we optimized away all the expensive loops. Our results weren't all to the good, though: the program ran so fast then that the "operations per second" output at the end overflowed to a negative integer.

Heh. Believe it or not, customers were once so naive that they actually looked at benchmark results as if they measured something relevant to their business .

beauty-is-a-more-objective-measure-ly y'rs - tim