Hans-Peter Nilsson - Re: Performance regression testing? (original) (raw)

This is the mail archive of the gcc@gcc.gnu.orgmailing list for the GCC project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

On Mon, 28 Nov 2005, Mike Stump wrote:

On Nov 28, 2005, at 6:21 PM, Hans-Peter Nilsson wrote:

I've attached the work-in-progress so I don't have to get into detail about what it does :-) except noting that you'll see in gcc.sum something like:

PASS: csibe -O1 runtime zlib-1.1.4:minigzip not slower than best PASS: csibe -O1 runtime zlib-1.1.4:minigzip not more than .1% slower than best PASS: csibe -O1 runtime zlib-1.1.4:minigzip not more than 1% slower than best PASS: csibe -O1 runtime zlib-1.1.4:minigzip not more than 10% slower than best

Hum, I'd prefer that the output format be:

PERF: %f name of test

You seem to be interpreting the gcc.sum format, thinking it's the raw "baseline" format. Which for the record is like: ... runtime,-O1,zlib-1.1.4:minigzip,previous 0.32 runtime,-O1,bzip2-1.0.2:bzip2.d,previous 0.32 runtime,-O1,bzip2-1.0.2:bzip2recover,previous 0.19 ... That was the "native" x86_64 output. Here's for cris-linux+cris-sim: .. runtime,-O1,zlib-1.1.4:minigzip,previous 1262089345.0 runtime,-O1,bzip2-1.0.2:bzip2.d,previous 945199067.0 runtime,-O1,bzip2-1.0.2:bzip2recover,previous 1555998754.0

Can't be compared with each other, if that's what you mean (how would that make sense?) but quite comparable to other baselines methinks. I refer to the implementation for further details.

then have an analysis package crunch that into the above format.

...and emitting PASS/FAIL after comparing two or more arbitrary runs according to some criteria? Like above? ;-)

This way, one can compare two arbitrary runs, which is a useful property. The number can be thought of as time, such as the number of clock cycles, but we only reeally care that it starts at zero, and gets bigger as things go bad.

You have to elaborate here. How does "biasing" the number of cycles to make it 0 help? Do you mean a deviation normalized between 0 and 1? How would you "reset" it? I think it'd just be confusing, and you'd have to expect and handle negative numbers. Better keep an explicit built-in baseline. Already implemented by looking in gcc.performance/csibe/baselines/$target_triplet and reading whatever file is there according to the format above. ...oops, a bug in the posted code; missing "$" on $subdir. And untested anyway.

brgds, H-P

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]