(original) (raw)

On Oct 7, 2016, at 6:46 PM, Hal Finkel via llvm-dev <llvm-dev@lists.llvm.org> wrote:

----- Original Message -----

From: "Hal Finkel via llvm-dev" <llvm-dev@lists.llvm.org>
To: "Sebastian Pop" <sebpop.llvm@gmail.com>
Cc: "Sebastian Paul Pop" <s.pop@samsung.com>, "llvm-dev" <llvm-dev@lists.llvm.org>, "Matthias Braun"
<matze@braunis.de>, "Clang Dev" <cfe-dev@lists.llvm.org>, "nd" <nd@arm.com>, "Abe Skolnik" <a.skolnik@samsung.com>
Sent: Friday, October 7, 2016 7:56:53 PM
Subject: Re: \[llvm-dev\] \[test-suite\] making the test-suite succeed with "-Ofast" and "-ffp-contract=on"

----- Original Message -----

From: "Sebastian Pop" <sebpop.llvm@gmail.com>
To: "Renato Golin" <renato.golin@linaro.org>
Cc: "Kristof Beyls" <Kristof.Beyls@arm.com>, "Sebastian Paul Pop"
<s.pop@samsung.com>, "llvm-dev"
<llvm-dev@lists.llvm.org>, "nd" <nd@arm.com>, "Abe Skolnik"
<a.skolnik@samsung.com>, "Clang Dev"
<cfe-dev@lists.llvm.org>, "Hal Finkel" <hfinkel@anl.gov>, "Stephen
Canon" <scanon@apple.com>, "Matthias Braun"
<matze@braunis.de>
Sent: Friday, October 7, 2016 7:34:40 PM
Subject: \[test-suite\] making the test-suite succeed with "-Ofast"
and "-ffp-contract=on"

Hi,

I would like to provide a summary of the different proposals on how
to
fix the test-suite to make it succeed when specifying extra CFLAGS
"-Ofast" and "-ffp-contract=on". I would like to expose the issue
and
proposed ways to fix it to other potential reviewers that could
provide extra feedback. We also need to decide which proposal (or
combination of) to implement and commit.

Proposal 1: https://reviews.llvm.org/D25277
modify the CMakes to compile and run each of these benchmarks
twice:
once with added CFLAGS -ffp-contract=off. Record on disk the full
output of both runs and compare with FP\_TOLERANCE. Hash the output
of
the run with -ffp-contract=off and exact match against the
reference
output.

The good for Proposal 1:
- changes contained in the build system: no change to the code of
the
benchmarks
- runs benchmarks under an extra configuration with CFLAGS +=
-ffp-contract=off

The bad for Proposal 1:
- compilation time will double
- running time on the device will double
- build system is more complex
- the build directory goes from 300M to 1.2G due to the extra
reference outputs recorded under -ffp-contract=off,
- when running test-suite over small devices it will cost 1G more
transfer over the network.

I prefer proposal 1 (although, to be fair, it was something I
suggested). Being the the business of trying to heavily modify every
benchmark that does floating-point computation, as in proposal 2,
does not seem to scale well, and can't always be done regardless.

We can make some effort to reduce the size of the problems being
computed by some of the benchmarks (e.g. pollybench); I think that
is reasonable and will help with the extra space requirements. That
having been said, functionally speaking, our test suite is at least
an order of magnitude too small, and so my sympathy is somewhat
limited. We're going to have to find a way to execute the test suite
in stages on smaller devices to limit the peak usage, if not because
of this then because we've added a lot more test applications and
benchmarks in the future.

Another aspect to this is that we should have this kind of infrastructure for other purposes as well. We have a similar lack of testing for -ffast-math. We don't even do a good job of (i.e. have good buildbot coverage for) running the test suite @ -O1\. -O2 and -O3 are much better tested. More regular testing at -O0 (especially with -g to pick up crashes in our debug-info generation logic) is needed as well.