test-suite Guide — LLVM 21.0.0git documentation (original) (raw)

Quickstart

  1. The lit test runner is required to run the tests. You can either use one from an LLVM build:
    % /bin/llvm-lit --version
    lit 20.0.0dev
    An alternative is installing it as a Python package in a Python virtual environment:
    % python3 -m venv .venv
    % . .venv/bin/activate
    % pip install git+https://github.com/llvm/llvm-project.git#subdirectory=llvm/utils/lit
    % lit --version
    lit 20.0.0dev
    Installing the official Python release of lit in a Python virtual environment could also work. This will install the most recent release of lit:
    % python3 -m venv .venv
    % . .venv/bin/activate
    % pip install lit
    % lit --version
    lit 18.1.8
    Please note that recent tests may rely on features not in the latest released lit. If in doubt, try one of the previous methods.
  2. Check out the test-suite module with:
    % git clone https://github.com/llvm/llvm-test-suite.git test-suite
  3. Create a build directory and use CMake to configure the suite. Use theCMAKE_C_COMPILER option to specify the compiler to test. Use a cache file to choose a typical build configuration:
    % mkdir test-suite-build
    % cd test-suite-build
    % cmake -DCMAKE_C_COMPILER=/bin/clang \
    -C../test-suite/cmake/caches/O3.cmake \
    ../test-suite

NOTE! if you are using your built clang, and you want to build and run the MicroBenchmarks/XRay microbenchmarks, you need to add compiler-rt to yourLLVM_ENABLE_RUNTIMES cmake flag.

  1. Build the benchmarks:
    % make
    Scanning dependencies of target timeit-target
    [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
    [ 0%] Linking C executable timeit-target
    ...
  2. Run the tests with lit:
    % llvm-lit -v -j 1 -o results.json .
    -- Testing: 474 tests, 1 threads --
    PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
    ********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESULTS **********
    compile_time: 0.2192
    exec_time: 0.0462
    hash: "59620e187c6ac38b36382685ccd2b63b"
    size: 83348

PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
...

NOTE! even in the case you only want to get the compile-time results(code size, llvm stats etc), you need to run the test with the above llvm-lit command. In that case, the results.json file will contain compile-time metrics.

  1. Show and compare result files (optional):

Make sure pandas and scipy are installed. Prepend sudo if necessary.

% pip install pandas scipy

Show a single result file:

% test-suite/utils/compare.py results.json

Compare two result files:

% test-suite/utils/compare.py results_a.json results_b.json

Structure

The test-suite contains benchmark and test programs. The programs come with reference outputs so that their correctness can be checked. The suite comes with tools to collect metrics such as benchmark runtime, compilation time and code size.

The test-suite is divided into several directories:

Benchmarks

Every program can work as a correctness test. Some programs are unsuitable for performance measurements. Setting the TEST_SUITE_BENCHMARKING_ONLY CMake option to ON will disable them.

The MultiSource benchmarks consist of the following apps and benchmarks:

MultiSource Language Application Area Remark
7zip C/C++ Compression/Decompression
ASCI_Purple C SMG2000 benchmark and solver Memory intensive app
ASC_Sequoia C Simulation and solver
BitBench C uudecode/uuencode utility Bit Stream benchmark for functional compilers
Bullet C++ Bullet 2.75 physics engine
DOE-ProxyApps-C++ C++ HPC/scientific apps Small applications, representative of our larger DOE workloads
DOE-ProxyApps-C C HPC/scientific apps “
Fhourstones C Game/solver Integer benchmark that efficiently solves positions in the game of Connect-4
Fhourstones-3.1 C Game/solver “
FreeBench C Benchmark suite Raytracer, four in a row, neural network, file compressor, Fast Fourier/Cosine/Sine Transform
llubenchmark C Linked-list micro-benchmark
mafft C Bioinformatics A multiple sequence alignment program
MallocBench C Benchmark suite cfrac, espresso, gawk, gs, make, p2c, perl
McCat C Benchmark suite Quicksort, bubblesort, eigenvalues
mediabench C Benchmark suite adpcm, g721, gsm, jpeg, mpeg2
MiBench C Embedded benchmark suite Automotive, consumer, office, security, telecom apps
nbench C BYTE Magazine’s BYTEmark benchmark program
NPB-serial C Parallel computing Serial version of the NPB IS code
Olden C Data Structures SGI version of the Olden benchmark
OptimizerEval C Solver Preston Brigg’s optimizer evaluation framework
PAQ8p C++ Data compression
Prolangs-C++ C++ Benchmark suite city, employ, life, NP, ocean, primes, simul, vcirc
Prolangs-C C Benchmark suite agrep, archie-client, bison, gnugo, unix-smail
Ptrdist C Pointer-Intensive Benchmark Suite
Rodinia C Scientific apps backprop, pathfinder, srad
SciMark2-C C Scientific apps FFT, LU, Montecarlo, sparse matmul
sim C Dynamic programming A Time-Efficient, Linear-Space Local Similarity Algorithm
tramp3d-v4 C++ Numerical analysis Template-intensive numerical program based on FreePOOMA
Trimaran C Encryption 3des, md5, crc
TSVC C Vectorization benchmark Test Suite for Vectorizing Compilers (TSVC)
VersaBench C Benchmark suite 8b10b, beamformer, bmm, dbms, ecbdes

All MultiSource applications are suitable for performance measurements and will run when CMake option TEST_SUITE_BENCHMARKING_ONLY is set.

Configuration

The test-suite has configuration options to customize building and running the benchmarks. CMake can print a list of them:

% cd test-suite-build

Print basic options:

% cmake -LH

Print all options:

% cmake -LAH

Common Configuration Options

Common CMake Flags

Displaying and Analyzing Results

The compare.py script displays and compares result files. A result file is produced when invoking lit with the -o filename.json flag.

Example usage:

count 506.000000
mean 20.563098
std 111.423325
min 0.003400
25% 0.011200
50% 0.339450
75% 4.067200
max 1222.896800

Continuous Tracking with LNT

LNT is a set of client and server tools for continuously monitoring performance. You can find more information athttps://llvm.org/docs/lnt. The official LNT instance of the LLVM project is hosted at http://lnt.llvm.org.

External Suites

External suites such as SPEC can be enabled by either

You can find further information in the respective README files such astest-suite/External/SPEC/README.

For the SPEC benchmarks you can switch between the test, train andref input datasets via the TEST_SUITE_RUN_TYPE configuration option. The train dataset is used by default.

In addition to SPEC, the multimedia frameworks ffmpeg and dav1d can also be hooked up as external projects in the same way. By including them in llvm-test-suite, a lot more of potentially vectorizable code gets compiled

Custom Suites

You can build custom suites using the test-suite infrastructure. A custom suite has a CMakeLists.txt file at the top directory. The CMakeLists.txt will be picked up automatically if placed into a subdirectory of the test-suite or when setting the TEST_SUITE_SUBDIRS variable:

% cmake -DTEST_SUITE_SUBDIRS=path/to/my/benchmark-suite ../test-suite

Profile Guided Optimization

Profile guided optimization requires to compile and run twice. First the benchmark should be compiled with profile generation instrumentation enabled and setup for training data. The lit runner will merge the profile files using llvm-profdata so they can be used by the second compilation run.

Example:

Profile generation run using LLVM IR PGO:

% cmake -DTEST_SUITE_PROFILE_GENERATE=ON
-DTEST_SUITE_USE_IR_PGO=ON
-DTEST_SUITE_RUN_TYPE=train
../test-suite % make % llvm-lit .

Use the profile data for compilation and actual benchmark run:

% cmake -DTEST_SUITE_PROFILE_GENERATE=OFF
-DTEST_SUITE_PROFILE_USE=ON
-DTEST_SUITE_RUN_TYPE=ref
. % make % llvm-lit -o result.json .

To use Clang frontend’s PGO instead of LLVM IR PGO, set -DTEST_SUITE_USE_IR_PGO=OFF.

The TEST_SUITE_RUN_TYPE setting only affects the SPEC benchmark suites.

Cross Compilation and External Devices

Compilation

CMake allows to cross compile to a different target via toolchain files. More information can be found here:

Cross compilation from macOS to iOS is possible with thetest-suite/cmake/caches/target-target-*-iphoneos-internal.cmake CMake cache files; this requires an internal iOS SDK.

Running

There are two ways to run the tests in a cross compilation setting:

% ninja
% ninja rsync
% llvm-lit -j1 -o result.json .

Running the test-suite via LNT

The LNT tool can run the test-suite. Use this when submitting test results to an LNT instance. Seehttps://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suitefor details.

Running the test-suite via Makefiles (deprecated)

Note: The test-suite comes with a set of Makefiles that are considered deprecated. They do not support newer testing modes like Bitcode orMicrobenchmarks and are harder to use.

Old documentation is available in thetest-suite Makefile Guide.