Better control of test-execution order · Issue #6266 · rust-lang/cargo (original) (raw)

Cargo currently does not take too much care about what order tests are executed. cargo_test::compile_tests() places them in order of (PackageID, TargetKind, Name). We could do better than that by ordering test execution by

If the test has failed before
If the test has been executed at all
(PackageID, TargetKind)
The time it took the last time to execute this test, in descending order
Name

This has some benefits:

Failing tests are executed first, without the need to stop cargo watch and filter the test we just broke.
Tests which have not been executed at all (due to fail-fast) are executed as soon as possible, so we get a least one reading
Slow tests are scheduled first, so their execution overlaps as long as possible, avoiding having just one or two slow tests be scheduled at the end and consuming just one cpu.

Notice that if the testsuite completes successfully, the ordering almost returns to current behaviour, with only latency-scheduling taking precedence over Name.

To do this, we'd need to persists some information to disk. We could do this by writing a "marker"-file somewhere in /target for every test (consider "target/testruns/test-hash(...)") where the files' ctime is the time the test were last started and the files' mtime is the time the tests were last completed successfully. We can reconstruct the above ordering from that.
Alternatively we serialize a single struct to disk.
Also notice that the ordering we derive from this information may be wrong or outdated (e.g. by changing test-bodies), yet this does not cause the testsuite to execute incorrectly as a whole and Is Only Wrong Once™