Proposal for a dedicated test suite for the parallel frontend (original) (raw)

Currently, the parallel front-end tests are included by UI tests. However, the tests of the front-end compiler should have an independent test set to verify compilation processes and results.

Context: rust-lang/rust#132051 and rust-lang/rust#143953

Especially, due to errors that may occur randomly under parallel compilation, there are some situations we face:

Crashes: repeatedly execute to ensure no bugs.
Deadlocks: interrupt it when a deadlock occurs.
Diagnostic changes: check if it won’t change diagnostic contents, even if the sequence of the messages has been altered under a parallel environment.
Differences in codegen: verify consistency of codegen between the parallel and serial compiler.

That means requiring a new execution and verification setup, which are different from UI tests.

And so on, we need to separate the parallel front-end tests into a new test suite. Moreover, many reported issues and cases ain’t collected into those tests, though some have been solved.

This MCP proposes to add the parallel front-end test suite and determine its mechanism.

Test coverage of the suite

The suite contains the cases that have been solved or are not reproducible so far. And those issues that are unsolved at the moment will be added once they are addressed.

Step 1: Enable ICE and deadlock tests on local

At the moment, a lot of reported issues with ICEs have been solved and thus they should be added into this suite.

For the test method, repeatedly execute a test case and check compiler crashes to try to find more problems under parallel compilation.
A detective to specify an iteration count for some tests, which need to be verified through more trials.
An environment variable to enable it artificially, and it won’t run on CI.

This means we can verify the parallel frontend on local, especially while changes get involved in it.

Step 2: Enable ICE and deadlock tests on CI

Compared to step 1, there are somethings we wanna do:

An iteration that is sufficient but doesn’t significantly affect CI time for every test
For possible deadlocks during tests, a timeout mechanism is necessary to prevent unlimited waiting.(The most easy way is to kill CI process if timeout)

There are two steps to enable tests on CI:

In the early step, reviewers can manually run the CI via a bot command on specific PRs. Additionally, someone from the working group of the parallel compiler will periodically (for example, weekly) submit a draft PR specifically for testing.
Once we have sufficient experience to ensure it won't bother daily PRs, we can integrate it into the try tasks that are automatically triggered when merging a (rollup) PR.

Step 3: Enable the parallel frontend for existing tests

At this step, we can enable the parallel frontend for the existing ui test suite to compare diagnostic messages. Also, the codegen and assembly test suite will contain tests to compare builds generated by the serial and parallel compiler.

This test suite is intent to assist us stabilize the parallel front-end feature next.

Mentors or Reviewers

@SparrowLii

Process

The main points of the Major Change Process are as follows:

File an issue describing the proposal.
A compiler team member who is knowledgeable in the area can second by writing @rustbot second or kickoff a team FCP with @rfcbot fcp $RESOLUTION.
- Refer to Proposals, Approvals and Stabilization docs for when a second is sufficient, or when a full team FCP is required.
Once an MCP is seconded, the Final Comment Period begins.
- Final Comment Period lasts for 10 days after all outstanding concerns are solved.
- Outstanding concerns will block the Final Comment Period from finishing. Once all concerns are resolved, the 10 day countdown is restarted.
- If no concerns are raised after 10 days since the resolution of the last outstanding concern, the MCP is considered approved.

Managed by @rustbot—see help for details.