maven-surefire (original) (raw)

Problem

JUnit 5 container-level failures (e.g., @BeforeAll / @AfterAll) are not handled correctly by Surefire.

FAILED containers produce <testcase name=""> due to missing method context
ABORTED containers do not emit any <testcase>, causing missing entries in XML reports

Solution

Assign fallback name "initializationError" when no method name is available for container failures
Emit testAssumptionFailure for ABORTED containers to ensure skipped testcases are generated
Preserve existing behavior for normal test methods and engine/root containers

Result

No empty testcase names
No missing testcases for container-level failures
Correct XML reporting for both FAILED and ABORTED scenarios

Tests

Updated RunListenerAdapterTest to cover:
- FAILED container case
- ABORTED container case

[x] I hereby declare this contribution to be licensed under the Apache License 2.0

Thanks for your contribution.
Would it be possible to an IT test for this as well.

olamy linked an issue

Mar 21, 2026

that may beclosed by this pull request

Thanks for the feedback.

Yes, I will add an integration test to cover this scenario and update the PR.

&& testExecutionResult.getStatus() != org.junit.platform.engine.TestExecutionResult.Status.SUCCESSFUL
&& testIdentifier.isContainer()
&& methodName == null) {
methodName = "initializationError";

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this a class constant?

Thanks for the feedback.

Yes, I will add an integration test to cover this scenario and update the PR.

Thanks a lot for this. Just a small comment.

sure

Thanks for the feedback.
Yes, I will add an integration test to cover this scenario and update the PR.

Thanks a lot for this. Just a small comment.

sure

Hi @olamy , i am checking why ci/cd fail . give me some time.

hi @olamy i fix the ci/cd failure . and test locally using mvn -B clean verify -Prun-its and build success.

would appreciate your review when you have time.
Thanks

thanks for fixing this!

to add another datapoint:
we are running our tests with <rerunFailingTestsCount>3</rerunFailingTestsCount> and recently added a test that deterministically fails in the teardown method:

  @AfterClass
  public static void tearDown() throws Exception {
     ...
  }

the surefire output looks like this:

[2026-03-27T09:10:32.374Z] [WARNING] Flakes: 
[2026-03-27T09:10:32.374Z] [WARNING] com.mycompany.MyTestClass.<beforeAll>
[2026-03-27T09:10:32.374Z] [ERROR]   Run 1: MyTestClass.tearDown:141 ? NullPointer
[2026-03-27T09:10:32.374Z] [ERROR]   Run 2: MyTestClass.tearDown:141 ? NullPointer
[2026-03-27T09:10:32.374Z] [ERROR]   Run 3: MyTestClass.tearDown:141 ? NullPointer
[2026-03-27T09:10:32.374Z] [ERROR]   Run 4: MyTestClass.tearDown:141 ? NullPointer
[2026-03-27T09:10:32.374Z] [INFO]   Run 5: PASS
[2026-03-27T09:10:32.374Z] [INFO]   Run 6: PASS
[2026-03-27T09:10:32.374Z] [INFO]   Run 7: PASS
[2026-03-27T09:10:32.374Z] [INFO]   Run 8: PASS
[2026-03-27T09:10:32.374Z] [INFO] 
[2026-03-27T09:10:32.374Z] [INFO] 
[2026-03-27T09:10:32.374Z] [WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1

so no errors are being reported and the jenkins junit report also shows no problems, but the single test method in the class is reported "passed" 4 times.

since this seems pretty serious, i would kindly ask for a new surefire release once this fix is in, thank you!

i just realized, my test is not junit5 based i.e. using @AfterClass from junit4.
makes me wonder whether the problem is not specific to junit5 ?
would it make sense to have a specific test for failures in test teardown i.e. @AfterAll and @AfterClass ? also cover junit4 ?

@XN137 ,

i think these are the test cases you are talking about

@MilanTyagi2004 is that a new test your are adding or an existing one (didnt find it on master)?
if the former yeah looks in the right direction, but it would mean this PR isnt specific about junit5 but all kind of setup/teardown errors?

this is new test i just added to make sure every thing is on right right direction

@olamy
could you please check these and review this.

@XN137 could you please tell me what i should do . should we create separate another pr for this.

@MilanTyagi2004 i am just a regular surefire user, so i can't tell you what you should do to be honest 😅

it also kind of depends on the findings with your added test.
if you realized that the problem of missing testcases not only happens on junit5 but also on junit4, then we should consider extending the scope of this PR.

alternatively, we should file a dedicated issue mentioning both junit4 and 5 and then this PR can keep the junit5 scope and we will need to follow-up with a fix and tests for the junit4 cases.

if your tests found no problems with junit4, you can ignore all my comments and we will need to investigate this separately.

what this PR is missing even if its only for junit5:

fix unrelated reformatting in surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java
add junit5 test that covers exceptions in @AfterAll.

@MilanTyagi2004 i am just a regular surefire user, so i can't tell you what you should do to be honest 😅

it also kind of depends on the findings with your added test. if you realized that the problem of missing testcases not only happens on junit5 but also on junit4, then we should consider extending the scope of this PR.

alternatively, we should file a dedicated issue mentioning both junit4 and 5 and then this PR can keep the junit5 scope and we will need to follow-up with a fix and tests for the junit4 cases.

if your tests found no problems with junit4, you can ignore all my comments and we will need to investigate this separately.

what this PR is missing even if its only for junit5:

fix unrelated reformatting in surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java

add junit5 test that covers exceptions in @AfterAll.

sure we can discuss the junit4 issues in another pr.

for RunListenerAdapter.java changes are needed to pass error to xml.

, and i understand but before adding Junit 5 test which cover @AfterAll i am waiting for the review from "olamy". Hope you understand.

well with master this should be handled the same way in case of junit 4 or 5 (if vintage engine is doing the right thing)
@XN137 ideally you can provide an it test for master branch to see how it goes
this could be different with 3.5.x branch though
but yeah definitely need another PR

@olamy is there any thing i nned to add in this pr.

XN137 mentioned this pull request

Mar 29, 2026

1 task

olamy added the bug

Something isn't working

label

Mar 30, 2026

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})