Support Junit @Nested test classes

[ratcashdev]

Description

original ticket: https://issues.apache.org/jira/browse/NETBEANS-6041

Given a test class like:

import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;

public class SampleTest {
  @Test
  public void testMyMethod1() {
    System.out.println("write this");
  }
  @Nested
  class NestedClass {
    @Test
    public void testMyMethod2() {
      System.out.println("nested write 2");
    }
  }
  @Nested
  class NestedClass2 {
    @Test
    public void testMyMethod1() {
      System.out.println("nested write 1");
    }
    @Test
    public void testMyMethod3() {
      System.out.println("nested write 3");
    }
    @Nested
    class DoubleNestedClass3 {
      @Test
      public void testMyMethod4() {
        System.out.println("double nested write 4");
      }
      @Test
      public void testNextedException() throws Exception {
        throw new Exception();
      }
    }
  }
}

we have two major issues:

1. The "Test Results" window shows only some of the executed tests (in this specific case only 3)
2. using the "Run focused test method" does not work - a bad combo of class and method is provided to the test runner.

Did some root-cause analysis too. The issue with Tests not showing is caused by the fact that the whole testrunner support was made with the assumption that test suites are executed sequentially. This, unfortunately is not true with nested tests, where the execution may look like

• A$B.method1
• A.method2
• A$B.method3
• A.method4

Fixing this needs significant changes to TestSession and TestProgressListener which assume that only one suite can be running at any given time (and thus measures the elapsed time).

It works somewhat if tests are only on equal level, e.g.

• A$B.method1
• A$B.method2
• A$C.method3
• A$C.method4

If there was a method directly on A, like A.otherMethod, then the results would not be shown again.

Use case/motivation

Work with such classes without issues

Related issues

No response

Are you willing to submit a PR?

• [x] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow the Apache Software Foundation's Code of Conduct

[ratcashdev]

Applies to MAVEN projects too.

[ratcashdev]

PR is waiting for review. Please have a look on it.

[dstutz]

What happened to this?

[matthiasblaesing]

Based on the work from @ratcashdev I had a look at the problem and created #8664. The PR has a nightly build attached and it would be great if you could test if that fixes the problems.

[ratcashdev]

Thank you, awesome work. Will do as soon as I am back from vacation.

[ratcashdev]

Tested dev-build_8664 that I have downloaded from https://github.com/apache/netbeans/actions/runs/16453544666/artifacts/3590774053 . Extracted, and run with bin/netbeans, with a fresh .netbeans directory. Running a nested test fails with error. It can't find the method, because it passes maven a wrong method qualifier. The nested class is missing. CTRL+F6 executes only non-nested test methods. Maybe I am doing something wrong, but it looks like this build has zero support for nested classes.

[matthiasblaesing]

Testproject (Maven): nested-test.zip

I reran the build and downloaded the artifact from PR: https://github.com/apache/netbeans/pull/8664/checks / https://github.com/apache/netbeans/suites/42168507738/artifacts/3683192807. Then I did this to run netbeans:

matthias@enterprise:~/tmp$ unzip ~/Downloads/dev-build_8664.zip
Archive:  /home/matthias/Downloads/dev-build_8664.zip
 extracting: NetBeans-dev-dev-53ea447f90c0a42000362d34d156bb043f47dfd6-full.zip  
matthias@enterprise:~/tmp$ unzip NetBeans-dev-dev-53ea447f90c0a42000362d34d156bb043f47dfd6-full.zip
Archive:  NetBeans-dev-dev-53ea447f90c0a42000362d34d156bb043f47dfd6-full.zip
   creating: netbeans/
   creating: netbeans/ide/
   creating: netbeans/ide/bin/
   creating: netbeans/ide/bin/nativeexecution/
   ... CUT ...
  inflating: netbeans/licenses/WSDL-2004  
  inflating: netbeans/netbeans.css   
matthias@enterprise:~/tmp$ netbeans/bin/netbeans --userdir dummy-userdir --cachedir dummy-cachedir

Product version is reported as : Apache NetBeans IDE DEV (Build dev-53ea447f90c0a42000362d34d156bb043f47dfd6)

I decline settings import and I activate the "Java SE" support via Plugin Manager:

I open the test project and open PackagedTopLevelTest. I then press "STRG+F6". This results in the maven call:

cd /home/matthias/tmp/nested-test; JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 /home/matthias/tmp/netbeans/java/maven/bin/mvn -Dtest=test.PackagedTopLevelTest --no-transfer-progress process-test-classes surefire:test

This is correct as the whole file is to be run. With that I get (enlarged the bottom area a bit):

The output is correct, the non-nested method test.PackagedTopLevel#test4 is reported and so is the nested method test.PackagedTopLevelTest$NestedClass2$DoubleNestedClass3#test9. Enabling display for succeeding tests I get:

Invoking "Run test8 method" (a method in a nested class):

results in:

cd /home/matthias/tmp/nested-test; JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 /home/matthias/tmp/netbeans/java/maven/bin/mvn -Dtest=test.PackagedTopLevelTest$NestedClass2$DoubleNestedClass3#test8 --no-transfer-progress process-test-classes surefire:test

Still looks sane to me. Same procedure also works fine with gradle test project.

[ratcashdev]

Found the difference. I had a different test project and a different test class. My test project was using:

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.22.2</version>
      </plugin>

So I opened your project and tested using that.

Observation:

1. Hitting CTRL+F6 with version 2.22.2 of the maven-surefire-plugin executed only a single test (test4). FAIL.
2. Hitting CTRL+F6 with version 3.5.3 of the maven-surefire-plugin executed all the tests. PASS.
3. clicking Run test5 method on the left gutter resulted in the test properly executed. PASS

I then added a new test class:

package test;

import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;

public class SampleTest {
  @Test
  public void testMyMethod1() {
    System.out.println("write this");
  }

  @Nested
  class NestedClass {
    @Test
    public void testMyMethod2() {
      System.out.println("nested write 2");
    }
  }

  @Nested
  class NestedClass2 {
    @Test
    public void testMyMethod3() {
      System.out.println("nested write 1");
    }

    @Test
    public void testMyMethod4() {
      System.out.println("nested write 3");
    }

    @Nested
    class DoubleNestedClass3 {
      @Test
      public void testMyMethod5() {
        System.out.println("double nested write 4");
      }

      @Test
      public void testmyMethod6() throws Exception {
        throw new Exception();
      }
    }
  }
}

Here, trying to Run testMyMethod2 resulted in no tests executed. The maven output showed:

cd ~/Downloads/nested-test; JAVA_HOME=/usr/lib/jvm/default ~/Downloads/netbeans/java/maven/bin/mvn -Dtest=test.SampleTest#testMyMethod2 --no-transfer-progress process-test-classes surefire:test

Here, as it can be seen, the nested class was not handed over to maven at all. Same was observed when using Run again in the test results window after CTRL+F6. I am unable to explain why this does not work.

[matthiasblaesing]

Thanks for the test. Apparently I only tested with too complex samples. This works as soon as there is a non-public toplevel classes present in addition to the normal public top level. I'll create an update in the next few days. SourceUtils needs to be modified for that case.

[ratcashdev]

@matthiasblaesing what about the old version of the maven-surefire-plugin ?

[matthiasblaesing]

@ratcashdev we could add a version guard, but at this point in time I'm not interested in expanding the horizon of this PR. We have 2025, if developers want cool features, they need to check their project setup. If I remember correctly nested test are not only not reported on old surefire version, they are also not executed.

[ratcashdev]

If I remember correctly nested test are not only not reported on old surefire version, they are also not executed.

Are we absolutely sure? CTRL+F6 was working without issues in my PR branch with that version.

[matthiasblaesing]

If I remember correctly nested test are not only not reported on old surefire version, they are also not executed.

Are we absolutely sure? CTRL+F6 was working without issues in my PR branch with that version.

You are correct. Tested again with your branch and that worked. It works because you are explicitly enumerating all nested classes. I'll have another look.

[matthiasblaesing]

Are we absolutely sure? CTRL+F6 was working without issues in my PR branch with that version.

You are correct. Tested again with your branch and that worked. It works because you are explicitly enumerating all nested classes. I'll have another look.

So I had another look and retestet. With old surefire plugin and a build from PR #3995 I get:

cd /home/matthias/tmp/nested-test; JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 /home/matthias/src/netbeans/nbbuild/netbeans/java/maven/bin/mvn -Dtest=test.SampleTest,test.SampleTest$* --no-transfer-progress process-test-classes surefire:test
Scanning for projects...

--------------------------< test:TestNested >---------------------------
Building TestNested 1.0-SNAPSHOT
  from pom.xml
--------------------------------[ jar ]---------------------------------

--- resources:3.3.1:resources (default-resources) @ TestNested ---
skip non existing resourceDirectory /home/matthias/tmp/nested-test/src/main/resources

--- compiler:3.13.0:compile (default-compile) @ TestNested ---
Nothing to compile - all classes are up to date.

--- resources:3.3.1:testResources (default-testResources) @ TestNested ---
skip non existing resourceDirectory /home/matthias/tmp/nested-test/src/test/resources

--- compiler:3.13.0:testCompile (default-testCompile) @ TestNested ---
Recompiling the module because of changed source code.
Compiling 3 source files with javac [debug release 17] to target/test-classes

--- surefire:2.22.2:test (default-cli) @ TestNested ---

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running test.SampleTest
write this
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 s - in test.SampleTest

Results:

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time:  2.306 s
Finished at: 2025-08-06T21:01:32+02:00
------------------------------------------------------------------------

The maven output already shows only one test ran. The log also shows that the branch was active, as else the test selection would not have been test.SampleTest,test.SampleTest$*.

I also updated the PR with a fix for the problem running test.SampleTest$NestedClass#testMyMethod2. Updated test build is available from checks page / directly.

@ratcashdev would you please retest?

[ratcashdev]

First, my branch, my test-project, with the old surefire version:

cd ~/Development/maven-test/mavenproject1; JAVA_HOME=/usr/lib/jvm/java-21-openjdk /usr/share/apache-netbeans/java/maven/bin/mvn -Dtest=io.test.maven.SampleTest,io.test.maven.SampleTest$* process-test-classes surefire:test

with the test results looking as follows:

Executing the same test (copy pasted to your test project), with the old surefire version results in the outcome as you described above (Nested classes not executed).

Here's my test project zipped: mavenproject1.zip

Did some further investigations. At the end it came down to the combination of jupiter-engine and surefire.

• Surefire 2.22.2 with version 5.11.3 (explicitely declared in the POM) works nicely. If 5.13.x is explicitly declared in the POM, it does not discover any tests. If in the POM I rely only on API and PARAMS package of jupiter (like in your project), then it does not execute nested tests.

• Using surefire 3.5.3 with jupiter 5.13 works, maven executes all the tests. It has the problem the Test Results window sometimes not picking up everything, but this is in my branch yet. This you already fixed.

Maybe we should add a warning of the invalid combinations in the POM?

[sephiroth-j]

Hey guys, I just wanted to let you know that there is a bug in surefire-plugin v3.5.3 that causes @Nested test classes to not be reported correctly (GH issue apache/maven-surefire#2601). If a test is not reported, try using the previous version 3.5.2.

[matthiasblaesing]

@ratcashdev I would not play whac-a-mole games with plugin versions. Before this change the IDE tried to add junit4 dependencies to a junit5 project because the plugin detection was not correct and thus a feature became an issue. As I already said the IDE now does the right thing. If the nested tests are executed by surefire, it will report them and if not, it will not do so.

To get some kind of end of this discussion. Is there a show stopper from your POV here? If not, I plan to do the squash and merge shortly after release of NB27.

[ratcashdev]

Testing dev build 8664

• surefire 3.5.3: all ok
• surefire 3.5.2: bad tree structure in the Test Results window.
• surefire 2.22.2 with jupiter 5.11.3 not working correctly with this build, only top-level tests are executed. This was working with my build.

[matthiasblaesing]

Thanks. Surefire 3.5.2 created wrong reports. The grouping in the reports does not match the contained classes. We could detect the broken condition, but then users could just go to a current version.

[matthiasblaesing]

I conclude, that I'll merge once NB27 is out.