Performance: archunit-junit5 may scan complete JDK for `@ArchTest`

[rweisleder]

Context

I'm currently developing a library with common ArchUnit rules. To test them, I evaluate them against different classpaths. Therefor I created a multi-module Maven project:

my-project
+--- my-rules
+--- tests-with-classpath1
+--- tests-with-classpath2
`--- tests-with-classpath3

The module my-rules contains my rules in src/main/java. The other modules have a dependency on my-rules and dependencies for test, but no src folder.

Description

In IntelliJ, when I select the module "tests-with-classpath1" and execute "Run 'All Tests'", the ArchUnitTestEngine tries to discover tests in the complete classpath, even in the JDK.

It's because IntelliJ calls the registered TestEngines with DiscoverySelectors.selectPackage(""). However, if at least one class is present in the module, IntelliJ uses the ClasspathRootSelector. (The different selectors look inconsistent, which may be a cause of a bug/missing feature in IntelliJ: IDEA-354486)

With the PackageSelector [packageName = ''], the ArchUnitTestEngine now scans the complete classpath including the JDK. On my machine, it takes about 30 seconds to discover all tests until the first gets executed.

The behavior and implementation of ArchUnitTestEngine looks plausible, but I would be positively surprised about any @ArchTest inside the JDK. I suggest to replace the implementation with org.junit.platform.commons.support.ReflectionSupport#findAllClassesInPackage which somehow does not scan the JDK.

Reproducer

I created an example to reproduce the issue: https://github.com/rweisleder/demo-test-engine

• "Run 'All Tests'" on execution-without-src takes about 30 seconds
• "Run 'All Tests'" on execution-with-main-class takes only a few seconds

[codecholeric]

Thanks for reporting this! I'm gonna take a look...

[codecholeric]

Just out of curiosity, what's the use of a module without any src folder? 🤔

[codecholeric]

I think the reason ArchUnit scans so much is because it has some pretty aggressive scanning to find classes in some corner cases (e.g. JAR files with missing entries). AFAIS JUnit Platform ReflectionSupport just uses classLoader.getResources("") which doesn't return any JDK URLs... I think in general it's fair to include JDK classes if you just say importPackage(""), but in the case where we use ArchUnit to discover test classes this is not optimal of course 😬 I need to see what the best solution there is, maybe just ignoring the JDK itself is good enough for now (I'm not sure if ClassLoader.getResources(..) actually has any other difference for standard cases 🤷)

[rweisleder]

Just out of curiosity, what's the use of a module without any src folder? 🤔

I have a test suite consisting of several JUnit tests and ArchUnit tests as my main artifact. To test these tests, I have several modules with different classpaths (mostly different versions of the same libraries) and check if the tests can be executed successfully with these classpaths.