Issue with custom JUnit 5 Engine and Surefire Reporting

[ph33rtehgd]

Affected version

3.0.0-M4 and later

Bug description

I have written a custom JUnit 5 engine to help me run ArchUnit tests across my projects using a common set of rules. Up until version 3.0.0-M3, this engine worked with no issues and reported failures normally like any regular test case as you'd expect. However starting in 3.0.0-M4 and later Surefire no longer reports test case failures from my custom engine. For example, I would call: TestExecutionResult.failed(error) in my failure path and this should fail, however my mvn test command will finish with a success status. The only solution I've found is to either:

1. Downgrade to 3.0.0-M3 or older of Surefire
2. In the case of a child test case failure, mark the root descriptor as failed as well. This lets me use the latest version of Surefire, but it goes against how root descriptor reporting is supposed to work in JUnit 5 (according to their documentation) and it also results in an extra failure being recorded (root failure and the actual child test case failure).

custom-junit-engine-test.zip

Attached above is a minimal set of projects that replicate the issue. "test-custom-junit-engine" is a very simple custom JUnit engine that simply has one test that always fails. "test-project-engine" is used to run the "mvn test" command. I've left the working Surefire plugin configuration commented out. You can run mvn test as is and see that no errors are reported, however if you change the Surefire version to 3.0.0-M3 and re-run mvn test, the test failure is properly reported.

I'm hoping there is something basic I am missing here, however given that it used to work I'm a bit skeptical. Please point me in the right direction if I'm doing something incorrect.