allure-framework
TestNg Setup/Before errors should NOT be separate test cases in report
Describe the bug Allure generates individual test case from testNG setup/before methods too if they break. Attached project to reproduce: bug_reports.zip
To Reproduce Steps to reproduce the behavior:
- unzip Attached project
- execute
mvn clean test - execute
cd targetandallure serve - Check generated report. (I attached screenshot too)
- It contains one additional test "myBeforeClass" which is broken. This confuses testers because 1-the number of tests may be varying in the test class 2-it links a method which may be totally unknown for them (if its located in parent or static class).
- 3-Also It does not add too much value because the same stack can be found linked to the "real" test.
Expected behavior I would NOT expect to see the setup/before hook's excpetion as independent tests in the report. I suggest deleting the onConfigurationFailure() method from io/qameta/allure/testng/AllureTestNg.java. Follow steps to verify: 11. Open attached project 12. comment or delete method onConfigurationFailure() in io/qameta/allure/testng/AllureTestNg.java 13. repeat steps 2,3 14. See that in this case only the real broken test is shown in the list. And we can also find the error starting from this.
BTW: the comment in code also indicates that configuration error should not be separate tests
Screenshots
If applicable, add screenshots to help explain your problem.
5b. if uploaded to allureEE it generated strange orphan tests.
==expected images===
Desktop (please complete the following information):
- OS: [MAC]
- Browser [chrome]
- Version [e.g. 22]
Smartphone (please complete the following information):
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
Still actual, kind of the same issue raised here as well https://github.com/allure-framework/allure-java/issues/623/ Is there a chance that there will be a fix?
