Catch2 icon indicating copy to clipboard operation
Catch2 copied to clipboard

Report TestType in XML reports for templated test cases

Open FoamScience opened this issue 3 years ago • 0 comments

Description

This tiny PR tries to allow XML reporter to add the actual TestType as an arg. to TestCase if it's a TEMPLATE_TEST_CASE. Proof of concept atm; if the idea attracts enough interest; I'll add it to other reports too as well as maybe generalizing this to arbitrary number of case properties.

Current behavior: Catch reports append the TestType to the case name; separated by a dash. Reason for this improvement: Want to see the actual failing TestType (if some fail and some pass) in my editor. But parsing the name is error prone (as I usually have dashes in case names, and there is no way to distinguish if it's a template test case - unless parsing the expression for TestType - which can be just mentioned there as a string value or variable ... you can see how this can get confusing quickly). Solution: As TestType is a property of the test case; adding it as a parameter to the corresponding TestCase tag avoids all confusion; No effect should be observed in regular test cases.

This quick test:

TEST_CASE( "Regular - test case") {
    REQUIRE( true );
}

template<class T>
bool isTrue() {
    return true;
}

template<>
bool isTrue<double>() {
    return false;
}

TEMPLATE_TEST_CASE( "Template - test case", "", double, int) {
    REQUIRE( isTrue<TestType>() );
}

results in this XML report (notice the testType="..." when it's a TEMPLATE_TEST_CASE)

<?xml version="1.0" encoding="UTF-8"?>
<Catch2TestRun name="test" rng-seed="3252414589" xml-format-version="2" catch2-version="3.1.1">
  <TestCase name="Regular - test case" filename="test.cpp" line="3">
    <OverallResult success="true"/>
  </TestCase>
  <TestCase name="Template - test case - double" testType="double" filename="test.cpp" line="17">
    <Expression success="false" type="REQUIRE" filename="test.cpp" line="18">
      <Original>
        isTrue&lt;TestType>()
      </Original>
      <Expanded>
        false
      </Expanded>
    </Expression>
    <OverallResult success="false"/>
  </TestCase>
  <TestCase name="Template - test case - int" testType="int" filename="test.cpp" line="17">
    <OverallResult success="true"/>
  </TestCase>
  <OverallResults successes="2" failures="1" expectedFailures="0"/>
  <OverallResultsCases successes="2" failures="1" expectedFailures="0"/>
</Catch2TestRun>

FoamScience avatar Nov 13 '22 18:11 FoamScience