Add new error type

[valegagge]

Working on the tests, I realized that in some error situations, I would not only stop the current test, but avoid that the framework continues to launch tests of the same suit.

Therefore I propose to add a new error type in the framework for satisfy this need. we could call it RTF_ESSERTER_SUIT_ERROR or RTF_ASSERT_CRITICALERROR.

Anyone has any suggestions?

[apaikan]

I can see the point :+1:

The question then arises: "If test B depends on test A, does A also depend on B?" in other words, if we reorder the tests execution in a suit, does the result change?

• If YES, then probably test A changing the state/configuration of the system (in our case the robot) without putting back into its initial state. For example, if test A moves the arm, it should also put it back to the initial configuration.
• If NO, then the configuration (setup) to run the tests from a suit is not prepared. A test case (or a set of tests) require specific setup (fixture). This fixture can be easily implemented as a plug-in (see the example here https://github.com/robotology/robot-testing/blob/master/examples/fixture-plugin/MyFixManager.cpp) and put into a test suits. The job of this fixture plug-in is to check whether the condition for running the tests are hold. The testrunner check whether the fixture is okay before launching next test from a suit.

For now the scope of the RTF_ASSERT_FAIL, RTF_ASSERT_ERROR is the current test case in which the macros have been called; We can extend these macros to work in wider scope such as suit or the test runner as you suggested.

@valegagge would you provide an example of your current experience to help me understand better the situation :)

[apaikan]

I have added an example of using a customized fixture manager with the test suit.

• See the example here: https://github.com/robotology/icub-tests/tree/master/example/cpp
• and here is the example (https://github.com/robotology/icub-tests/blob/master/example/suit/test_suit.xml) how a fixture manager can be used withing a test suit:

<?xml version="1.0" encoding="UTF-8"?>

<suit name="Test Suite">
    <description>A Suit example</description>
    <environment></environment>
    <fixture param="--probability 0.5"> ExampleFixture </fixture>

    <!-- Test cases -->
    <test type="dll" param="--name Example1"> ExampleTest </test>
    <test type="dll" param="--name Example2"> ExampleTest </test>
    <test type="dll" param="--name Example3"> ExampleTest </test>
    <test type="dll" param="--name Example4"> ExampleTest </test>

</suit>

• The suit include a fixture manager (i.e., ExampleFixture) and multiple instances of the ExampleTest. Before running each instance of the tests cases, the testrunner calles the ExampleFixture::check() method. This method is responsible to check whether the condition (fixture) to run the test cases is fine or not. In this specific example, it randomly return true/false Value. In case of ExampleFixture::check() = true , the testrunner launch the next test case. IF not, it restart the fixture by calling the ExampleFixture::setup() and checking it again. If this time, the fixture is fine it launch the test case otherwise proceed with the next test cases from the suit and so on.
• This is one way to solve the issue. If different test cases require different fixtures, then they should be divided in different suits. Alternatively, each test case can check for its own condition (fixture). That is why each test case has its own "TestCase::setup()" method.

[valegagge]

My situation is following: I created a suit that contains joint limits test for each part of robot, so the test is the same but with different config file (.ini). The test moves the part in a home position, then checks limits moving joints individually and then moves all joints in home position again. If one of joint can't reached home position I would like to stop entire test suit, in order to avoid that a joint, moved by next test, beats the joint not in home position.

I think the solution of adopt fixture is a good way to solve my problem, but currently it is not applicable because it needs to re-write a little some tests and now I have not time enough.

I will use fixture for next test implementaion, but currently I can use "exit()" or a RTF function/macro. I think use a RTF function/macro is a cleaner way than exit(), but it needs requires a your intervention. @apaikan What is your opinion? If you think a new RTF function/macro for exit from testsuite is useless, I'll use "exit()" function.

[lornat75]

I think it may be useful to have a way to stop further tests on failure of some tests, especially for safety reasons.

One concern may be that one may still wish to tests some components even if others fail (e.g. testing cameras even if tests on the motors failed). This seems to be a peculiar issue of our application scenario due to the fact that we are testing a physical system (rather than a software component) which is not easy to reset.