Catch2 icon indicating copy to clipboard operation
Catch2 copied to clipboard

Skipping last section runs the test another time without sections

Open martinzink opened this issue 2 years ago • 1 comments

Describe the bug If the last section is skipped then it seems that the TEST_CASE will run an additional time without going into any section, which can cause failures if the test contains common assertions after the sections.

Expected behavior Skipping the last section would end the TEST_CASE

Reproduction steps Steps to reproduce the bug.

TEST_CASE("Last Section skipped") {
  int foo = 0;
  SECTION("1") {
    foo = 1;
  }
  SECTION("2") {
    SKIP();
  }
  CHECK(foo != 0);
}

TEST_CASE("First Section skipped") {
  int foo = 0;
  SECTION("2") {
    SKIP();
  }
  SECTION("1") {
    foo = 1;
  }
  CHECK(foo != 0);
}

In the example above if the Last section is skipped the CHECK will fail, however if we change the order of the sections it will pass.

Platform information:

  • OS: macOS 13.4.1 22F82 arm64
  • Compiler+version: Apple clang version 14.0.3 (clang-1403.0.22.14.1)
  • Catch version: v3.4.0

  • OS: Arch Linux x86_64
  • Compiler+version: gcc 12.3.0
  • Catch version: v3.4.0

martinzink avatar Jul 20 '23 08:07 martinzink

Yeah, this is a known bug/issue. Exiting the last section in a TEST_CASE irregularly (throwing exception/SKIP/etc) will cause the test case to be entered again, to see if there is another section to execute.

This should be fixable in most (all?) cases, but would require significant rewrite of the section tracking logic.

The reason this works fine when you exit a non-last section is that you won't notice the extra rerun, as there is not-yet executed section to enter.

horenmar avatar Jul 22 '23 19:07 horenmar