If one test fails, can we not continue running remaining tests either in a test class or belonging to a certain category?

[letaoana]

[CharliePoole]

That's what nunit does. Each test runs independently... what are you seeing?

[rprouse]

Also, how are you running your tests? What test runner are you using?

[letaoana]

@CharliePoole I am running tests belonging to a specific category. Say there are 13 tests with "Fruits" as a category, for example, and a test that is run first fails, then I want the outstanding 12 tests to not run at all. @rprouse I am running my tests using the NUnit Test Adapter as part of a build that runs in a Continuous Integration pipeline process.

[rprouse]

I think we all misread "can we not continue", sorry.

The only way that I can think of to prevent other tests from running is to set a boolean flag variable, toggle it if a test fails and then use Assume.That to check before you run your tests. Something like this if it is one class,

[TestFixture]
public class StopTestRuns
{
    bool _stopTests;

    [SetUp]
    public void SetUp()
    {
        Assume.That(_stopTests, Is.False);
    }

    [TearDown]
    public void TearDown()
    {
        if(TestContext.CurrentContext.Result.Outcome.Status != NUnit.Framework.Interfaces.TestStatus.Passed)
        {
            _stopTests = true;
        }
    }

    [Test]
    public void TestOne()
    {
        Assert.Pass("This passed");
    }

    [Test]
    public void TestTwo()
    {
        Assert.Fail("This failed too");
    }

    [Test]
    public void TestThree()
    {
        Assert.Fail("This failed three");
    }
}

Which results in this test run in VS,

Notice TestTwo didn't run because TestThree ran before it and failed.

[letaoana]

@rprouse I want my tests to do the same as you have shown in your demo above, however, these tests are not in the same class. I have automated acceptance tests which are grouped into different classes as per requirement and have tagged all of them with "Acceptance" category. In our CI pipeline, there is a task that is solely dedicated to run only the acceptance tests and only if they have all passed that the next task to run the regression tests will be kicked off.

[jnm2]

We could provide a TestContext.Exit() in a similar vein to Environment.Exit() and Application.Exit() for those that would rather lose any remaining information in an effort to see the previously-completed results sooner and stop taking up CI resources sooner. I don't know if we should encourage that, though!

[OsirisTerje]

@letaoana What if you set the stopTest flag as a public static property ?

[letaoana]

@OsirisTerje I will have to test that approach first and feedback. @jnm2 I merely want to avoid wasting build time running all the acceptance tests even though one of them had failed. Simple, if 1 acceptance test fails and irrespective of how many from the same category are yet to be run, I want the runner to stop and fail the build immediately.

[jnm2]

I mean, I wouldn't be hard to persuade, but it would have to be popular with the @nunit/framework-team (or highly demanded) before I felt comfortable pushing for an NUnit .Exit().

[CharliePoole]

We should note that the framework already supports stopping on the first error. This does not allow for fine tuning, such as skipping other tests in a class or category on failure, but it works well in situations where you want a single failure to stop the entire run.

The console accesses this using the --stoponerror option, which sets the "StopOnError" package setting to true. The adapter could, of course, expose it in the .runsettings file if desired.

Doing more than this, however, seems to be very similar to the implementation of test dependencies as I've always defined them and I would think it calls for a complete design first.

[CharliePoole]

@letaoana @rprouse I saw the "not", but I took it as in "can we not have ice cream for dessert?" which generally would mean the speaker wants ice cream... as opposed to "can we NOT have spinach for once?" 😸

[letaoana]

@OsirisTerje I implemented your suggestion and it works! 😄 @CharliePoole 🤣 You are right to point out that in so doing, it would cause the runner to skip other tests with different categories (or without) that are in the same class. If I was to move all the acceptance tests to a single class, I feel like that would be costly considering that each would require a different setup.

[josuermz]

I think we all misread "can we not continue", sorry.

The only way that I can think of to prevent other tests from running is to set a boolean flag variable, toggle it if a test fails and then use Assume.That to check before you run your tests. Something like this if it is one class,

[TestFixture]
public class StopTestRuns
{
    bool _stopTests;

    [SetUp]
    public void SetUp()
    {
        Assume.That(_stopTests, Is.False);
    }

    [TearDown]
    public void TearDown()
    {
        if(TestContext.CurrentContext.Result.Outcome.Status != NUnit.Framework.Interfaces.TestStatus.Passed)
        {
            _stopTests = true;
        }
    }

    [Test]
    public void TestOne()
    {
        Assert.Pass("This passed");
    }

    [Test]
    public void TestTwo()
    {
        Assert.Fail("This failed too");
    }

    [Test]
    public void TestThree()
    {
        Assert.Fail("This failed three");
    }
}

Which results in this test run in VS,

Notice TestTwo didn't run because TestThree ran before it and failed.

You can add the Attribute [Order(1)] [Order(2)], put the attributes on the top of each test method in the specific order that you want them to run.