When ClassInitialize fails only one test fails, others executed regardless

[cklutz]

Description

This is based on this StackOverflow question: https://stackoverflow.com/q/59624616/21567

When you have a test class with a ClassInitialize-Method and that method fails. Only one of the actual test methods is reported as failed. The other methods seem to execute regardless.

Steps to reproduce

• Create a .NET Core UnitTest project in Visual Studio 2019
• Add the following test class

    [TestClass]
    public class UnitTest1
    {
        [ClassInitialize]
        public static void ClassInit(TestContext tc)
        {
            throw new Exception("DIE IN INITIALIZE");
        }

        [TestMethod] public void TestMethod1() { Console.WriteLine(nameof(TestMethod1)); }
        [TestMethod] public void TestMethod2() { Console.WriteLine(nameof(TestMethod2)); }
        [TestMethod] public void TestMethod3() { Console.WriteLine(nameof(TestMethod3)); }
    }

Project File:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <IsPackable>false</IsPackable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.2.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.0.0" />
    <PackageReference Include="MSTest.TestFramework" Version="2.0.0" />
    <PackageReference Include="coverlet.collector" Version="1.0.1" />
  </ItemGroup>
</Project>

• run dotnet test

Expected behavior

All test methods are "failed" (i.e. not run, because the initializer method failed). That was also the behaviour with MSTest 1.3.0 / full Framework.

Actual behavior

Only the first test (here TestMethod1) is failed with the correct error message (that the initializer failed). Other test methods run anyway (output Console.WriteLine messages). Messages will possibly only be visible from VS Test Explorer because of https://github.com/Microsoft/vstest/issues/799.

Environment

• Windows 10
• Visual Studio 2019 16.3
• MSTest.TestAdapter 2.0.0
• MSTest.TestFramework 2.0.0

[thedefside]

I noticed the same issue today. VS 2019 .NET Framework 4.7.2 MSTest 2.0.0

[abhijitpai]

Seems like this behavior may have changed in https://github.com/microsoft/testfx/pull/577/ inadvertently.

[ravarnamsft]

We are seeing this behavior as well for .Net Core. This is making debugging very hard as a lot of tests fail later on because the setup was incomplete. We have to go and try finding which test kicked off the class initialize and see its logs. @parrainc and @AbhitejJohn is there a workaround or a fix planned for this?

[AbhitejJohn]

Tagging @Haplois to help.

[Evangelink]

Hey folks! I am reviewing this issue and managed to reproduce with the version indicated but it seems that the issue was fixed with 2.2.0 and later versions.

I will move forward and close this issue as fixed.