DataTestMethod skips DataRows that provide different data arrays, only keeping last

[sandersaares]

Steps to reproduce

[DataTestMethod]
[DataRow(0, new int[] { })]
[DataRow(0, new int[] { 0 })]
[DataRow(0, new int[] { 0, 0, 0 })] // Is executed
[DataRow(1, new int[] { 1 })]
[DataRow(1, new int[] { 0, 1 })] // Is executed
public void TestMethod1(int expectedSum, int[] buckets)
{
    Assert.AreEqual(expectedSum, buckets.Sum());
}

Expected behavior

Test is executed with 5 sets of arguments.

Actual behavior

Test is executed with only 2 sets of arguments (see comments in snippet).

Test discovery DOES see all 5 test cases:

========== Starting test discovery ==========
========== Test discovery finished: 5 Tests found in 450,8 ms ==========
========== Starting test run ==========
========== Test run finished: 2 Tests (2 Passed, 0 Failed, 0 Skipped) run in 69 ms ==========

Environment

Windows 11, Visual Studio 2022 v17.0.1, .NET SDK 6.0.100.

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.11.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.7" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.7" />
    <PackageReference Include="coverlet.collector" Version="3.1.0" />
  </ItemGroup>

[notcake]

I suspect that all DataRow tests with the same display name end up with the same GUID, and so only one survives:

https://github.com/microsoft/testfx/blob/57d2d00637e2bb3b5b9c4f9334a10ef2f6463146/src/Adapter/MSTest.CoreAdapter/ObjectModel/UnitTestElement.cs#L232-L252

Note line 247, where testCase.DisplayName is fed to TestIdProvider, which generates the GUID.

Untested, but I think this should be:

              if (this.TestMethod.DataType != DynamicDataType.None)
              {
                  idProvider.AppendString(testCase.DisplayName);
+                 foreach (string serializedData in this.TestMethod.SerializedData)
+                 {
+                     idProvider.AppendString(serializedData);
+                 }
              }

[Haplois]

That's possible solution @notcake, but it breaks compatibility.

@sandersaares, could you try our latest version, this issue should be fixed in it.

[sandersaares]

Latest version I see is TestFramework 2.2.10 and SDK 17.2.0 which still reproduces the issue. I can upgrade to some SDK 17.3.x prerelease, but this also changes nothing.

What exactly is the version number that should have the fix?

[Hejle]

I have the same problem, with this very simple Unittest, which should fail at execution 2, but passes because of the bug making it only run the first test. However if I run the test from the Commandline using dotnet test, it correctly fails. :)

[DataTestMethod]
[DataRow("")] //Test is executed
[DataRow(null)] //Test is ignored
public void TestNotNull(string input)
{
    //Assert
    Assert.IsNotNull(input);
}

For others who have the same problem, a simple workaround is setting the DisplayName in the DataRow:

[DataTestMethod]
[DataRow("", DisplayName = $"{nameof(TestNotNull)}_WithEmptyString")]
[DataRow(null, DisplayName = $"{nameof(TestNotNull)}_WithNullString")]
public void TestNotNull(string input)
{
    //Assert
    Assert.IsNotNull(input);
}

[Evangelink]

We are working on the fix as I write these lines. I just wanted to mention/keep track of the fact that doing a test with Microsoft.NET.Test.Sdk version 17.3.2 and MSTest version 2.2.10 I can correctly reproduce the issue in VS (only 2 tests appear) but result is correct in command line (5 tests appear).

[Evangelink]

Fixed through #1306