TestCase with DataRow attribute with null parameter ignored

[daangruijters]

Description

When using MSTest.TestAdapter version 2.2.4 or higher, DataRow testcases with a null parameter are not discovered or executed. When using 2.2.3, all 3 testcases are discovered.

[TestMethod]
[DataRow(null)]
[DataRow("")]
[DataRow(" ")]
public void TestMethod(string parameter)
{

}

Steps to reproduce

Create a test project targetting .NET5 or .NET6, and make sure the version of the package MSTest.TestAdapter is 2.2.4 or higher. Add test method as seen above

Expected behavior

3 testcases should have been executed, but only the last two are executed

Actual behavior

Only the testcases with "" and " " as paramter are executed

Environment

MSTest.TestAdapter 2.2.4 or higher

[touchofevil-dev]

To further the use case this doesn't seem to be anything with null necessarily but the top most attribute seems to be skipped.

It seems the tests are being executed in the reverse order of attributes (bottom one is the first input), and it misses out on the top most DataRow input. eg. In below it will miss out on calling with the null as input.

[TestMethod]
[DataRow(null)]
[DataRow("")]
[DataRow(" ")]
public void ValidateString(string? input)
{
    Assert.IsTrue(string.IsNullOrWhiteSpace(input));
    Assert.IsNotNull(input); // This should have failed for null input
}

[Haplois]

You need to set TestDataSourceDiscovery to DuringExecution in your test assembly as described here.

[marnilss]

You need to set TestDataSourceDiscovery to DuringExecution in your test assembly as described here.

@Haplois, surely that is a temporary workaround, not a solution? So why close this issue?

Even though you have changed the implementation of how discovery of tests are done, shouldn't the new one also be fixed to return the expected amount of testcases? Instead of saying - run the old behavior if you want something that works!!! Because I can't see, nor has anyone else pointed it out, anything wrong with the sample provided by the TS - so it should work even with new discovery method.

[MikeRosoft]

I can confirm that the error is still occurring in version 2.2.7. The behavior is apparently the following: If any of the [DataRow] attributes have value null (or default value for the given type), then the first row will be silently skipped. (That means that null will be passed into the unit test, as long as it is not in the first position; but the first row is skipped.)

NuGet package references:

<PackageReference Include="coverlet.msbuild" Version="2.9.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />
<PackageReference Include="MSTest.TestAdapter" Version="2.2.7" />
<PackageReference Include="MSTest.TestFramework" Version="2.2.7" />
<PackageReference Include="coverlet.collector" Version="3.1.0" />

Updating to MsTest 2.2.10 (and the rest of unit test packages) did not fix the error.

[MikeRosoft]

Update: apparently the error only occurs when running from Visual Studio; when running from the command line the tests seem to be executed correctly.

[Evangelink]

Hi @MikeRosoft, thanks for the update. We will look into this issue.

[notcake]

Sounds like a duplicate of #1016, where test cases get ignored when they have the same display name.

[juharris]

Thanks @notcake! Doing

[DataRow(null, DisplayName = "null input")]

got the test to run in VS for me.

[Evangelink]

Fixed through #1306