DateTime equality test fails after upgrading from 2.2.3

[sharkk121]

Description

Hello, after updating the version of MSTest.TestAdapter and MSTest.TestFramework from 2.2.3 to 2.2.10 I found that the test that used to pass, started failing. On version 2.2.3 the date, when passed into the method is an actual DateTime object. On 2.2.10, the date is serialized as string instead. What's strange is that a very similar test, but with an extra complex type included passes as expected.

While searching for potential breaking changes I only found the following in the release notes of version 2.2.5, but I'm not sure if it's related:

• https://github.com/microsoft/testfx/pull/864

Steps to reproduce

Simply create a new empty Unit Test project that targets .NET 6 and paste the following in the test file. Compare the results with packages MSTest.TestAdapter and MSTest.TestFramework installed at version 2.2.3 and 2.2.10.

    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        [TestDataSource]
        public void Test(object[] data)
        {
            Assert.AreEqual("string", data[0]);
            Assert.AreEqual(new DateTime(2022, 6, 2), data[1]);
        }

        [TestMethod]
        [TestDataSourceWithComplexObject]
        public void Test2(object[] data)
        {
            Assert.AreEqual("string", data[0]);
            Assert.AreEqual("Bob", ((TestPerson) data[1]).FirstName);
            Assert.AreEqual("Test", ((TestPerson) data[1]).LastName);
            Assert.AreEqual(new DateTime(2022, 6, 2), data[2]);
        }

        public class TestDataSourceAttribute : Attribute, ITestDataSource
        {
            public IEnumerable<object[]> GetData(MethodInfo methodInfo)
            {
                yield return new object[]
                {
                    new object[]
                    {
                        "string",
                        new DateTime(2022, 6, 2)
                    }
                };
            }

            public string GetDisplayName(MethodInfo methodInfo, object[] data)
            {
                return $"Custom - {methodInfo.Name} - {Guid.NewGuid()}";
            }
        }

        public class TestDataSourceWithComplexObjectAttribute : Attribute, ITestDataSource
        {
            public IEnumerable<object[]> GetData(MethodInfo methodInfo)
            {
                yield return new object[]
                {
                    new object[]
                    {
                        "string",
                        new TestPerson { FirstName = "Bob", LastName = "Test" },
                        new DateTime(2022, 6, 2)
                    }
                };
            }

            public string GetDisplayName(MethodInfo methodInfo, object[] data)
            {
                return $"Custom - {methodInfo.Name} - {Guid.NewGuid()}";
            }
        }

        public class TestPerson
        {
            public string FirstName { get; set; }

            public string LastName { get; set; }
        }
    }

Expected behavior

Both tests pass.

Actual behavior

Test 1 fails with an error. Test 2 passes.

Assert.AreEqual failed. Expected:<02.06.2022 00:00:00 (System.DateTime)>. Actual:<2022-06-02T00:00:00.0000000 (System.String)>.

Environment

Please share additional details about the test environment.

• Operating system
• Build version of vstest.console
• Package version of MSTest framework and adapter
• Other installed packages and their versions on the test project

• Tested on Windows Server 2019 and Windows 11
• 17.2.0-preview-20220401-07
• 2.2.10
• coverlet.collector (3.1.2), Microsoft.NET.Test.Sdk (17.2.0) both installed by default in an empty unit test project

[Haplois]

Value returned from GetDisplayName should be unique for each data. Can you fix that, and try again?

[sharkk121]

I've updated the sample code. Both DataSource attributes return only one row of data, so each test is run once. Unfortunately it made no difference.

2.2.10 2.2.3

It also makes no difference if I use [DataTestMethod] or [TestMethod].

[Haplois]

I am investigating the issue, meanwhile as a workaround, you can add this to your assembly.

[assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)]

[Evangelink]

Hey @sharkk121, sorry for the huge delay, your issue is next on my list. I am starting the investigation at the moment and expect to post some results early next week.

[Evangelink]

Alright so I did some progress in the investigation. MSTest is using DataContractJsonSerializer to serialize data. This serializer works well for primitive types and any type annotated with DataContract but not for other types. In case of you 1st attribute, you only use primitive types, so serialization happens and consider all items to be string while when you introduce the complex type, serialization fails so we don't "unfold" data leading to the real type being passed to the method and not the deserialized object.

I need to debug further to understand how I can move forward with doing a better serialization.

[Evangelink]

Hi @sharkk121, so the fix is going to be quite big as it requires changing the serializer used. I will postpone the fix for v4. In the meantime, you can use the trick described by @Haplois:

I am investigating the issue, meanwhile as a workaround, you can add this to your assembly.

[assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)]

[Evangelink]

Closing this issue as it will be handled as part of the more global ticket #1462