Serialization of tests causes the referece equality to fail when running via an adapter

[drognanar]

Reporting this on behalf of https://developercommunity.visualstudio.com/content/problem/603414/tests-fail.html

Here's an example test class. When running the theory test using an adapter it serializes/deserializes the xunit test case. After deserialization reference equality isn't preserved and a test fails when running from VS, even though variables have an equal value.

using System.Collections;
using System.Collections.Generic;
using Xunit;

namespace Some.Namespace
{
    public class RefEqTests
    {
        [Theory]
        [MemberData(nameof(TestData))]
        public void Theory(string a, string b)
        {
            Assert.Same(a, b);
        }

        public static IEnumerator<object[]> GetEnumerator()
        {
            yield return new object[] { "foo", "foo" };
            yield return new object[] { "bar", "bar" };
        }
    }
}

[bradwilson]

You are running up against an unfortunate set of issues:

1. If you allow pre-enumeration of theories during discovery, then we will serialize each individual data set as a unique test
2. The serialization process causes identity to be broken, since the serialized data cannot preserve identity across processes

This affects solely the VS runner, since it's the only runner that utilizes the current serialization process.

There are two workarounds:

If testing identity is critical

Turn off theory pre-enumeration (either globally via the preEnumerateTheories config entry, or adding DisableDiscoveryEnumeration=true to any data attribute that depends on identity of data elements, like your MemberData).

By turning off pre-enumeration, you'll end up with just a single test case for all of your test data (so when you run it, it will have several results, one for each data element). In this case, the data enumeration happens when the test is run, which means it happens in the correct process and should preserve identity).

If testing identity isn't critical

Use equality testing rather than identity testing (that is, use Assert.Equal instead of Assert.Same). This will require you to implement some kind of comparer for classes, since their default comparer is identity comparison; structs, on the other hand, have property value comparers provided for them automatically.

[RussKie]

I feel this change of behaviour will cause a lot of grief to our consumers.

[bradwilson]

This is not a change of behavior in xUnit.net. It's been there for probably 6 or 7 years.