Tests fail to run for UWP projects with Minimum Versions set to RS2 (15063) or earlier - Regressed with TestAdapter 2.2.1

[MSLukeWest]

Description

Attempting to run tests for a UWP unit test project with Minimum Version set to RS2 (15063) or earlier fails with an error about not being able to load System.Xml.ReaderWriter. This regressed starting with MSTest.TestAdapter version 2.2.1. It reproes both in and outside of Visual Studio.

Steps to reproduce

1. Create a C# UWP test project. This brings up a wizard where you select two values. You can leave "Target Version" set to the default, but for "Minimum Version" change it to "Windows 10 Creators Update (10.0; Build 15063)". Anything older will repro the issue as well.
2. Manage NuGet Packages for the project and update the version used for MSTest.TestAdapter to 2.1.2.
3. Build the project and run all tests, it should work fine.
4. Go back to Manage NuGet Packages and update the version used for MSTest.TestAdapter to 2.2.1 or anything newer.
5. Rebuild and run all tests again.

Expected behavior

The test should pass.

Actual behavior

The test fails to run, with the following error listed in the Test pane of the output window:

An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.Xml.ReaderWriter, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

Environment

• Operating system: Win10, repro'd on various versions between RS5 and 21H1
• Build version of vstest.console: Repro'd with 16.10.0-release-20210520-03 and 16.11.0-release-20210626-04
• Package version of MSTest.TestAdapter: 2.2.1 - 2.2.5

[MSLukeWest]

Debugging observations:

In the working case Framework assemblies (System.*) are being loaded from the CoreFramework app directory, for example System.Xml.ReaderWriter.dll is loaded from "C:\Program Files\WindowsApps\Microsoft.NET.CoreFramework.Debug.2.2_2.2.29301.2_x86__8wekyb3d8bbwe\System.Xml.ReaderWriter.dll".

In the failing case these assemblies get loaded from the app layout directory, for example "C:\Users\lukewest\source\repos\UnitTestProject30\UnitTestProject30\bin\x86\Debug\AppX". And while System.Xml.ReaderWriter.dll exists in this directory, it's version is 1.0.24212.1. In the working scenario it's 4.6.29301.02.

Here's the full call stack of the exception, I have a dmp available on request:

Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Discovery.AssemblyEnumerator.DiscoverTestsInType(string assemblyFileName, string runSettingsXml, System.Reflection.Assembly assembly, System.Type type, System.Collections.Generic.List<string> warningMessages) Line 207	C#
 	Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Discovery.AssemblyEnumerator.EnumerateAssembly(string assemblyFileName, out System.Collections.Generic.ICollection<string> warnings) Line 111	C#
 	Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Discovery.AssemblyEnumeratorWrapper.GetTestsInIsolation(string fullFilePath, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunSettings runSettings, out System.Collections.Generic.ICollection<string> warnings) Line 121	C#
 	Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Discovery.AssemblyEnumeratorWrapper.GetTests(string assemblyFileName, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunSettings runSettings, out System.Collections.Generic.ICollection<string> warnings) Line 58	C#
 	Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.DiscoverTestsInSource(string source, Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging.IMessageLogger logger, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestCaseDiscoverySink discoverySink, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IDiscoveryContext discoveryContext) Line 61	C#
 	Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(System.Collections.Generic.IEnumerable<string> sources, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext runContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle frameworkHandle, Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.TestRunCancellationToken cancellationToken) Line 106	C#
 	Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll!Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(System.Collections.Generic.IEnumerable<string> sources, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext runContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle frameworkHandle) Line 111	C#
 	Microsoft.TestPlatform.CrossPlatEngine.dll!Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.Utilities.LazyExtension<Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor, Microsoft.VisualStudio.TestPlatform.Common.Interfaces.ITestExecutorCapabilities> executor, System.Tuple<System.Uri, string> executorUriExtensionTuple, Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Adapter.RunContext runContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle frameworkHandle)	Unknown
 	Microsoft.TestPlatform.CrossPlatEngine.dll!Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(System.Collections.Generic.IEnumerable<System.Tuple<System.Uri, string>> executorUriExtensionMap, long totalTests)	Unknown
 	Microsoft.TestPlatform.CrossPlatEngine.dll!Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestsInternal()	Unknown
 	Microsoft.TestPlatform.CrossPlatEngine.dll!Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTests()	Unknown
 	Microsoft.TestPlatform.CrossPlatEngine.dll!Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.ExecutionManager.StartTestRun(System.Collections.Generic.Dictionary<string, System.Collections.Generic.IEnumerable<string>> adapterSourceMap, string package, string runSettings, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ClientProtocol.TestExecutionContext testExecutionContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ITestCaseEventsHandler testCaseEventsHandler, Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.ITestRunEventsHandler runEventsHandler)	Unknown
 	Microsoft.TestPlatform.CrossPlatEngine.dll!Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler.OnMessageReceived.AnonymousMethod__3()	Unknown
 	Microsoft.TestPlatform.CrossPlatEngine.dll!Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler..ctor.AnonymousMethod__18_1(System.Action action)	Unknown
 	Microsoft.TestPlatform.CoreUtilities.dll!Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue<System.Action>.SafeProcessJob(System.Action job)	Unknown
 	Microsoft.TestPlatform.CoreUtilities.dll!Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue<System.Action>.BackgroundJobProcessor()	Unknown
 	Microsoft.TestPlatform.CoreUtilities.dll!Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue<System.__Canon>..ctor.AnonymousMethod__12_0()	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.Tasks.Task.InnerInvoke()	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.Tasks.Task.Execute()	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.Tasks.Task.ExecutionContextCallback(object obj)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.Tasks.Task.ExecuteEntry(bool bPreventDoubleExecution)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.Tasks.ThreadPoolTaskScheduler.LongRunningThreadWork(object obj)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.ThreadHelper.ThreadStart(object obj)	Unknown
 	[Native to Managed Transition]	
 	kernel32.dll!@BaseThreadInitThunk@12()	Unknown
 	ntdll.dll!__RtlUserThreadStart()	Unknown
 	ntdll.dll!__RtlUserThreadStart@8()	Unknown

[MSLukeWest]

There's two possible workarounds, either:

1. Go to "Manage NuGet Packages" for your test project, and update MSTest.TestAdapter to version 2.1.2, or
2. In the project properties for your test project in the Application tab, change "Min Version" to "Windows 10 Fall Creators Update (10.0; Build 16299)" or anything newer

[MSLukeWest]

To help narrow down the regression I tried using each of the two preview releases that came between 2.1.2 and 2.2.1, but they both failed with other errors:

2.2.0-preview-20201126-03: Build failed during GenerateResource task:

error MSB3816: Loading assembly "C:\Users\lukewest.nuget\packages\mstest.testadapter\2.2.0-preview-20201126-03\build_common\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll" failed. System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. File name: 'Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type) at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext) at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments) at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg) at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType, Boolean mustBeInheritable, IList derivedAttributes, Boolean isDecoratedTargetSecurityTransparent) at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeAssembly assembly, RuntimeType caType) at System.Reflection.RuntimeAssembly.GetCustomAttributes(Type attributeType, Boolean inherit) at System.Attribute.GetCustomAttributes(Assembly element, Type attributeType, Boolean inherit) at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit) at System.Reflection.CustomAttributeExtensions.GetCustomAttribute[T](Assembly element) at Microsoft.Build.Tasks.ProcessResourceFiles.ReadAssemblyResources(String name, String outFileOrDir)

2.2.0-preview-20210115-03: Discovery failed:

[MSTest][Discovery][entrypoint\UnitTestProject8.exe] MSTestAdapter failed to discover tests in class 'UnitTestProject8.UnitTest1' of assembly 'entrypoint\UnitTestProject8.exe' because Could not load type 'Microsoft.VisualStudio.TestPlatform.ObjectModel.ManagedNameUtilities.ManagedNameHelper' from assembly 'Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'..

[MSLukeWest]

I found that this change that updated PlatformServices.Universal from using Microsoft.Internal.TestPlatform.ObjectModel v14.0.0 to Microsoft.TestPlatform.ObjectModel v16.9.0-preview-20201021-10 is what caused this issue. If I sync to the 2.1.2 build label and make just that change I repro this issue.

[Evangelink]

Hi @MSLukeWest,

I am slowly taking over MSTest.

Could you confirm this is still an issue for you? Given the delay, I am wondering if the version is still a matter/supported.

[MSLukeWest]

@Evangelink - While this issue can still be reproduced, it's not very impactful because most users don't need to run tests using projects with TargetPlatformMinVersion < RS3 (10.0.16299.0).

[Evangelink]

I did some investigation and indeed there were some implicit breaking change done by updating Test Platform. It's going to be too complex to do any fix on older versions because of all the other changes...

On MSTest v3, I have made it explicit that we are dropping some older target frameworks that are no longer supported (see https://github.com/microsoft/testfx/issues/1274#issue-1383430926).

I will discuss with management to see what options we could provide but I guess the only suitable at the moment is for you to stay on the older version that was still supporting UWP version you want to target.

[Evangelink]

After discussion with management we agreed on doing nothing as the issue is quite old and the number of impacted users is quite low.