xunit
The [Collection] attribute is ignored when running unit tests from contextual menu in the "code view".
To reproduce the bug, I have created the following unit tests class which contains 4 unit tests that must not be executed in parallel, for that I added the [Collection("Foobar")] attribute on the unit test class.
I added in this unit test class some assertions to check that unit tests are not executed in parallel.
namespace XunitParallelExecutionVsRunnerBug
{
[Collection("Foobar")]
public class UnitTest1
{
private static readonly bool[] TestRun = new bool[typeof(UnitTest1).GetMethods().Length];
[Fact]
public async Task Test1()
{
StartTestRun(1);
await Task.Delay(3000);
EndTestRun(1);
}
[Fact]
public async Task Test2()
{
StartTestRun(2);
await Task.Delay(3000);
EndTestRun(2);
}
[Fact]
public async Task Test3()
{
StartTestRun(3);
await Task.Delay(3000);
EndTestRun(3);
}
[Fact]
public async Task Test4()
{
StartTestRun(4);
await Task.Delay(3000);
EndTestRun(4);
}
private static void StartTestRun(int number)
{
lock (TestRun)
{
// Assert no other tests is running.
for (var i = 0; i < TestRun.Length; i++)
{
Assert.False(TestRun[i], $"The Test{i + 1} is running in parallel with the Test{number}...");
}
TestRun[number - 1] = true;
}
}
private static void EndTestRun(int number)
{
lock (TestRun)
{
TestRun[number - 1] = false;
}
}
}
}
When I run the unit tests from the Test Explorer, we can see that all the unit tests are executed sequentially.
But when I run the unit tests from the code with the contextual menu (when right click on empty code line between two unit test methods), the unit tests are executed in parallel:
Can you provide a repro project? I'm not able to repro this (it passes in both scenarios).
Please include the version of Visual Studio you're using and the results of running dotnet --info.
Thanks!
@bradwilson, I created a repo with a project to reproduce the bug: https://github.com/GillesTourreau/visualstudio.xunit-ParallelExecutionVsRunnerBug
An other scenario to reproduce the error:
- Clean the output of the project (right click on the project and select
Clean). - Run the unit test (right click between 2 test methods and select
Run Tests) => Normally the unit test are executed in sequential. - Remove the
Test4method (just cut the code in the clipboard). - Run the unit test (right click between 2 test methods and select
Run Tests) => Normally the unit test are executed in sequential. - Restore the code of the
Test4from the clipboard. - Run the unit test (right click between 2 test methods and select
Run Tests) => Now the unit tests execution are failed, because they are executed in parallel.
Also, you can increase the value of the Task.Delay() to 3 or 5 seconds to see the parallel or sequential execution (I use 1 second, because to reduce the time of the gif video recording).
About my CPU, I have a 12th Gen Intel(R) Core(TM) i9-12900H 2.50 GHz (14 Cores, 20 logical processors recognized by Windows).
I use the last VS version (17.11.5).
The result of dotnet --info command:
.NET SDK:
Version: 8.0.403
Commit: c64aa40a71
Workload version: 8.0.400-manifests.e99c892e
MSBuild version: 17.11.9+a69bbaaf5
Runtime Environment:
OS Name: Windows
OS Version: 10.0.22631
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\8.0.403\
.NET workloads installed:
Configured to use loose manifests when installing new manifests.
[aspire]
Installation Source: VS 17.11.35327.3
Manifest Version: 8.1.0/8.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.1.0\WorkloadManifest.json
Install Type: FileBased
Host:
Version: 8.0.10
Architecture: x64
Commit: 81cabf2857
.NET SDKs installed:
8.0.403 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]
Environment variables:
Not set
global.json file:
Not found
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
This is still a problem, and has very bad impact especially when running integration tests with db connections, as it causes deadlocks.
This is, unfortunately, not something that we can fix.
What's happening here is that Test Explorer is launching two independent instances of the test project and running them in parallel, asking each to run a subset of the tests. This causes the tests to run in parallel because neither instance of the test project knows about the other.
To the best of my knowledge, the only way I'm aware of to prevent this behavior is to create a .runsettings file and see MaxCpuCount to 1: https://learn.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=visualstudio#the-runsettings-file