microsoft
TestContext not Thread Safe
Description
The concept of the TestContext as a property of a class is not thread safe. This is especially true if it is made static, but is even true if it is class instance specific. It needs to be made an optional parameter to the test level functions to make it so. This bug is related to #385. Although they address slightly different problems, if you had to choose between the two to fix, I think this way of addressing the threading issues would be cleaner.
Steps to reproduce
Add the .cs file in the attached zip into a MSTest 1.3.0-beta2 project with MethodLevel scope parallelize functionality activated.
Expected behavior
The Test Initialize and Test Cleanup functions, at the very least, should be able to take a TestContext parameter. For Data Driven tests with attached data sources, the Test Method functions should also get one, as the DataRow property in a Class level TestContext variable is bound to get jumbled between the functions. If you get to the point of parallelizing a data driven function's test cases, this separation will become even more critical.
Actual behavior
Because the TestContext object is shared at the class level, the output from the two test functions and their respective Test Initialize and Cleanup functions gets jumbled.
@jchesshir , thanks for bringing this issue up. I agree that currently TestContext is not Thread Safe, & can have repercussions in case of method level parallelism. However the changes to do so are significant, hence marking this as an enhancement. I'll bring this issue up in internal team discussion, to see when can we pick this story.
Also in case you do get a chance to work on it, please feel free to pick it up 😃
I encountered this issue. TestContext is not thread safe. Please escalate this one. Eagerly waiting for this issue to be fixed as we've recently migrated from NUnit to this framework and encountered many failures and ended up in a major disappointment.
The issue can be replicated easily by running the following test methods in parallel with execution scope at MethodLevel.
After the execution, in the output of one or many test methods you can see its mixed up showing output from the many methods.
[TestMethod]
public void TestMethod1()
{
Function1("Test Method 1");
Function2("Test Method 1");
Function3("Test Method 1");
Function4("Test Method 1");
Assert.Fail();
}
[TestMethod]
public void TestMethod2()
{
Function1("Test Method 2");
Function2("Test Method 2");
Function3("Test Method 2");
Function4("Test Method 2");
Assert.Fail();
}
[TestMethod]
public void TestMethod3()
{
Function1("Test Method 3");
Function2("Test Method 3");
Function3("Test Method 3");
Function4("Test Method 3");
Assert.Fail();
}
[TestMethod]
public void TestMethod4()
{
Function1("Test Method 4");
Function2("Test Method 4");
Function3("Test Method 4");
Function4("Test Method 4");
Assert.Fail();
}
[TestMethod]
public void TestMethod5()
{
Function1("Test Method 5");
Function2("Test Method 5");
Function3("Test Method 5");
Function4("Test Method 5");
Assert.Fail();
}
[TestMethod]
public void TestMethod6()
{
Function1("Test Method 6");
Function2("Test Method 6");
Function3("Test Method 6");
Function4("Test Method 6");
Assert.Fail();
}
//Functions
public void Function1(string value)
{
Console.WriteLine($"From Function 1 : {value}");
}
public void Function2(string value)
{
Console.WriteLine($"From Function 2 : {value}");
}
public void Function3(string value)
{
Console.WriteLine($"From Function 3 : {value}");
}
public void Function4(string value)
{
Console.WriteLine($"From Function 4 : {value}");
}
@saparia If I switch Console.Writeline to TestContext.Writeline the example works for me
Starting test execution, please wait...
MSTest Executor: Test Parallelization enabled for source\repos\TestUnitTests\bin\Debug\TestUnitTests.dll (Workers: 8, Scope: MethodLevel).
X TestMethod2 [30ms]
Error Message:
Assert.Fail failed.
Stack Trace:
at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.HandleFail(String assertionName, String message, Object[] parameters)
at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail()
at TestUnitTests.UnitTest1.TestMethod2() in source\repos\TestUnitTests\UnitTest1.cs:line 27
Standard Output Messages:
TestContext Messages:
From Function 1 : Test Method 2
From Function 2 : Test Method 2
From Function 3 : Test Method 2
From Function 4 : Test Method 2
X TestMethod5 [31ms]
Error Message:
Assert.Fail failed.
Stack Trace:
at TestUnitTests.UnitTest1.TestMethod5() in source\repos\TestUnitTests\UnitTest1.cs:line 56
Standard Output Messages:
TestContext Messages:
From Function 1 : Test Method 5
From Function 2 : Test Method 5
From Function 3 : Test Method 5
From Function 4 : Test Method 5
X TestMethod6 [29ms]
Error Message:
Assert.Fail failed.
Stack Trace:
at TestUnitTests.UnitTest1.TestMethod6() in source\repos\TestUnitTests\UnitTest1.cs:line 66
Standard Output Messages:
TestContext Messages:
From Function 1 : Test Method 6
From Function 2 : Test Method 6
From Function 3 : Test Method 6
From Function 4 : Test Method 6
X TestMethod4 [30ms]
Error Message:
Assert.Fail failed.
Stack Trace:
at TestUnitTests.UnitTest1.TestMethod4() in source\repos\TestUnitTests\UnitTest1.cs:line 46
Standard Output Messages:
TestContext Messages:
From Function 1 : Test Method 4
From Function 2 : Test Method 4
From Function 3 : Test Method 4
From Function 4 : Test Method 4
X TestMethod3 [31ms]
Error Message:
Assert.Fail failed.
Stack Trace:
at TestUnitTests.UnitTest1.TestMethod3() in source\repos\TestUnitTests\UnitTest1.cs:line 36
Standard Output Messages:
TestContext Messages:
From Function 1 : Test Method 3
From Function 2 : Test Method 3
From Function 3 : Test Method 3
From Function 4 : Test Method 3
I think the issue raised in this ticket was fixed as a side-effect of some other fix. I cannot reproduce the issue on newer version of MSTest but I can repro on old versions.
I will move forward by closing this issue. Please feel free to ping if you are still facing the issue.
