microsoft
(Assembly/Class)(Initialize/Cleanup) methods as independent entries in Test Explorer
Summary
Consider AssemblyInitialize, AssemblyCleanup, ClassInitialize and ClassCleanup as specific "test case" so that they are displayed as specific items in Visual Studio Test Explorer. This would ease understanding issues on these steps.
Background and Motivation
Before MSTest v3, errors or warnings during the (Assembly/Class)(Initialize/Cleanup) were shown only in the logs (except for specific cases). Starting with MSTest v3, AssemblyInitialize messages/errors are associated with 1st test run for given assembly, AssemblyCleanup to last test run for the assembly, and similarly, ClassInitialize to 1st test run for given class and ClassCleanup for last test run for given class.
This new behaviour allows to not miss any issue but is not ideal, especially in a parallel run context where 1st/last test run could differ, causing different test to be marked as failed.
Proposed Feature
Consider these methods as specific, non-runnable, entries.
Alternative Designs
None
AB#2051873
@Evangelink We're trying to upgrade to 3.0.x and this is causing issues for us. We start and stop SQL instances in our Assembly Initialize and Cleanup functions using SqlLocalDb via https://github.com/martincostello/sqllocaldb. Sometimes when we try to delete the instance it does not work because the "specified instance is currently in use" or other issues. We've done a good amount of debugging on our end and we believe it's a bug in SqlLocalDb where even trying to force delete an instance does not work as expected - https://github.com/martincostello/sqllocaldb/issues/74. Now that the errors cause failures in the test runs instead of just logging it would be too great of a flaky test problem to upgrade the package. Is there an option for Assembly Init and Cleanup to function similar to 2.0.x?
Hi @Cjewett,
I am not aware of the behavior before MSTest v2.2.8 so I will need to investigate a bit.
As far as I understand, it seems normal to me that of the init or cleanup fails it causes an error. I'm wondering if it's not possible for you to do a try/catch inside the init/cleanup where you expect the exception to happen.
@Evangelink The try/catch workaround is what I've done for now but I was hoping there was some assembly attribute trickery similar to TestIdGenerationStrategy or TestDataSourceDiscovery. Understood if it doesn't exist though.
@Cjewett, there is nothing I am aware of but I need to understand what changed (and how) to decide if we simply document this change of behaviour or if we want to give some option.
✅ Successfully linked to Azure Boards work item(s):
