[17.6.0] No suitable test runtime provider was found for any source in this run.

[nohwnd]

Description

On AzureDevOps images windows-latest and windows-2022 you can run into No suitable test runtime provider was found for any source in this run. error.

This error is caused by additional dlls that were ignored in 17.5.0, being incorrectly passed on in 17.6.0.

To replicate this issue you need to be on 17.6.0 version of TestPlatform, and provide more dlls than what you are testing, for example by using the default assembly filter in AzureDevOps:

      **\*test*.dll
      !**\*TestAdapter.dll
      !**\obj\**

This filter will match .NET Standard and .NET Portable dlls, that are coming from your test adapter, and incorrectly pass them to be executed. In 17.5.0 those dlls would be silently ignored, but in 17.6.0 they fail to resolve the runtime provider and fail the run.

Workaround 1

Reduce your filter to only match the test dlls, and nothing more, for example if your test dlls use the following naming convention:

MyProduct.UnitTests.dll
MyProduct.IntegrationTests.dll

MyOtherProduct.Unit.Test.dll

You can use:

**\bin\**\*Tests.dll
**\bin\**\*Test.dll

Workaround 2

Use TestPlatform Installer task to install VSTest 17.5.0 (or 17.6.2), and instruct your vstest task to use it:

# Install VSTest 17.5.0
- task: VisualStudioTestPlatformInstaller@1
  inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '17.5.0'

- task: VSTest@2
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **\*test*.dll
      !**\*TestAdapter.dll
      !**\obj\**
    searchFolder: '$(System.DefaultWorkingDirectory)'
    # ------------  Use the version installed by installer -----------
    vsTestVersion: 'toolsInstaller'

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/visual-studio-test-platform-installer-v1?view=azure-pipelines

[nohwnd]

17.6.2 was published that solves this issue. The fix will be inserted into VS and dotnet sdk as soon as we can, but will take some time to propagate.

[ytidhar]

Since yesterday we got this error in many pipeline/releases OS - Windows Server 1016

2023-06-08T07:59:58.0103984Z ==============================================================================
2023-06-08T07:59:58.0104120Z Task         : Download Artifacts - External Build
2023-06-08T07:59:58.0104197Z Description  : Download artifacts from external VSTS/TFS build
2023-06-08T07:59:58.0104273Z Version      : 15.223.1
2023-06-08T07:59:58.0104321Z Author       : ms-vscs-rm
2023-06-08T07:59:58.0104924Z Help         : 
2023-06-08T07:59:58.0104994Z ==============================================================================
2023-06-08T07:59:58.1592706Z module.js:478
2023-06-08T07:59:58.1594539Z     throw err;
2023-06-08T07:59:58.1594674Z     ^
2023-06-08T07:59:58.1594733Z 
2023-06-08T07:59:58.1594839Z Error: Cannot find module 'vsts-task-lib/task'
2023-06-08T07:59:58.1595036Z     at Function.Module._resolveFilename (module.js:476:15)
2023-06-08T07:59:58.1595204Z     at Function.Module._load (module.js:424:25)
2023-06-08T07:59:58.1595453Z     at Module.require (module.js:504:17)
2023-06-08T07:59:58.1595614Z     at require (internal/module.js:20:19)
2023-06-08T07:59:58.1596393Z     at Object.<anonymous> (C:\agent-cloud-100\_work\_tasks\DownloadExternalBuildArtifacts_b099689b-039e-4450-8658-c72e3895dd3f\15.223.1\download.js:14:12)
2023-06-08T07:59:58.1596642Z     at Module._compile (module.js:577:32)
2023-06-08T07:59:58.1596813Z     at Object.Module._extensions..js (module.js:586:10)
2023-06-08T07:59:58.1596982Z     at Module.load (module.js:494:32)
2023-06-08T07:59:58.1597128Z     at tryModuleLoad (module.js:453:12)
2023-06-08T07:59:58.1597289Z     at Function.Module._load (module.js:445:3)
2023-06-08T07:59:58.2052847Z ##[error]Exit code 1 returned from process: file name 'C:\agent-cloud-100\externals\node\bin\node.exe', arguments '"C:\agent-cloud-100\_work\_tasks\DownloadExternalBuildArtifacts_b099689b-039e-4450-8658-c72e3895dd3f\15.223.1\download.js"'.
2023-06-08T07:59:58.2064012Z ##[section]Finishing: Download artifact - BKC.NVM.Images (External TFS Build)

[nohwnd]

@ytidhar Thank you for your report, this look like this issue: https://github.com/Microsoft/azure-pipelines-task-lib/issues/274 and unrelated to vstest. Please try to solution provided in the linked issue, and if that won't solve it, please consider filing the issue on their issues page.

[faix]

@nohwnd do you have an ETA on propagation? Got hit by this issue today.

[nohwnd]

It depends on how you are consuming test platform, dotnet sdk preview installers are available on dotnet/installer and 7.0.3xx 7.0.4xx and net8 should all already have the fix inserted. Official release of next sdk patch should be in July.

Visual Studio waits for approval to insert the fix, and I think the next fix release should be before the end of June.

Nuget packages that can be used directly via installer are already published on nuget.

[faix]

Thanks!

[317jamtay317]

the work around worked for me, Thanks!

[cperezCyf]

Ran with the following tasks:

- task: VisualStudioTestPlatformInstaller@1
  inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '17.6.2'

- task: VSTest@2
  displayName: 'Run Unit Tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: |
       **\*test*.dll
       !**\*TestAdapter.dll
       !**\obj\**
      searchFolder: '$(System.DefaultWorkingDirectory)'
    # ------------  Use the version installed by installer -----------
      vsTestVersion: 'toolsInstaller'

But still got the original error

[nohwnd]

@cperezCyf Could you please post the error, and a little bit of additional output around the error? Sometimes there are 2 or more errors. Or maybe you see this on a public pipeline and could link me to it?

[cperezCyf]

Sure, here is the error that I get:

[nohwnd]

@cperezCyf Is there a version of vstest.console printed in the console? This looks like 17.6.0 or 17.6.1 error, because it failed on .net standard 2.0 library.

[cperezCyf]

vstest.console is present in the console

for additional context here is the yaml:

trigger:
- none

pool:
  vmImage: 'windows-latest'

variables:
  solution: '**/*.sln'
  project: '**/Project.csproj'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Debug'


steps:

- task: NuGetToolInstaller@1
  name: 'NuGetToolInstaller'
  displayName: 'NuGet Tool Installer'
  inputs:
    checkLatest: true

- task: NuGetCommand@2
  name: 'NuGetRestore'
  displayName: 'NuGet restore packages'
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  displayName: 'build'
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactStagingDirectory)\WebApp.zip" /p:DeployIisAppPath="Default Web Site"'     
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'


- task: VisualStudioTestPlatformInstaller@1
  inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '17.6.2'

- task: VSTest@2
  displayName: 'Run Unit Tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: |
       **\*test*.dll
       !**\*TestAdapter.dll
       !**\obj\**
      searchFolder: '$(System.DefaultWorkingDirectory)'
    # ------------  Use the version installed by installer -----------
      vsTestVersion: 'toolsInstaller'

[nohwnd]

@cperezCyf I don't see the version of the vstest.console that is being used. Based on your yaml it should be the latest we released, but the error seems to be still the same as what it is fixing. Using this test project filter should solve your issue, if your test dll names match it:

**\bin\**\*Tests.dll
**\bin\**\*Test.dll

[georgewwww]

@cperezCyf The version for the vstest.console is shown a bit lower, after RunSettings block, for me was on the line 46. By the way, I also encountered this problem, and it's solved by using these two tasks for testing step:

variables:
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

...

- task: VisualStudioTestPlatformInstaller@1
  inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '17.6.2'

- task: VSTest@2
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    vsTestVersion: 'toolsInstaller'

Thank you @nohwnd :)

[cperezCyf]

Added both @nohwnd and @georgewwww but it doesn't seem to find the tests

The only way I have seem it to find my test is by filtering through: testAssemblyVer2: | **Test.cs

Although it doesn't seem to be running them

[nohwnd]

https://github.com/microsoft/vstest/issues/4516#issuecomment-1589191544

@cperezCyf Based on the output here, I don't think your projects is overriding the output path to be something different than bin (e.g. some projects use out instead). So maybe the names of your dlls actually don't match *test.dll / *tests.dll ?

Can you look at a build that succeeded previously and check what the name of the test dll is? And maybe post it here?

[cperezCyf]

Unfortunately I don't have a previous one that succeeded but I'll have a look at the names of the dlls. Thanks!

Update: it was the path, thanks again @nohwnd!!

[Blue101black]

Commenting here because this is where Azure is linking to for issues with Azure Pipelines at the moment.

Our pipelines have been failing intermittently the past couple of days and timing out / hanging on the build step mostly.

.Net 6 This was our build task for example:

- task: DotNetCoreCLI@2
          displayName: 'Build Web project'
          inputs:
            version: '6.0.x'
            packageType: sdk
            command: build
            arguments: ' --configuration Release -warnaserror '
            projects: '$(web_project)'

We fixed our pipelines by changing version from 6.0.x to 6.0.100

[nohwnd]

This is 4 versions old, the problem can still be encountered but there is no immediate action to take. Workaround 1 above is still recommended as a good default and was propagated to VSTest task in azdo, until this regression: https://github.com/microsoft/azure-pipelines-tasks/issues/19555