Unable to filter benchmarks with TestAdapter

[olstakh]

Steps to reproduce

1. Follow https://benchmarkdotnet.org/articles/features/vstest.html (console project + TestAdapter package + Test.Sdk)
2. Define 2 benchmarks with different [BenchmarkCategory("MyTest1")] and [BenchmarkCategory("MyTest2")] attributes
3. Run dotnet test -c Release --filter "Category=MyTest1"

Expected: 1 benchmark executed (category attribute is being added as a trait )

Actual: both benchmarks executed

Also [unrelated] - are these benchmarks supposed to show up in Test Explorer in VS? I'd assume they would, since dotnet test picks them up

0.14.0 version of TestAdapter is used

[timcassell]

Is that filter behavior the same without the TestAdapter?

cc @caaavik-msft

[olstakh]

Without TestAdapter dotnet test doesn't work on a console project with benchmarks (meaning it finds no tests)

I also tried to filter with --anyCategory switch for benchmark runner, but it's unclear how to pass these parameters via dotnet test command to bm executor, as it complains about unknown switches.

[timcassell]

I meant if you run it the old way with dotnet run.

[olstakh]

It doesn't, because it's not the right syntax for filtering benchmarks. I can use --anyCategories "MyTest1", but not --filter "Category=MyTest1".

[olstakh]

May be i'm misunderstanding the purpose of TestAdapter package?

My thoughts were that it allows integration with vstest and exposes proper hooks to filter / configure the test run (which i also saw in code linked in the original post, like providing category traits, etc). If that's the case - i don't see it being respected currently.

It's as if --filter is not respected at all with this vstest adapter, as i can put --filter "Category=WhoCares" and it will still run both.

[timcassell]

My guess is because --filter is a command line option used by BenchmarkDotNet since forever, so it's conflicting with the VSTest filter option. But I'm not sure, so I will defer to @caaavik-msft.

[olstakh]

I think it gets parsed prior to BenchmarkDotNet logic, since i can't use regular BDN commands with dotnet test (unless im doing it wrong)

Like dotnet test -c Release --anyCategories "MyTest1" will fail with unknown switch --anyCategories, so --filter might not conflict between BDN and VsTest, but can be wrong of course

[newmasterSG]

@timcassell Hi, can you help me to find out how to debug test adapter? Because I tried several ways, but none did work

[timcassell]

@newmasterSG Maybe this will help? https://benchmarkdotnet.org/articles/guides/troubleshooting.html

Also, I think the test adapter automatically hides benchmarks in Debug mode, so you may need to dig into the test adapter code and disable that.

[newmasterSG]

@timcassell Also I have a question about is test adapter is activated by dotnet test? And what I need to disable?

[newmasterSG]

I asked because I had these test but I couldn't see them in test explorer `
public class Test { [Benchmark] [BenchmarkCategory("MyTest1")] public int Test1() { return Fibonacci(10); }

   [Benchmark]
   [BenchmarkCategory("MyTest2")]
   public int Test2()
   {
       return Fibonacci(15);
   }

   private int Fibonacci(int n)
   {
       return n <= 1 ? n : Fibonacci(n - 1) + Fibonacci(n - 2);
   }

} `

[kant2002]

Maybe @AndreyAkinshin also can help

[timcassell]

https://github.com/dotnet/BenchmarkDotNet/pull/2438#issuecomment-1856509049

It looks like if you apply [InProcess] it should work.

[newmasterSG]

@timcassell

I didn't see tests even I added [InProcess] attribute for class Test

Also every time when I write dotnet test -c Debug --framework net8.0 --filter "Category=Category1" I get the next message : Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Or Shall I do this in Sample project?