`ReturnValueValidator` does not run `GlobalSetup` methods for each `Params`

[snowfrogdev]

When running this benchmark:

using System;
using System.Linq;
using BenchmarkDotNet.Attributes;
using Ardalis.Extensions.Enumerable;
using System.Collections.Generic;

namespace Ardalis.Extensions.Benchmarks.Enumerable;

[MemoryDiagnoser]
[ReturnValueValidator(failOnError: true)]
public class ProductBenchmarks
{
  [Params(4, 5, 6, 7)]
  public int N { get; set; }

  private int[] _numbers;

  [GlobalSetup]
  public void GlobalSetup()
  {
    _numbers = System.Linq.Enumerable.Range(1, N).ToArray();
    Console.WriteLine($"N = {N} and _numbers has a length of {_numbers.Length}");
  }


  [Benchmark(Baseline = true)]
  public long ProductAggregate()
  {
    return _numbers.ProductAggregate();
  }

  [Benchmark]
  public int ProductForEach()
  {
    return _numbers.ProductForEach();
  }

  [Benchmark]
  public int Product()
  {
    return _numbers.Product();
  }
}

static class ProductBenchmarksExtensions
{
  public static int ProductAggregate(this IEnumerable<int> numbers)
  {
    return numbers.Aggregate(1, (acc, x) => acc * x);
  }

  public static int ProductForEach(this IEnumerable<int> numbers)
  {
    int product = 2; // <- Set to 2 so that we have different return values and make validation fail
    foreach (var x in numbers)
    {
      product *= x;
    }
    return product;
  }
}

I get the following output:

// Validating benchmarks:
N = 4 and numbers has a length of 4
Inconsistent benchmark return values in ProductBenchmarks: ProductAggregate: 24, ProductForEach: 48, Product: 24 [N=4]
Inconsistent benchmark return values in ProductBenchmarks: ProductAggregate: 24, ProductForEach: 48, Product: 24 [N=5]
Inconsistent benchmark return values in ProductBenchmarks: ProductAggregate: 24, ProductForEach: 48, Product: 24 [N=6]
Inconsistent benchmark return values in ProductBenchmarks: ProductAggregate: 24, ProductForEach: 48, Product: 24 [N=7]

The methods being benchmarked are returning the factorial of N. As you can see from the return values being identical, the GlobalSetup method that updates the List of integers to multiply doesn't seem to be ran before validating the return values for the subsequent Ns.

I can confirm that the GlobalSetup method runs before each benchmark for each N when there is not failing validation and we proceed with benchmarking.

Feel free to clone this repo, here is the exact commit to reproduce the above state: https://github.com/snowfrogdev/Ardalis.Extensions/tree/b17763f52b890e25098557095c1ec954168cafe4

[snowfrogdev]

Hey folks. OP here.

I was trying to use ReturnValueValidator on a set of benchmarks today and I ran into this issue again. Had a good laugh when I realized I had already ran into this two years ago and created an issue for it.

I see there's a PR pending that would fix this issue but it seems to be stuck because of some problems with xUnit. Is there anything we can do to get this moving?

I haven't combed through the entire PR but I got the feeling that it was doing more than fixing this issue. If that PR is indeed frozen for reasons we can't control, are we opened to making a smaller code change that would only deal with the current issue?

[timcassell]

I'm not sure what @YegorStepanov's plans are for his PR, but it hasn't been touched in a while. Feel free to take a crack at it if you want @snowfrogdev (you can use his PR as a starting point if you want).

[Daniel-Svensson]

~~if you make a fix for this it would be great if you could make sure that attribute /attributesource are also supported.~~

~~I just ran into a issue with "missing test/property" when trying to use arguments and return value validation.~~ I used the same arguments for all methods, but maybe you can use params with complex types instead to get the something similar (I did not want "all pairs")