BenchmarkDotNet icon indicating copy to clipboard operation
BenchmarkDotNet copied to clipboard

--join does not respect types and ratio

Open timcassell opened this issue 3 years ago • 0 comments

When benchmarking multiple types with --join option, the resulting table does not group by types, and Ratio is completely messed up.

|         Type |       Method | Pending |       Mean |    Error |   StdDev | Ratio | RatioSD | Code Size |  Gen 0 | Allocated | Alloc Ratio |
|------------- |------------- |-------- |-----------:|---------:|---------:|------:|--------:|----------:|-------:|----------:|------------:|
|   AsyncAwait |     Callback |   False |   287.8 ns |  2.01 ns |  1.88 ns |  1.00 |    0.00 |   1,561 B | 0.2651 |     832 B |        1.00 |
| ContinueWith |     Callback |   False |   293.9 ns |  0.70 ns |  0.66 ns |  1.02 |    0.01 |   1,572 B | 0.2651 |     832 B |        1.00 |
|   AsyncAwait | ProtoPromise |   False |   352.5 ns |  1.53 ns |  1.36 ns |  1.22 |    0.01 |   2,010 B |      - |         - |        0.00 |
| ContinueWith | ProtoPromise |   False |   495.5 ns |  8.59 ns |  8.03 ns |  1.72 |    0.03 |  11,185 B |      - |         - |        0.00 |
|   AsyncAwait |   RsgPromise |   False |         NA |       NA |       NA |     ? |       ? |         - |      - |         - |           ? |
| ContinueWith |   RsgPromise |   False |   573.8 ns |  3.57 ns |  2.79 ns |  1.99 |    0.02 |   2,476 B | 0.3290 |    1032 B |        1.24 |
|   AsyncAwait |         Task |   False |   319.3 ns |  1.35 ns |  1.05 ns |  1.11 |    0.01 |   2,496 B | 0.0610 |     192 B |        0.23 |
| ContinueWith |         Task |   False | 2,231.1 ns | 26.55 ns | 24.84 ns |  7.75 |    0.09 |   9,632 B | 0.3891 |    1224 B |        1.47 |
|   AsyncAwait |      UniTask |   False |   367.2 ns |  1.68 ns |  1.57 ns |  1.28 |    0.01 |   2,425 B |      - |         - |        0.00 |
| ContinueWith |      UniTask |   False |   708.9 ns |  8.67 ns | 15.86 ns |  2.51 |    0.07 |   8,385 B |      - |         - |        0.00 |
|   AsyncAwait |    ValueTask |   False |   485.4 ns |  2.04 ns |  1.59 ns |  1.68 |    0.01 |   2,991 B |      - |         - |        0.00 |
| ContinueWith |    ValueTask |   False |         NA |       NA |       NA |     ? |       ? |         - |      - |         - |           ? |

Types should be grouped separately with --join and Ratio respected for each type.

See also #1864 which was regarding categories, but this issue is without categories.

timcassell avatar Jun 21 '22 22:06 timcassell