Setup step of `BENCHMARK_ADVANCED` may be skipped during estimation phase

[qnzhou]

Describe the bug In the following toy example, setup() may not be executed before every run() during estimation phase.

BENCHMARK_ADVANCED("advanced")(Catch::Benchmark::Chronometer meter) {
    setup();
    meter.measure([&]() { return run(); });
};

If each function print out its name, the output can be something like the following:

setup()
run()
setup()
run()
run()
setup()
run()
run()
run()
run()
...

Thus, if run() makes the assumption that setup() is executed (e.g. clear out the result buffer that run() uses), it may break the benchmark test.

Expected behavior setup() is executed before every run().

Reproduction steps See description code snippet.

Platform information:

• OS: MacOS (x86_64-apple-darwin19.6.0)
• Compiler+version: Apple clang version 12.0.0 (clang-1200.0.32.28)
• Catch version: v3.0.0-preview4

[nandanvasudevan]

Its not just that, the number of times setup and benchmark is invoked is very much different. Check this code: https://nandanv.godbolt.org/z/EY9daer53 Counter values are way out of proportion.

setup: 127  bench: 134963327

[horenmar]

This is done to avoid timing skew from the setup code. When you are in the setup() step, meter.runs() tells you how many times run() happens and you need to prepare separate data for all of it, to avoid tainting the timings.

Example

TEST_CASE("Remove from front", "[!benchmark]") {
	const auto num_elements = GENERATE(1'000, 100'000, 10'000'000);
	const auto data = generate_data(num_elements);

	BENCHMARK_ADVANCED("Remove by erase, i := " + std::to_string(num_elements))(Catch::Benchmark::Chronometer meter) {
		std::vector<std::vector<int>> bench_data(meter.runs(), data);
		meter.measure([&](int iter) {
                        auto& iter_data = bench_data[iter];
                        iter_data.erase(iter_data.begin());
                        return iter_data.back(); // optimization barrier
		});
	};
}