DigitalInBlue
Samples and iterations are only computed for first size of problem space (division by zero)
Bug Report
I have a Fixture which returns number of experiments:
std::vector<celero::TestFixture::ExperimentValue> getExperimentValues() const override
{
// Problem space as number of points (pairs of X/Y)
std::vector<celero::TestFixture::ExperimentValue> v;
v.emplace_back(1, 0);
v.emplace_back(256, 0);
v.emplace_back(512, 0);
v.emplace_back(1024, 0);
return v;
}
The baseline and other benchmarks are defined similarly to this::
BASELINE_F(wkt, to_string, Fixture, 0, 0)
{
for (auto const& p : this->points_)
{
celero::DoNotOptimizeAway(std::to_string(p.x()));
celero::DoNotOptimizeAway(std::to_string(p.y()));
}
}
Consider this output, where subsequent sizes of problem space receive unexpected number of iterations, 0 leading to division by zero or similar:
| Group | Experiment | Prob. Space | Samples | Iterations | Baseline | us/Iteration | Iterations/sec |
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|wkt | to_string | 1 | 30 | 262144 | 1.00000 | 0.94479 | 1058432.12 |
|wkt | to_string | 256 | 30 | 0 | 1.00000 | -nan(ind) | -nan(ind) |
|wkt | to_string | 512 | 30 | 0 | 1.00000 | -nan(ind) | -nan(ind) |
|wkt | to_string | 1024 | 30 | 0 | 1.00000 | -nan(ind) | -nan(ind) |
|wkt | stringstream | 1 | 30 | 131072 | 1.63591 | 1.54559 | 647000.75 |
|wkt | stringstream | 256 | 30 | 0 | -1.00000 | -nan(ind) | -nan(ind) |
|wkt | stringstream | 512 | 30 | 0 | -1.00000 | -nan(ind) | -nan(ind) |
|wkt | stringstream | 1024 | 30 | 0 | -1.00000 | -nan(ind) | -nan(ind) |
|wkt | lexical_cast | 1 | 30 | 131072 | 1.70723 | 1.61298 | 619972.00 |
|wkt | lexical_cast | 256 | 30 | 0 | -1.00000 | -nan(ind) | -nan(ind) |
|wkt | lexical_cast | 512 | 30 | 0 | -1.00000 | -nan(ind) | -nan(ind) |
|wkt | lexical_cast | 1024 | 30 | 0 | -1.00000 | -nan(ind) | -nan(ind) |
If I change the getExperimentValues to read
std::vector<celero::TestFixture::ExperimentValue> getExperimentValues() const override
{
std::uint64_t const total_tests = 4;
// Problem space as number of points (pairs of X/Y)
std::vector<celero::TestFixture::ExperimentValue> v;
v.emplace_back(1, uint64_t(std::pow(2, total_tests - 0)));
v.emplace_back(256, uint64_t(std::pow(2, total_tests - 1)));
v.emplace_back(512, uint64_t(std::pow(2, total_tests - 2)));
v.emplace_back(1024, uint64_t(std::pow(2, total_tests - 3)));
return v;
}
and I keep the BASELINE_F(wkt, to_string, Fixture, 0, 0), then I get this
| Group | Experiment | Prob. Space | Samples | Iterations | Baseline | us/Iteration | Iterations/sec |
|:--------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|:---------------:|
|wkt | to_string | 1 | 30 | 262144 | 1.00000 | 0.91043 | 1098376.39 |
|wkt | to_string | 256 | 30 | 8 | 1.00000 | 249.87500 | 4002.00 |
|wkt | to_string | 512 | 30 | 4 | 1.00000 | 489.50000 | 2042.90 |
|wkt | to_string | 1024 | 30 | 2 | 1.00000 | 980.00000 | 1020.41 |
|wkt | stringstream | 1 | 30 | 262144 | 1.59528 | 1.45240 | 688517.27 |
|wkt | stringstream | 256 | 30 | 8 | 1.19510 | 298.62500 | 3348.68 |
|wkt | stringstream | 512 | 30 | 4 | 1.21655 | 595.50000 | 1679.26 |
|wkt | stringstream | 1024 | 30 | 2 | 1.21939 | 1195.00000 | 836.82 |
where first experiment always gets the number of iterations calculated by Celero, 262144 instead of pre-calculated 16, but subsequent experiments within a set get the iterations from the getExoerimentValues spec.
I'm confused, is the preference of 0 from BASELINE_F for the first experiment an expected behaviour?
I am seeing the same issue with version 2.9.0. When applying "problem space" feature, Celero does not seem to respect the "zero" iterations feature.
