Microbenchmarks icon indicating copy to clipboard operation
Microbenchmarks copied to clipboard

regularize mandelperf timings to include sums & assertions or not

Open johnfgibson opened this issue 7 years ago • 2 comments

The timings of the mandelbrot calculations are not consistent between all languages. In some (C, Rust), the timings include summing the total number of iterations over the specified grid and asserting that number is 14791. In others (Python, Julia), the timings only include iterating over the grid, not the sum or assertion. Some languages repeat mandelperf 100 times and return total time / 100, others return the time for a single run.

johnfgibson avatar May 21 '18 18:05 johnfgibson

IIRC, the languages where we only time a single run are the ones where that run takes as long or longer than 100 runs of faster languages. Feel free to make it more consistent but I’d really rather not have to wait for several hours while Octave runs the benchmarks 100 times.

StefanKarpinski avatar May 21 '18 18:05 StefanKarpinski

I noticed that parse_integers might also have similar inconsistencies: The Julia version checks the outcome of parse only at the end of t iterations (line 26, and Lua too), but the C version... presumably it would check every single parsed number (line 263), but asserts made with assert can be cleared out in compile time if the NDEBUG flag is set. ~~I don't think this is the case, but I can double-check that.~~ Just checked, the asserts are there regardless of the optimization level.

As the Rust version was influenced by C's and Julia's, it checks all parsed numbers unconditionally (line 278).


Update: this has been addressed in #18. The check in parse_int should indeed happen inside the loop.

Enet4 avatar May 23 '18 09:05 Enet4