tinylibs
Results are incorrect
I don't know what causes this, but all results seem completely off-bounds.
Reproduction
/* eslint-disable no-console */
import { Bench } from "tinybench";
const suite = new Bench({ iterations: 100_000 });
suite
.add("10", () => {
for (let i = 0; i < 10; i++);
})
.add("1 000", () => {
for (let i = 0; i < 1_000; i++);
})
.add("100 000", () => {
for (let i = 0; i < 100_000; i++);
});
suite.tasks.forEach((task) => {
task.addEventListener("complete", () => {
console.log(`${task.name}\t\t${task.result.mean * 1000}`);
});
});
await suite.warmup();
await suite.run();
console.table(suite.tasks.map(({ name, result }) => ({ "Task Name": name, "Average Time (ps)": result.mean * 1000, "Variance (ps)": result.variance * 1000 })));
Output:
10 33.51586236828007
1 000 33.56626915780828
100 000 33.56723708981648
┌─────────┬───────────┬───────────────────┬─────────────────────┐
│ (index) │ Task Name │ Average Time (ps) │ Variance (ps) │
├─────────┼───────────┼───────────────────┼─────────────────────┤
│ 0 │ '10' │ 33.51586236828007 │ 0.13611378835207122 │
│ 1 │ '1 000' │ 33.56626915780828 │ 0.12475075068569264 │
│ 2 │ '100 000' │ 33.56723708981648 │ 0.13970421827784552 │
└─────────┴───────────┴───────────────────┴─────────────────────┘
Two things here:
- They all seem to take as long, which is not possible.
- They all send their "complete" event at the same time.
Maybe it's an engine optimization that causes this!
Please try adding eval('') before each benchmark (in the task function body) so we see if it's the same still.
The code provided is just an example. I've run into such an issue for all the benchmarks I've run with tinybench.
Well, I'll look into this, but if you want to find out about this, please try to, I'd like to see your findings, you already helped so much with tinybench!
let me know when I should look into it.
Closed in https://github.com/tinylibs/tinybench/commit/41af489a19110a6805c96550750dded78456aa2f