vortex icon indicating copy to clipboard operation
vortex copied to clipboard

Benchmarking Improvements

Open AdamGS opened this issue 9 months ago • 0 comments

  • [ ] Clear OS caches after each run
  • [ ] Label each run with query idx in Polar Signals
  • [ ] Move Clickbench to run on the single file variant and potentially a smaller multi-file one, to reduce the amount of time spent of "discovering" files. If we think that that part is important enough, we can build another benchmark.
  • [ ] Look into something like hyperfine to orchestrate runs, allowing us to remove some of the code that currently has to exist in every benchmark.
  • [ ] Split data generation/conversion from the main benchmark binaries into dedicated tools.
  • [ ] Sharing as little state as possible between runs, reducing the effect of caches/other random state that might creep in.

AdamGS avatar Apr 01 '25 15:04 AdamGS