vortex
vortex copied to clipboard
Benchmarking Improvements
- [ ] Clear OS caches after each run
- [ ] Label each run with query idx in Polar Signals
- [ ] Move Clickbench to run on the single file variant and potentially a smaller multi-file one, to reduce the amount of time spent of "discovering" files. If we think that that part is important enough, we can build another benchmark.
- [ ] Look into something like hyperfine to orchestrate runs, allowing us to remove some of the code that currently has to exist in every benchmark.
- [ ] Split data generation/conversion from the main benchmark binaries into dedicated tools.
- [ ] Sharing as little state as possible between runs, reducing the effect of caches/other random state that might creep in.