ReBench
ReBench copied to clipboard
smarr
Execute and document benchmarks reproducibly.
Currently, we retry benchmarks on the same executor even when the binary is missing. I think, we could correctly interpret the error, and abort them, which may slightly improve the...
It would be nice to pass on environment variables to the binary/vm - on a per-binary basis - on a per-benchmark basis - on a per-suite basis - on a...
The data files don't have a header line, which makes it a bit a challenge to guess which column is which. We probably also should document the data format in...
Expand the basic documentation to cover all features, and expand the set of tutorials to cover the common use cases.
To help new users with ReBench, 5min tutorials would be very helpful. Goal of such tutorials would be to show that it takes less time to learn ReBench than it...
The community has a number of analyses that "everyone" uses. Providing them as reusable building blocks would make the day-to-day data analysis much easier. Each analysis should come with documentation,...
The goal is to have a somewhat common UI that enables easy reuse of analyses from previous projects, or experiments. ReBenchDB will need an API to access data, as well...
Add support for exporting data and scripts to create self-contained and archivable artifacts to ReBenchDB. Data and scripts need to be packaged so that evaluators of artifacts can reproduce results,...
Expand ReBenchDB's support for data provenance by detecting environmental differences that affect the comparability of results. Operating system, software, or CPU microcode updates can change performance and make comparison with...
Optimize result storage to increase scalability for ReBenchDB to reduce memory and storage requirements. Hopefully, we can reduce the system's requirements, allow a larger number of active users/requests, and make...