benchmark-action
Genericize support for other tools
In addition to runtime performance benchmark work, one thing our project would be interested in is examining how our codebase measures across time for binary size and memory consumption metrics.
It would expand the notion of the tool parameter here to become a source of any type of metric values rather than just the output from a pre-configured performance benchmark tool. Much of github-action-benchmark's behavior -- collecting historical data, creating a dashboard, publishing -- would be just as beneficial for these other types of benchmark metrics.
So specifically, could there be an extra option for tool, maybe called other, and then have the JSON format/schema expected by the GH Action documented for this other scenario, and thereby allow more flexible uses of the action?
I could still the benefit of commonly-used benchmark tools being supported out-of-the-box (example: #43), so this proposal is in addition to that, not intended as a replacement for that.
It would be really cool to see this happen because I want to use this action so badly, but I use my custom benchmark suite and it would be nice to stick with this suite and still use this action.
AFAIK this was already done and documented here https://github.com/benchmark-action/github-action-benchmark#:~:text=Additionally%2C%20even%20though%20there%20is%20no%20explicit%20example%20for%20them%2C%20you%20can%20use%20customBiggerIsBetter%20and%20customSmallerIsBetter%20to%20use%20this%20action%20and%20create%20your%20own%20graphs%20from%20your%20own%20benchmark%20data.%20The%20name%20in%20these%20tools%20define%20which%20direction%20%22is%20better%22%20for%20your%20benchmarks.
