As @jrhemstad mentioned here one could add hidden benchmarks using a "hidden" tag using a tag feature as proposed in #81 to be able to define benchmarks that will not run per default.