Need an easier way to figure out if something is wrong with a benchmark run

[CaptainIRS]

Please share the technical limitation of Caliper that you encountered.

As of now the only ways to figure out if something is wrong with a benchmark run (for example, failed transactions) are to look visually at the generated HTML report or scroll through the logs and look out for failed transactions in the tables printed out by Caliper. This however is not suitable for scripts to figure out if something has failed during the benchmark so that it could be used to report a failure in a CI workflow with an actionable error message.

Please detail your feature idea that could alleviate the limitation.

• Caliper's launch manager command could exit with a non-zero exit code if any transaction fails throughout the benchmark run. There could also be a key called expectFailure: true|false in the round/workflow configuration section of benchconfig.yaml so that Caliper exits fine if transaction failures are expected from the workload by design.
• Errors in individual transactions could be aggregated and a list of unique errors could be shown at the end of a benchmark run, along with information to track down the round and workload during which the error occurred.
• Ability to generate machine readable (JSON/CSV) report of transaction statistics when provided a flag to the CLI (related issue: #320)

Please share some details about your use case if possible, and how the new feature would make Caliper a better performance benchmarking framework.

• This would be useful for users who would like to integrate Caliper as a part of their CI workflow, where it is essential to take note of any failure that happens during the workflow.
• This would be useful for developers of Caliper and the Caliper Benchmark repositories as it would be helpful in tracking down issues with Caliper itself or the official benchmark suite.

Please share any suggestions about the new feature's code/configuration API (using formatted YAML segments or pseudo-code).

If idea#1 is implemented, a new key like expectFailure would be needed in the round/workflow configuration of benchconfig.yaml:

test:
    workers:
        number: 2
    rounds:
    - label: query1
      txNumber: 50
      rateControl:
        type: composite-rate
        opts:
          weights: [2,1]
          rateControllers:
          - type: fixed-rate
            opts:
              tps: 2
          - type: fixed-rate
            opts:
              tps: 4
       workload:
         module: ./../query.js
+        expectFailure: true