Adjust test report emails to show only relevant test results

[tracywalker]

For example, the report currently shows: Passed: 212, Failed: 14, Skipped: 53 (Blocked: 13), Errors: 0 -- Total: 279

Skipped is not really relevant to the test run. At this point in the Iris project life, it would be clearer to leave these out of the report and add a percentage to passed and failed: For example:

Passed: 212 (93.8%), Failed: 14(6.2%)

Note: All the Skipped, Blocked and Error results should continue to be included in the actual run output and logs. I just want these email reports to reflect the state of tests actually executed during a test run.

[mwobensmith]

We need to convey what we think the audience of the email report expects to learn. What is the current quality of their product?

This should be measured by the actual tests run, not details of all tests that exist in our repo, relevant or not. In addition, "Blocked" is a measure of accrued problems, and while it's important to state, it is conceptually different and hard to explain here.

"Skipped" is more of a metric about the repo of installed tests and is not pertinent to the run itself.

An email report should help answer the questions "how does this run measure up to previous runs? Are there new issues being surfaced?" Given the brevity of the email report, I think we should stick to basics and let the Control Center present more details.

Proposal:

Total run: 226
Passed: 212 (93.8%)
Failed: 14 (6.2%)
(Blocked and not run: 13)

I think we should leave "Skipped" off of the email report, as we'd need to explain all of the reasons why a test is in this category - experimental, retired, broken, platform-specific, not relevant, etc.

Also worth noting - we have discussed better reporting in Iris 2.0, including HTML-formatted emails. Given more tools, we'd have more options to present greater detail.

[tracywalker]

The email reports still need to be adjusted to report just what was run and pass/fail (%) of just the test that were run.

@ashughes1. For Iris as a service, do you want any other changes to the email reports?