Improve test printout in CI builds
Our current CI pipeline uses go test -v which prints test execution as you'd expect it -- in a sequential, verbose logging format of everything that happened. This has some downsides.
When a test fails, it's not clear what failed since no summary is printed. One has to find the text "FAIL: " to hunt for the test failure. Even when running without verbose go test -v, it now prints FAIL test output, but doesn't list useful summary information like: number of tests run/skipped/failed.
Proposal: I think we should use gotestsum for running our tests. It's a tool that reads go test -json and can summarize test runs nicely. The additional advantage is that since it understands a standard test2json file, we can reparse the JSON file and then convert to different formats.
We should also publish the test results to Azure DevOps. Here's a community guide on how to do it (converting to JUnit test report): https://ruddles.github.io/Go-test-results-azure
The workflow could look like:
-
gotestsum --jsonfile test.json-- Prints out summary to stdout and saves the test runJSON as a file. - We can then reparse the json file and create a verbose output file OR log stdout in CI:
cat test.json | gotestsum --format standard-verboseORcat test.json | gotestsum --format standard-verbose > test.log
Let's try to use this approach together with the azdo test tab. There's a way to publish test results to azdo to produce something like: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1804702&view=ms.vss-test-web.build-test-results-tab