azure-dev icon indicating copy to clipboard operation
azure-dev copied to clipboard

Improve test printout in CI builds

Open weikanglim opened this issue 3 years ago • 1 comments

Our current CI pipeline uses go test -v which prints test execution as you'd expect it -- in a sequential, verbose logging format of everything that happened. This has some downsides.

When a test fails, it's not clear what failed since no summary is printed. One has to find the text "FAIL: " to hunt for the test failure. Even when running without verbose go test -v, it now prints FAIL test output, but doesn't list useful summary information like: number of tests run/skipped/failed.

Proposal: I think we should use gotestsum for running our tests. It's a tool that reads go test -json and can summarize test runs nicely. The additional advantage is that since it understands a standard test2json file, we can reparse the JSON file and then convert to different formats.

We should also publish the test results to Azure DevOps. Here's a community guide on how to do it (converting to JUnit test report): https://ruddles.github.io/Go-test-results-azure

The workflow could look like:

  1. gotestsum --jsonfile test.json -- Prints out summary to stdout and saves the test runJSON as a file.
  2. We can then reparse the json file and create a verbose output file OR log stdout in CI: cat test.json | gotestsum --format standard-verbose OR cat test.json | gotestsum --format standard-verbose > test.log

weikanglim avatar Aug 18 '22 20:08 weikanglim

Let's try to use this approach together with the azdo test tab. There's a way to publish test results to azdo to produce something like: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1804702&view=ms.vss-test-web.build-test-results-tab

vhvb1989 avatar Aug 25 '22 16:08 vhvb1989