execution-spec-tests icon indicating copy to clipboard operation
execution-spec-tests copied to clipboard

feat: CI action to generate a `diff` file between the fixtures generated from `main` against the PR's

Open marioevz opened this issue 2 years ago • 2 comments

Add a CI action that automatically generates a diff file of the comparison between the fixtures generated by the current main branch, and the PR's branch.

The diff file could be made available somewhere in the PR for review and download if necessary.

marioevz avatar Jan 18 '24 19:01 marioevz

@spencer-tb and I just brainstormed.

Here's a suggestion for scoping this out:

  1. Add an entry point, e.g., ./src/entry_points/diff_fixtures.py that takes two URIs (local or remote/vanilla directory or tarball) as arguments and diffs them. This would allow specifying a release tarball as a source to diff from. Or just two local directories. The command reports: a. A diff of the fixtures directory structure; i.e., new/removed fixture json files. Don't fail if only new. TBD: Return non-zero if fixture json files are missing? A test rename would lead to false positives here. b. Writes a diff to stdout and returns non-zero if already existing fixtures differ. This output can be written to stdout, to file, or both.
  2. Github Actions: Upload the fixtures (tarballs) from the existing test.yaml flow as an artifact upon every: a. Commit on main in test.yaml, and, b. Every tag/release in fixtures.yaml - note, this is already done :wink:
  3. TBD! Tox: Add a new diff testenv. That: a. Downloads the latest artifact from main (and/or latest release?) b. Runs diff_fixtures and writes the diff to file.
  4. Github Actions: Add a new Github Action that downloads the fixture artifact(s) generated from this flow's test.yaml (that ran fill) and diffs them against main using tox env from 3. Upload the diff as an artifact.

Other Questions

  1. Github Actions typically have to pass in order to merge; how do we handle differences that are allowed? For example: a. A framework change that intentionally changes fixtures. b. Updating a test implementation that changes the fixture. I.e., how do we make it clear in the flow that there are changes that potentially need review, but that don't necessarily block merging.
  2. Can we readily download Github artifacts from the command-line to run the diff locally (before commit). It seems so: https://docs.github.com/en/rest/actions/artifacts
  3. How to view the diffs in Github UI for each PR, respectively, what format to write the diff output in, so that it can be readily viewed in the Github UI for each PR.

danceratopz avatar Feb 02 '24 16:02 danceratopz

If we don't care about the details of the modifications to each file and we only care about which files were affected, there's a great suggestion here in this StackOverflow answer: https://stackoverflow.com/a/75232801

It uses git notes to save a list of the fixtures (artefacts in their case) plus a hash of each file on the commit's notes, so it should not affect the commit's contents, but we can retrieve this info on the PR's head and make the comparison of both lists.

In our case the hash should be calculated without the "_info" key, but that should be it.

marioevz avatar Feb 02 '24 18:02 marioevz