feat: CI action to generate a `diff` file between the fixtures generated from `main` against the PR's
Add a CI action that automatically generates a diff file of the comparison between the fixtures generated by the current main branch, and the PR's branch.
The diff file could be made available somewhere in the PR for review and download if necessary.
@spencer-tb and I just brainstormed.
Here's a suggestion for scoping this out:
- Add an entry point, e.g.,
./src/entry_points/diff_fixtures.pythat takes two URIs (local or remote/vanilla directory or tarball) as arguments and diffs them. This would allow specifying a release tarball as a source to diff from. Or just two local directories. The command reports: a. A diff of the fixtures directory structure; i.e., new/removed fixture json files. Don't fail if only new. TBD: Return non-zero if fixture json files are missing? A test rename would lead to false positives here. b. Writes a diff to stdout and returns non-zero if already existing fixtures differ. This output can be written to stdout, to file, or both. - Github Actions: Upload the fixtures (tarballs) from the existing
test.yamlflow as an artifact upon every: a. Commit on main intest.yaml, and, b. Every tag/release infixtures.yaml- note, this is already done :wink: - TBD! Tox: Add a new
difftestenv. That: a. Downloads the latest artifact from main (and/or latest release?) b. Runsdiff_fixturesand writes the diff to file. - Github Actions: Add a new Github Action that downloads the fixture artifact(s) generated from this flow's test.yaml (that ran
fill) and diffs them againstmainusing tox env from 3. Upload the diff as an artifact.
Other Questions
- Github Actions typically have to pass in order to merge; how do we handle differences that are allowed? For example: a. A framework change that intentionally changes fixtures. b. Updating a test implementation that changes the fixture. I.e., how do we make it clear in the flow that there are changes that potentially need review, but that don't necessarily block merging.
- Can we readily download Github artifacts from the command-line to run the diff locally (before commit). It seems so: https://docs.github.com/en/rest/actions/artifacts
- How to view the diffs in Github UI for each PR, respectively, what format to write the diff output in, so that it can be readily viewed in the Github UI for each PR.
If we don't care about the details of the modifications to each file and we only care about which files were affected, there's a great suggestion here in this StackOverflow answer: https://stackoverflow.com/a/75232801
It uses git notes to save a list of the fixtures (artefacts in their case) plus a hash of each file on the commit's notes, so it should not affect the commit's contents, but we can retrieve this info on the PR's head and make the comparison of both lists.
In our case the hash should be calculated without the "_info" key, but that should be it.