ss3-source-code icon indicating copy to clipboard operation
ss3-source-code copied to clipboard

[Admin]: Add gh action that does line by line ss_summary.sso comparison

Open k-doering-NOAA opened this issue 4 years ago • 5 comments

Thanks @Rick-Methot-NOAA for this idea; this may help catch differences among models in quantities that are not tested

The difficult part is that there will always be some differences (e.g., the date of the model run may be different , the number of digits or use of sci notation or not may be different), so it may be difficult to make this job completely automated.

A first step toward figuring out how this could be useful, though, is simply to develop a job that does the comparison.

k-doering-NOAA avatar Dec 13 '21 15:12 k-doering-NOAA

To do this I think it would have to run ss3 on a/the user example or test model(s) so that there would be something to compare. I guess the question is how complex a/the test model file(s) would need to be to compare all the important quantities. @Rick-Methot-NOAA do you have any more thoughts on this as we move issues over from the workflows repo to the other repos in the nmfs-stock-synthesis org?

e-perl-NOAA avatar May 31 '23 12:05 e-perl-NOAA

I am looking for a comparison that is functionally equivalent to what I do when I run a character-by-character, side-by-side text comparison and visually scan for color-coded differences that are meaningful. When I do that, my eye-mind is keying on significant digits. A difference in the 2nd or 3rd digit is a big deal, a difference in the 6th digit is not. Challenge is that the test needs to deal with a wide range of numbers and also sign differences (e.g. -0.000001 is not much different than +0.000001). So, I think we need to train the gha to do that numerically for every number in the output. Suggest starting by researching available algorithms to do such a comparison.

Rick-Methot-NOAA avatar May 31 '23 15:05 Rick-Methot-NOAA

I do not think we need to run this with every commit. Is is possible to create a gha dashboard so we could manually run when needed?

Rick-Methot-NOAA avatar May 31 '23 19:05 Rick-Methot-NOAA

Yes, you can run it manually as long as we add workflow_dispatch: to on:.

kellijohnson-NOAA avatar May 31 '23 22:05 kellijohnson-NOAA