sapp icon indicating copy to clipboard operation
sapp copied to clipboard

Add SARIF schema test

Open the-storm opened this issue 2 years ago • 2 comments

Pre-submission checklist

  • [x] I've ran the following linters locally and fixed lint errors related to the files I modified in this PR
    • [x] black .
    • [x] usort format .
    • [x] flake8
  • [x] I've installed dev dependencies pip install -r requirements-dev.txt and completed the following:
    • [x] I've ran tests with ./scripts/run-tests.sh and made sure all tests are passing

Summary

This PR adds initial tests to SARIF output from SAPP. Currently the SARIF output has no test coverage at all. This API add a schema test that the output SARIF passes the defined schema.

Test Plan

./scripts/run-tests.sh
----------------------------------------------------------------------
Ran 111 tests in 1.719s
...

and running the specific new tests

(venv) ielsayed@ibrahims-mbp-2 ~/s/r/personal-sapp (add-sarif-schema-validation-tests) [SIGINT]> python3 -m unittest -v sapp.ui.tests.sarif_test
testSarifSchemaCheckWithIssuesNoTraces (sapp.ui.tests.sarif_test.SarifTest.testSarifSchemaCheckWithIssuesNoTraces) ... ok
testSarifSchemaCheckWithIssuesWithTraces (sapp.ui.tests.sarif_test.SarifTest.testSarifSchemaCheckWithIssuesWithTraces) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.134s

OK

the-storm avatar Sep 13 '23 13:09 the-storm

I think the failing test is just a flaky test. I don't think the failure is from this PR

the-storm avatar Sep 14 '23 19:09 the-storm

/cc @arthaud not sure if you guys missed this PR :D

the-storm avatar Oct 11 '23 14:10 the-storm