sigstore
Bring up our coverage and reenable coverage testing
We've temporarily disabled coverage testing while we're fleshing out the basic signing/verification functionality. Once things take a bit more shape, we should begin adding back coverage.
I expect this issue to be broken up into smaller PRs to test each module separately since there's probably quite a lot of work to do.
As of 4079bff2e85b54371d8a68fbdde693bcdb726980:
Name Stmts Miss Cover Missing
---------------------------------------------------------------------
sigstore/__init__.py 2 0 100%
sigstore/__main__.py 1 1 0% 15
sigstore/_internal/__init__.py 0 0 100%
sigstore/_internal/fulcio/__init__.py 3 0 100%
sigstore/_internal/fulcio/client.py 167 65 61% 76-79, 128-129, 135, 145, 177-178, 182-187, 198-291, 297-303, 325, 331
sigstore/_internal/merkle.py 42 26 38% 50-52, 62-68, 77-79, 83-85, 89-91, 100-126
sigstore/_internal/oidc/__init__.py 25 18 28% 33-63
sigstore/_internal/oidc/ambient.py 88 0 100%
sigstore/_internal/oidc/issuer.py 22 22 0% 15-52
sigstore/_internal/oidc/oauth.py 104 104 0% 15-212
sigstore/_internal/rekor/__init__.py 3 0 100%
sigstore/_internal/rekor/client.py 119 37 69% 59-65, 115-116, 122, 131-145, 151, 156-161, 169-189, 219, 223
sigstore/_internal/sct.py 110 53 52% 60-73, 80-90, 110-135, 144, 155-156, 159, 171, 180, 205, 233-237, 241-244, 269-314
sigstore/_internal/set.py 19 9 53% 42-60
sigstore/_sign.py 51 19 63% 63-125
sigstore/_store/__init__.py 0 0 100%
sigstore/_verify.py 118 61 48% 147-287
---------------------------------------------------------------------
TOTAL 874 415 53%
Most of _verify.py should be easy to add coverage for, since we can generate testvectors (and perturb them, to produce invalid states).
We'll probably want to do something similar to what we did on pip-audit, i.e. mark certain tests as "online" to emphasize that they need to connect to a CT log or some other service.
Does coverage have any way to maintain state in the repo and make an assertion like "coverage has not dropped for any files"?
Does
coveragehave any way to maintain state in the repo and make an assertion like "coverage has not dropped for any files"?
Not that I know of, unfortunately -- IIRC it uses a sqlite DB under the hood, which we could theoretically check in (if we don't mind the binary artifact), but I have no idea if they guarantee any amount of integrity/consistency between hosts.
We could consider a coverage provider, I believe cryptography has had moderate success with codecov (and our situation is thankfully simpler, since we're 100% Python with no platform specifics.)
I'm going to look into an online coverage provider, again -- our unit tests here require a certain amount of online functionality, so it doesn't make sense to enforce coverage on the client side (where tests might be skipped).
Looks like I don't have whatever org-level permissions are necessary to do this: I have Codecov enabled on my GitHub account and I can see Sigstore as one of my approved orgs, but none of the Sigstore repos show up in my dashboard.
Unblocked; turns out I was wrong about needing special permissions for Codecov. #253 has the changes.
Current local coverage is 68%:
================================ 81 passed, 2 skipped in 7.11s =================================
Name Stmts Miss Cover Missing
---------------------------------------------------------------------
sigstore/__init__.py 2 0 100%
sigstore/__main__.py 1 1 0% 15
sigstore/_internal/__init__.py 0 0 100%
sigstore/_internal/ctfe.py 49 12 76% 60, 103-127
sigstore/_internal/fulcio/__init__.py 3 0 100%
sigstore/_internal/fulcio/client.py 171 64 63% 75, 171-172, 176-181, 192-273, 279-293, 318, 324
sigstore/_internal/merkle.py 45 3 93% 98, 107, 127
sigstore/_internal/oidc/__init__.py 25 18 28% 33-63
sigstore/_internal/oidc/ambient.py 92 1 99% 97
sigstore/_internal/oidc/issuer.py 22 22 0% 15-52
sigstore/_internal/oidc/oauth.py 131 131 0% 15-269
sigstore/_internal/rekor/__init__.py 3 0 100%
sigstore/_internal/rekor/client.py 176 30 83% 152, 210, 264-269, 280-294, 303-323, 372-375, 410
sigstore/_internal/sct.py 67 28 58% 52-53, 56, 68, 77, 101, 128-130, 134-137, 152-209
sigstore/_internal/set.py 14 2 86% 46-47
sigstore/_sign.py 53 19 64% 65-126
sigstore/_store/__init__.py 0 0 100%
sigstore/_utils.py 44 12 73% 45-46, 49, 84-106
sigstore/_verify/__init__.py 4 0 100%
sigstore/_verify/models.py 57 1 98% 174
sigstore/_verify/policy.py 85 2 98% 69-70
sigstore/_verify/verifier.py 98 16 84% 188-189, 197, 206, 212, 225-226, 235-241, 251-252, 261-262, 270
---------------------------------------------------------------------
TOTAL 1142 362 68%
It's probably a bit higher in CI, due to tests with ambient credentials.