Grafeas unit tests are flaky

[priyawadhwa]

The grafeas [pkg/chains/storage/grafeas TestBackend_ListOccurrences] unit test is flaky:

    grafeas_test.go:85: Wrong list of occurrences received for empty filter, got=  (*grafeas_go_proto.ListOccurrencesResponse)(Inverse(protocmp.Transform, protocmp.Message{
          }))

an example failure log provided.

cc @chuangw6 who added this in, do you have any ideas why it might be flaky?

[chuangw6]

Hi @priyawadhwa , Thanks for reporting this. Just digged into this and found out it's because when retrieving all occurrences stored as a map in the fake server, the retrieve order sometimes is not same as the insertion order (as Go maps do not maintain the insertion order). So I added the code to sort all occurrences by resource uri when calling ListOccurrences. This will make sure the test is not flaky anymore.

[savitaashture]

Hi @chuangw6 and @priyawadhwa even we see chains nightly failing here https://dashboard.dogfooding.tekton.dev/#/namespaces/tekton-nightly/pipelineruns/chains-release-nightly-5wt2l?pipelineTask=unit-tests&step=unit-test because of flakiness

[priyawadhwa]

Thank you @chuangw6 for looking into it! @savitaashture hopefully tonight's build passes now that the fix is about to be merged 🤞🏽

[chuangw6]

Yeah, now it should fix the issue. From the log, I see it failed because of the same issue I just mentioned. @savitaashture Please let me know if you see any further issue.

[dprotaso]

This consistently fails here: https://github.com/knative/pkg/runs/6265134542?check_suite_focus=true

All that check is doing is testing the latest knative.dev/pkg with the latest chains version

--- FAIL: TestInitializeBackends (0.01s)
    logger.go:130: 2022-05-02T22:24:40.280Z	DEBUG	storage/storage_test.go:81	Backend: map[]
    logger.go:130: 2022-05-02T22:24:40.280Z	DEBUG	storage/storage_test.go:81	Backend: map[tekton:0xc000710348]
    --- FAIL: TestInitializeBackends/gcs (0.01s)
        storage_test.go:78: InitializeBackends() error = dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    logger.go:130: 2022-05-02T22:24:40.287Z	DEBUG	storage/storage_test.go:81	Backend: map[oci:0xc000631a40]
    --- FAIL: TestInitializeBackends/grafeas (0.00s)
        storage_test.go:78: InitializeBackends() error = google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    --- FAIL: TestInitializeBackends/multi (0.00s)
        storage_test.go:78: InitializeBackends() error = dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    logger.go:130: 2022-05-02T22:24:40.288Z	DEBUG	storage/storage_test.go:81	Backend: map[pubsub:0xc00016ed00]
FAIL
FAIL	github.com/tektoncd/chains/pkg/chains/storage	0.174s

[chuangw6]

This consistently fails here: https://github.com/knative/pkg/runs/6265134542?check_suite_focus=true

All that check is doing is testing the latest knative.dev/pkg with the latest chains version

--- FAIL: TestInitializeBackends (0.01s)
    logger.go:130: 2022-05-02T22:24:40.280Z	DEBUG	storage/storage_test.go:81	Backend: map[]
    logger.go:130: 2022-05-02T22:24:40.280Z	DEBUG	storage/storage_test.go:81	Backend: map[tekton:0xc000710348]
    --- FAIL: TestInitializeBackends/gcs (0.01s)
        storage_test.go:78: InitializeBackends() error = dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    logger.go:130: 2022-05-02T22:24:40.287Z	DEBUG	storage/storage_test.go:81	Backend: map[oci:0xc000631a40]
    --- FAIL: TestInitializeBackends/grafeas (0.00s)
        storage_test.go:78: InitializeBackends() error = google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    --- FAIL: TestInitializeBackends/multi (0.00s)
        storage_test.go:78: InitializeBackends() error = dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
    logger.go:130: 2022-05-02T22:24:40.288Z	DEBUG	storage/storage_test.go:81	Backend: map[pubsub:0xc00016ed00]
FAIL
FAIL	github.com/tektoncd/chains/pkg/chains/storage	0.174s

From looking at the logs, it seems not related to tests of ListOccurrences , and not even related to grafeas support because TestInitializeBackends/gcs test also throws error.

I guess it might be because your github actions don't set any google application default credentials. And the tests failed, which indicated by the message like google: could not find default credentials.

[dprotaso]

weird - email reply didn't go through

I guess it might be because your github actions don't set any google application default credentials

That makes sense - but why would a unit test require this?

[chuangw6]

weird - email reply didn't go through

I guess it might be because your github actions don't set any google application default credentials

That makes sense - but why would a unit test require this?

Hey @dprotaso , sorry for the delay. Just saw @imjasonh disabled the unit tests (for gcs backend storage creation) that require setting up credentials in #438.

I also disabled the unit tests for grafeas backend storage creation in #446. Hope these two prs will get rid of issues caused by missing credential setup. Thanks.

[tekton-robot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale with a justification. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten with a justification. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen with a justification. Mark the issue as fresh with /remove-lifecycle rotten with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen with a justification. Mark the issue as fresh with /remove-lifecycle rotten with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dprotaso]

These are unit tests - so I’m surprised by that expectation

On Mon, May 2, 2022 at 21:07 Chuang Wang @.***> wrote:

This consistently fails here: https://github.com/knative/pkg/runs/6265134542?check_suite_focus=true

All that check is doing is testing the latest knative.dev/pkg with the latest chains version

--- FAIL: TestInitializeBackends (0.01s) logger.go:130: 2022-05-02T22:24:40.280Z DEBUG storage/storage_test.go:81 Backend: map[] logger.go:130: 2022-05-02T22:24:40.280Z DEBUG storage/storage_test.go:81 Backend: map[tekton:0xc000710348] --- FAIL: TestInitializeBackends/gcs (0.01s) storage_test.go:78: InitializeBackends() error = dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. logger.go:130: 2022-05-02T22:24:40.287Z DEBUG storage/storage_test.go:81 Backend: map[oci:0xc000631a40] --- FAIL: TestInitializeBackends/grafeas (0.00s) storage_test.go:78: InitializeBackends() error = google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. --- FAIL: TestInitializeBackends/multi (0.00s) storage_test.go:78: InitializeBackends() error = dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. logger.go:130: 2022-05-02T22:24:40.288Z DEBUG storage/storage_test.go:81 Backend: map[pubsub:0xc00016ed00] FAIL FAIL github.com/tektoncd/chains/pkg/chains/storage 0.174s

From looking at the logs, it seems not related to tests of ListOccurrences , and not even related to grafeas support because TestInitializeBackends/gcs test also throws error.

I guess it might be because your github actions don't set any google application default credentials. And the tests failed, which indicated by the message like

--- FAIL: TestInitializeBackends/gcs (0.01s)
    storage_test.go:78: InitializeBackends() error = dialing: **google: could not find default credentials.** See https://developers.google.com/accounts/docs/application-default-credentials for more information.

— Reply to this email directly, view it on GitHub https://github.com/tektoncd/chains/issues/426#issuecomment-1115516150, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAERAX6MDY62AQ7DMUR3JDVIB33ZANCNFSM5T7VCFKQ . You are receiving this because you commented.Message ID: @.***>

[dprotaso]

That's weird - I didn't post that unless I did it in my sleep.

Maybe it was a draft - that GitHub posted automatically