google-cloud-java icon indicating copy to clipboard operation
google-cloud-java copied to clipboard

IT Failure(google-cloud-compute): ITPaginationTest.testPaginationIterating zone not found

Open emmileaf opened this issue 2 years ago • 5 comments

Failures observed in GraalVM nightly build and presubmit. This error is happening often (in a number of presubmits and re-triggers), but not consistently:

Problematic test - com.google.cloud.compute.v1.integration.ITPaginationTest.testPaginationIterating

Failures (1):
  JUnit Vintage:ITPaginationTest:testPaginationIterating
    MethodSource [className = 'com.google.cloud.compute.v1.integration.ITPaginationTest', methodName = 'testPaginationIterating', methodParameterTypes = '']
    => java.lang.AssertionError: Zone us-central1-a was not found for cloud-java-ci-test in zones list (size: 65).
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       com.google.cloud.compute.v1.integration.ITPaginationTest.testPaginationIterating(ITPaginationTest.java:124)
       [email protected]/java.lang.reflect.Method.invoke(Method.java:568)
       org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
       org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
       org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
       org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
       org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
       org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
       [...]

This could be a similar issue with interfering compute resources from integration tests as observed/addressed in https://github.com/googleapis/google-cloud-java/pull/8260.

emmileaf avatar Feb 17 '23 14:02 emmileaf

Update - seems like this is a previously observed flake coming back: https://github.com/googleapis/java-compute/issues/683, https://github.com/googleapis/java-compute/pull/682#issuecomment-1108555217 look relevant.

emmileaf avatar Feb 17 '23 17:02 emmileaf

An initial investigation from locally reproducing this error w/ Terraform:

gcloud compute zones list returns 119 zones, which also consistent with the response.iterateAll() size from testList().

In testPaginationIterating(), iterating over zones sometimes gets stuck at around this point:

...
Zone: europe-west9-b
Zone: europe-west9-a
Zone: me-central1-b
Zone: me-central1-c

response.IterateAll() size: 46

And sometimes able to get a full list including us-central1-a (DEFAULT_ZONE):

...
Zone: europe-west9-b
Zone: europe-west9-a
Zone: me-west1-a
Zone: me-west1-c
Zone: me-west1-b
...
Zone: us-central1-a
...

emmileaf avatar Feb 17 '23 23:02 emmileaf

The nightly builds are back to green, and I am no longer able to reproduce this error 🤔. Leaving this issue open to track and revisit if it resurfaces again.

emmileaf avatar Feb 21 '23 21:02 emmileaf

Closing this issue now. It's been ~3 months since this last flaked. We can re-open if this test flakes again.

lqiu96 avatar Apr 25 '23 14:04 lqiu96

Observed this flake again in nightly build - reopening to track.

emmileaf avatar Jun 09 '23 14:06 emmileaf