Getting error "Status{code=UNAVAILABLE, description=io exception failed: Connection reset"

[thanajpa]

We got the error for a week and Google Support API escalate to this channel. could you please help to investigate?

"Status{code=UNAVAILABLE, description=io exception, cause=io.grpc.netty.shaded.io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset"

v10 request logs - REDACTED.txt

[jradcliff]

Hi,

The result set for that request is going to contain over 750,000 rows, so it will be extremely large. A few questions I have are:

1. I see that you get some results back but then the request fails before all results are returned. How long does it take for the request to fail? Are you setting any custom timeouts when making the request?
2. Do you need to retrieve all of the ads in the account? If not, please try filtering to only the ads you need. If you do need the entire set of ads, try breaking up the request into multiple requests. For example, you could batch requests by campaign.id by adding campaign.id IN (a,b,c) to the WHERE clause in one request, then campaign.id IN (d,e,f) in the next request, and so on.
3. Do you need to retrieve all of those fields for all 750,000+ ads? Reducing the list of requested fields would reduce the size of the responses, so you should only request the fields you need.

Thanks, Josh, Google Ads API Team

[thanajpa]

We want to download all those and the fields are mandatory to download. so please share an example of setting customer timeout.

[jradcliff]

Hi,

Could you provide an answer to my first question of how long it takes for requests to fail?

Thanks

[thanajpa]

Hi,

Could you provide an answer to my first question of how long it takes for requests to fail?

Thanks

around 10-12 mins

[jradcliff]

Hi,

We're looking into reducing these UNAVAILABLE errors, but in the meantime I recommend retrying with an exponential backoff policy when you encounter UNAVAILABLE.

You can adjust the timeout settings for your requests using the approach in our Advanced usage guide.

Thanks

[thanajpa]

retrying with an exponential backoff

We're adding the retry and timeout code but didn't work. so we're trying to reduce fields and downloads per each ad type. We still got the problem with the responsive ad.

GrpcCallContext
     .createDefault()
      .withTimeout(Duration.of(60, ChronoUnit.MINUTES))
      .withRetrySettings(
        RetrySettings
          .newBuilder()
          .setInitialRetryDelay(Duration.ofMillis(10L))
          .setMaxRetryDelay(Duration.ofSeconds(10L))
          .setRetryDelayMultiplier(1.4)
          .setMaxAttempts(10)
          .setLogicalTimeout(Duration.ofMinutes(30L))
          .build()
      )

responsive request logs.txt

[thanajpa]

I've tried to request with REST, so it's seems processing stuck and didn't throw the exception

[devchas]

@thanajpa that makes sense because this issue seems to be specifically related to grpc. In doing some research, it seems like this is most likely a network connection issue (e.g. the connection was dropped mid-stream). Per the grpc doc, status code UNAVAILABLE means that it is a retryable error, so I agree with Josh's suggestion of using exponential backoff.

While it would be slower, especially considering the size of the result set, you could also consider using paged search instead of searchStream. While I wouldn't necessarily recommend this approach for large requests in general, it could function as a workaround if this issue persists because you can pick up your search request even with a dropped network connection using page tokens like in this example.

[jradcliff]

We made several improvements to the API since this was last reported, so I'll close this issue, but please reopen with new details if you're still encountering this problem.

Thanks, Josh, Google Ads API Team