stackdriver-prometheus-sidecar icon indicating copy to clipboard operation
stackdriver-prometheus-sidecar copied to clipboard

Avoid retry on non-recoverable errors.

Open StevenYCChou opened this issue 6 years ago • 5 comments

Rationale behind this change:

in retrieval.manager(), retry with exponential backoff is used when building Stackdriver sample. However, there are 2 types of errors which will always be thrown, and retrying wouldn't help:

  1. Unknown metric suffix: https://github.com/Stackdriver/stackdriver-prometheus-sidecar/blob/bfe393828ef37496968bada6060b35c5d3dfec25/retrieval/transform.go#L96
  2. Unknown metric type: https://github.com/Stackdriver/stackdriver-prometheus-sidecar/blob/bfe393828ef37496968bada6060b35c5d3dfec25/retrieval/transform.go#L114

This goal of this change is to avoid retrying on these 2 types of errors. The implementation is to only retry on recoverable errors.

When I implemented this, I also found that seriesCache.refresh() will potentially throw errors which is mostly not recoverable, hence making sure we wouldn't retry on non-recoverable errors.

StevenYCChou avatar Mar 07 '19 01:03 StevenYCChou

Can you include the rationale for this change?

jkohen avatar Mar 07 '19 14:03 jkohen

Still work in progress - as I haven't provided rationale yet.

StevenYCChou avatar Mar 11 '19 15:03 StevenYCChou

@jkohen ready for review. please take a look on the new change. The rationale is updated in 1st comment of the PR.

StevenYCChou avatar Mar 12 '19 14:03 StevenYCChou

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

googlebot avatar Aug 18 '19 02:08 googlebot

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

googlebot avatar Aug 18 '19 05:08 googlebot