cloudbeat icon indicating copy to clipboard operation
cloudbeat copied to clipboard

Account for rate limiting in AWS fetchers

Open orouz opened this issue 1 year ago • 0 comments

Motivation we need to account for rate limiting in our AWS fetchers to avoid losing resources we want to evaluate. each AWS fetcher uses a different client, as opposed to GCP's assets inventory or Azure's graph explorer. because of that, we need to check each fetcher separately, and evaluate its usage to understand the api quota limitations.

for example, calling DescribeBuckets using the S3 client has no quota (AFAIK). but calling ListKeys using the KMS client, has a quota of 500 requests per second

in addition, we may need to use the servicequotas package to get the current user's quotas.

Definition of done

  • figure out the quotas for each AWS fetcher
  • when applicable, AWS fetchers method usage does not exceed the default quota
  • add a retry (with backoff) mechanism for failed requests

Out of scope

  • synced cloudbeat instances consuming the same quota

orouz avatar Apr 01 '24 08:04 orouz