cloudbeat icon indicating copy to clipboard operation
cloudbeat copied to clipboard

Account for rate limiting in GCP fetchers

Open orouz opened this issue 1 year ago • 0 comments

Motivation

GCP fetchers mainly use the assets inventory api via the ListAssets method, which has the following quotas:

  1. 100 per minute per consumer project
  2. 800 per minute per organization
  3. 650,000 per day per organization

for both our use cases - single-account and organization-account, we always use the same project to consume the quota, as we set it once, when deploying an agent. (can be verified with gcloud config get billing/quota_project). this means we can use just the per-project quota for rate limiting.

Definition of done

  • GCP assets inventory's ListAssets method usage does not exceed the default quota (100 per minute per project)

Out of scope

  • synced cloudbeat instances consuming the same quota

orouz avatar Apr 01 '24 08:04 orouz