prometheus-engine icon indicating copy to clipboard operation
prometheus-engine copied to clipboard

Additional buckets getting inserted with sum & rate/increase for histograms

Open clupprich opened this issue 3 years ago • 1 comments

We're experiencing a weird issue with histogram buckets on a few of our GCP projects. The following query returns a different set of buckets on different projects, even though the config is the same:

sum(increase(nginx_ingress_controller_request_duration_seconds_bucket{exported_namespace=~"(production|staging)-receiving"}[5m])) by (le)

Running the above query on GCP project A returns the following buckets (note the added buckets):

CleanShot 2022-09-07 at 14 39 20@2x

Running it on GCP projects B + C returns the following buckets (note the additional buckets that get inserted):

CleanShot 2022-09-07 at 14 38 32@2x

This is only happening for a combination of sum(increase(bucket)) by (le) and sum(rate(bucket)) by (le), but not for sum(irate(bucket)) by (le).

Any idea what might be going on here?

clupprich avatar Sep 07 '22 19:09 clupprich

Could you share your project numbers with us so we can investigate? You can email them to me at (my github username) at (the company I work for) .com

lyanco avatar Sep 09 '22 16:09 lyanco