Additional buckets getting inserted with sum & rate/increase for histograms
We're experiencing a weird issue with histogram buckets on a few of our GCP projects. The following query returns a different set of buckets on different projects, even though the config is the same:
sum(increase(nginx_ingress_controller_request_duration_seconds_bucket{exported_namespace=~"(production|staging)-receiving"}[5m])) by (le)
Running the above query on GCP project A returns the following buckets (note the added buckets):
Running it on GCP projects B + C returns the following buckets (note the additional buckets that get inserted):
This is only happening for a combination of sum(increase(bucket)) by (le) and sum(rate(bucket)) by (le), but not for sum(irate(bucket)) by (le).
Any idea what might be going on here?
Could you share your project numbers with us so we can investigate? You can email them to me at (my github username) at (the company I work for) .com