Histogram buckets rendered differently than in other languages

[Dlougach]

Bucket boundaries {1, 2, 3} would be rendered as "1.0", "2.0", "3.0" in Go/Python, but as "1", "2", and "3" in this library. This causes some strange effects when you collecting the same metric from applications written in various languages.

[Dlougach]

Some more context on what happens when there is inconsistency is here: https://github.com/prometheus/prometheus/issues/8367

[gjasny]

Dealing with floating point numbers makes me drink. I'll take a look.

[gjasny]

In your golang example: did you set EnableOpenMetrics to true? Could you please share the http request / response including headers?

[gjasny]

That one: https://github.com/prometheus/client_golang/blob/8184d76b3b0bd3b01ed903690431ccb6826bf3e0/prometheus/promhttp/http.go#L356

[Dlougach]

In your golang example: did you set EnableOpenMetrics to true? Could you please share the http request / response including headers?

I am using Prometheus python client in the other app. It uses the following logic to convert to strings (apparently to replicate Go's version): https://github.com/prometheus/client_python/blob/master/prometheus_client/utils.py

I didn't set EnableOpenMetrics to true anywhere, but as far as I can tell it doesn't change anything.

I can share full HTTP request/response, but would need to clean any proprietary hints from there. Overall I don't think that it's going to help much with the debugging IMO, because the issue is very obvious: printf with "%.*g" or to_chars will not produce a decimal point for integers, unlike the typical Go formatting.

[gjasny]

I had a look at tho golang code (which I consider authoritative). If I explicitly enable EnableOpenMetrics the behavior is the following if either no explicit or the v0.0.4 text format is requested:

$ curl -H "Accept: text/plain;version=0.0.4" -v  http://localhost:2112/metrics

go_threads 11
duckpond_temperature_celsius_bucket{le="0"} 0
duckpond_temperature_celsius_bucket{le="0.5"} 0
duckpond_temperature_celsius_bucket{le="1"} 1
duckpond_temperature_celsius_bucket{le="1.5"} 1

In case I request OpenMetrics format:

$ curl -H "Accept: application/openmetrics-text; version=0.0.1" -v  http://localhost:2112/metrics

go_threads 10.0
duckpond_temperature_celsius_bucket{le="0.0"} 0
duckpond_temperature_celsius_bucket{le="0.5"} 0
duckpond_temperature_celsius_bucket{le="1.0"} 1
duckpond_temperature_celsius_bucket{le="1.5"} 1

Responsible for float rendering is writeFloat or writeOpenMetricsFloat.

Right now the prometheus-cpp generated metrics seems to be in-line with the golang exporter. Unfortunately, this does not solve your problem. Maybe we should make the behavior configurable - paired with some good documentation.

I also noticed that the OpenMetrics exporter does not use the openmetrics float format for everything (see the histogram bucket counters above). That might make the logic somewhat icky.

In the long run we should add support for OpenMetrcis. But that involves more than just the floating point rendering.

@beorn7 @brian-brazil Does that all make sense? The test-format-details don't talk about that issue at all.

Thanks, Gregor

[brian-brazil]

In case I request OpenMetrics format:

The label value should not be changing based on the content type anyway, it sounds like this should be controlled only using EnableOpenMetrics here.

[gjasny]

The label value should not be changing based on the content type anyway, it sounds like this should be controlled only using EnableOpenMetrics here.

But that's what's happening: if EnableOpenMetrics is true HandlerFor calls NegotiateIncludingOpenMetrics which returns the format that is passed into NewEncoder.

It's that way since https://github.com/prometheus/common/pull/219 / 1.9.0.