Explain difference between http.server.requests.max (gauge) vs http.server.requests (histogram's "max")

[artemik]

Describe the bug

On a Spring Boot app with Micrometer instrumentation, with all the defaults, OpenTelemetry outputs the following metrics:

• http.server.requests histogram (who has "max");
• http.server.requests.max gauge.

The gauge is apparently coming from here: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/9ecf7965aa455d41ea8cc0761b6c6b6eeb106324/instrumentation/micrometer/micrometer-1.5/library/src/main/java/io/opentelemetry/instrumentation/micrometer/v1_5/OpenTelemetryTimer.java#L80

2024-07-17T16:42:06.275Z  INFO 11576 --- [backend] [cMetricReader-1] i.o.e.logging.LoggingMetricExporter      : metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={host.arch="amd64", host.name="DESKTOP", os.description="Windows 10 10.0", os.type="windows", process.command_line="C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:11799,suspend=y,server=n -Dotel.instrumentation.micrometer.enabled=true -Dotel.instrumentation.jdbc-datasource.enabled=true -Dotel.instrumentation.spring-webmvc.enabled=true -Dotel.metrics.exporter=console -Dotel.jmx.target.system=tomcat -XX:TieredStopAtLevel=1 -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -Dmanagement.endpoints.jmx.exposure.include=* -javaagent:C:\Users\Artem\AppData\Local\JetBrains\IntelliJIdea2024.1\captureAgent\debugger-agent.jar=file:/C:/Users/Artem/AppData/Local/Temp/capture.props -Dfile.encoding=UTF-8 com.backend.BackendApplication", process.executable.path="C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe", process.pid=11576, process.runtime.description="Eclipse Adoptium OpenJDK 64-Bit Server VM 17.0.10+7", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="17.0.10+7", service.instance.id="e5198df6-aa26-469a-a58a-19c99409ac27", service.name="backend", telemetry.distro.name="opentelemetry-spring-boot-starter", telemetry.distro.version="2.5.0-alpha", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.39.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.micrometer-1.5, version=null, schemaUrl=null, attributes={}}, name=http.server.requests, description=, unit=s, type=HISTOGRAM, data=ImmutableHistogramData{aggregationTemporality=CUMULATIVE, points=[ImmutableHistogramPointData{getStartEpochNanos=1721233445765907200, getEpochNanos=1721234526244083100, getAttributes={error="none", exception="none", method="GET", outcome="SUCCESS", status="200", uri="/rest/services/onboarding"}, getSum=0.12368740099999999, getCount=3, hasMin=true, getMin=0.009611, hasMax=true, getMax=0.100249601, getBoundaries=[], getCounts=[3], getExemplars=[]}, ...]}}

2024-07-17T16:42:06.260Z  INFO 11576 --- [backend] [cMetricReader-1] i.o.e.logging.LoggingMetricExporter      : metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={host.arch="amd64", host.name="DESKTOP", os.description="Windows 10 10.0", os.type="windows", process.command_line="C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:11799,suspend=y,server=n -Dotel.instrumentation.micrometer.enabled=true -Dotel.instrumentation.jdbc-datasource.enabled=true -Dotel.instrumentation.spring-webmvc.enabled=true -Dotel.metrics.exporter=console -Dotel.jmx.target.system=tomcat -XX:TieredStopAtLevel=1 -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -Dmanagement.endpoints.jmx.exposure.include=* -javaagent:C:\Users\Artem\AppData\Local\JetBrains\IntelliJIdea2024.1\captureAgent\debugger-agent.jar=file:/C:/Users/Artem/AppData/Local/Temp/capture.props -Dfile.encoding=UTF-8 com.backend.BackendApplication", process.executable.path="C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe", process.pid=11576, process.runtime.description="Eclipse Adoptium OpenJDK 64-Bit Server VM 17.0.10+7", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="17.0.10+7", service.instance.id="e5198df6-aa26-469a-a58a-19c99409ac27", service.name="backend", telemetry.distro.name="opentelemetry-spring-boot-starter", telemetry.distro.version="2.5.0-alpha", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.39.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.micrometer-1.5, version=null, schemaUrl=null, attributes={}}, name=http.server.requests.max, description=, unit=s, type=DOUBLE_GAUGE, data=ImmutableGaugeData{points=[ImmutableDoublePointData{startEpochNanos=1721233445765907200, epochNanos=1721234526244083100, attributes={error="none", exception="none", method="GET", outcome="SUCCESS", status="200", uri="/rest/services/onboarding"}, value=0.0, exemplars=[]}, ...]}}

The difference between the "max"es seems to be:

• the max from "http.server.requests" histogram is a total (application lifespan) max;
• the "http.server.requests.max" gauge is a time window max.

It seems that a time window gauge metric gets added to all other histogram metrics as well.

Steps to reproduce

Expected behavior

1. Can anybody explain why OpenTelemetry Micrometer instrumentation adds a time window "max" gauge?
2. The "http.server.requests.max" time window gauge becomes "http_server_requests_max" in prometheus and therefore overrides the histogram's max value because it would have the same name, as I understand. So it's impossible (out of the box) to get histogram's max value (in case anybody would need it). Is it on purpose? Is it going to stay like that in future?
3. Such setup is very confusing. Can there be some documentation explaining that it's happening and why?

I'm creating it as a bug because it's not clear if it's a documentation issue only and it pertains to Java instrumentation specifically.

Actual behavior

Javaagent or library instrumentation version

2.5.0

Environment

JDK: 17 OS: Windows 10

Additional context

No response

[trask]

hi @artemik! I dug around and found where the feature was added, can you check out #5303 and the discussion in #5292 to see if that helps at all?

[breedx-splk]

Hey @artemik did you see @trask 's comment above? Did that help resolve your issue?

[artemik]

@breedx-splk I'm sorry it's been some time ago, and it's hard to get into context right now. I understand there are referenced issues, but there has been long discussions, not everything necessarily is relevant, so maybe somebody could provide a gist to my questions in the ticket?

[breedx-splk]

I'm confused. There's a ticket?

[github-actions[bot]]

This issue has been labeled as stale due to lack of activity and needing author feedback. It will be automatically closed if there is no further activity over the next 7 days.