StackOverflowError when accessing current span

[adamleantech]

StackOverflowError observed when running a Spring Boot 3.0.2 app with micrometer tracing/metrics and logback metrics enabled - it appears to be linked to lazy loading of the Tracer within LazyTracingSpanContextSupplier which results in a recursive loop of logging and counter incrementing.

Disabling logback metrics in application.yml resolves the issue

Spring boot version 3.0.2

pom.xml snippet

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>3.0.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
            ...
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
        </dependency>

stack trace snippet

        at org.springframework.boot.actuate.autoconfigure.tracing.prometheus.PrometheusExemplarsAutoConfiguration$LazyTracingSpanContextSupplier.currentSpan(PrometheusExemplarsAutoConfiguration.java:91)
        at org.springframework.boot.actuate.autoconfigure.tracing.prometheus.PrometheusExemplarsAutoConfiguration$LazyTracingSpanContextSupplier.isSampled(PrometheusExemplarsAutoConfiguration.java:82)
        at io.prometheus.client.exemplars.DefaultExemplarSampler.doSample(DefaultExemplarSampler.java:46)
        at io.prometheus.client.exemplars.DefaultExemplarSampler.sample(DefaultExemplarSampler.java:34)
        at io.micrometer.prometheus.PrometheusCounter.updateExemplar(PrometheusCounter.java:79)
        at io.micrometer.prometheus.PrometheusCounter.increment(PrometheusCounter.java:58)
        at io.micrometer.core.instrument.Counter.increment(Counter.java:38)
        at io.micrometer.core.instrument.binder.logging.MetricsTurboFilter.decide(LogbackMetrics.java:205)
        at ch.qos.logback.classic.spi.TurboFilterList.getTurboFilterChainDecision(TurboFilterList.java:49)
        at ch.qos.logback.classic.LoggerContext.getTurboFilterChainDecision_0_3OrMore(LoggerContext.java:251)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:375)
        at ch.qos.logback.classic.Logger.log(Logger.java:780)
        at org.apache.commons.logging.LogAdapter$Slf4jLocationAwareLog.trace(LogAdapter.java:473)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:256)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1628)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1585)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1368)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.getObject(DefaultListableBeanFactory.java:2002)
        at org.springframework.util.function.SingletonSupplier.get(SingletonSupplier.java:97)
        at org.springframework.util.function.SingletonSupplier.obtain(SingletonSupplier.java:115)
        at org.springframework.boot.actuate.autoconfigure.tracing.prometheus.PrometheusExemplarsAutoConfiguration$LazyTracingSpanContextSupplier.currentSpan(PrometheusExemplarsAutoConfiguration.java:91)

[wilkinsona]

Thanks for the report. I've asked the observability team to take a look.

[jonatan-ivanov]

It's not in the repro steps but out of the box this works to me (in many sample/example/test apps). In order to reproduce the issue, I needed to make the app log something out triggered by the PrometheusExemplarsAutoConfiguration$LazyTracingSpanContextSupplier.currentSpan call.

For example, if I set logging.level.org.springframework.beans.factory.support=TRACE, I am able to repro the issue, without it, it works.

We are looking into how to fix it, in the meantime another workaround could be modifying some of your log levels.

[wilkinsona]

This is going to be addressed in Micrometer. Please subscribe to https://github.com/micrometer-metrics/micrometer/issues/3623 for updates.