Feign metrics are incompatible with RestTemplate metrics when using Prometheus

[snussbaumer]

Describe the bug When a spring boot/cloud project has observations, metrics and prometheus enabled with Feign, the http_client_requests_active metric registered by Feign is not compatible with the one registered by notably RestTemplate (maybe others). This is a problem as soon as you have for exemplle a DiscoveryClient like EurekaDiscoveryClient that uses RestTemplates under the hood.

This results in the following warning when first calling a method of the FeignClient :

The meter (MeterId{name='http.client.requests.active', tags=[tag(clientName=com.example.demo.TestFeignClient),tag(http.method=GET),tag(http.status_code=CLIENT_ERROR),tag(http.url=/outside/test)]}) registration has failed: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'http_client_requests_active_seconds' containing tag keys [client_name, exception, method, outcome, status, uri]. The meter you are attempting to register has keys [clientName, http_method, http_status_code, http_url]. Note that subsequent logs will be logged at debug level.

Consequently, et more importantly, the feign metrics are not exported by prometheus.

The metrics are effectively not compatible :

RestTemplate http_client_requests_active_seconds_count :
http_client_requests_active_seconds_count{client_name="localhost",exception="none",method="GET",outcome="UNKNOWN",status="CLIENT_ERROR",uri="/test"} 0
http_client_requests_active_seconds_sum{client_name="localhost",exception="none",method="GET",outcome="UNKNOWN",status="CLIENT_ERROR",uri="/test"} 0.0

FeignClient http_client_requests_active_seconds_count : 
http_client_requests_active_seconds_count{clientName="com.example.demo.TestFeignClient",http_method="GET",http_status_code="CLIENT_ERROR",http_url="/outside/test"} 0
http_client_requests_active_seconds_sum{clientName="com.example.demo.TestFeignClient",http_method="GET",http_status_code="CLIENT_ERROR",http_url="/outside/test"} 0.0

The FeignClient metric tags should probly look like this instead :

http_client_requests_active_seconds_count{client_name="com.example.demo.TestFeignClient",method="GET",status="CLIENT_ERROR",uri="/outside/test"} 0
http_client_requests_active_seconds_sum{client_name="com.example.demo.TestFeignClient",method="GET",status="CLIENT_ERROR",uri="/outside/test"} 0.0

Sample I have a simple repro here

Just run the tests and they will fail. If you replace the restTemplateBuilder.build() by new RestTemplate() the test passes. Demonstrating that the problem effectively comes from the restTemplate being observerd.

Below the classes in the project :

@SpringBootApplication
@RestController
@EnableFeignClients
@RequiredArgsConstructor
public class TestApplication {

    private final TestFeignClient testFeignClient;

    @GetMapping("/test")
    public String test() {
        return testFeignClient.test();
    }

    public static void main(String[] args) {
        SpringApplication.run(TestApplication.class, args);
    }
}

@FeignClient(value = "TestFeignClient", name = "TestFeignClient", url = "http://localhost:9004")
public interface TestFeignClient {

    @GetMapping("/outside/test")
    String test();

}

@SpringBootTest(classes = TestApplication.class, webEnvironment = WebEnvironment.DEFINED_PORT, properties = {
        "spring.application.name=demo",
        "server.port=18080",
        "management.tracing.enabled=true",
        "management.prometheus.metrics.export.enabled=true",
        "management.endpoints.web.exposure.include=prometheus"
})
@WireMockTest(httpPort = 9004)
class DemoTestApplicationTests {

    @Autowired
    private RestTemplateBuilder restTemplateBuilder;

    @Test
    void repro() {
        WireMock.stubFor(get(urlPathEqualTo("/outside/test")) //
                .willReturn(aResponse() //
                        .withStatus(200) //
                        .withHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE) //
                        .withBody("hello world")));

        RestTemplate restTemplate = restTemplateBuilder.build(); // replace by "new RestTemplate()" for test to pass
        String response = restTemplate.getForObject("http://localhost:18080/test", String.class);
        assertAll(
                () -> assertThat(response).isEqualTo("hello world"),
                () -> assertThat(logAppender.list)
                        .extracting(ILoggingEvent::getMessage)
                        .noneMatch(msg -> msg.contains("registration has failed")),
                () -> assertThat(restTemplate.getForObject("http://localhost:18080/actuator/prometheus", String.class))
                        .contains("http_client_requests_active_seconds_count{clientName=\"com.example.demo.TestFeignClient")
        );
    }

    // setup of log appender to capture logs

    private ListAppender<ILoggingEvent> logAppender;

    private Logger logger;

    @BeforeEach
    void setUp() {
        logger = (Logger) LoggerFactory.getLogger(io.micrometer.prometheusmetrics.PrometheusMeterRegistry.class);
        logAppender = new ListAppender<>();
        logAppender.start();
        logger.addAppender(logAppender);
    }

    @AfterEach
    void tearDown() {
        logger.detachAppender(logAppender);
    }

}

[snussbaumer]

~maybe linked to https://github.com/spring-cloud/spring-cloud-openfeign/issues/1174 ?~ (nope that ticket speaks about AOT)

in any case if you point me to how to change that I'd be willing to submit a PR.

[snussbaumer]

Workaround (ugly ...) create a MeterFilter that overrides "map" and either renames the Meter, either makes the tag identical with the ones from RestTemplate.