spring-boot-admin icon indicating copy to clipboard operation
spring-boot-admin copied to clipboard

gateway timeout on long running JMX request

Open clockrun opened this issue 3 years ago • 4 comments

We have some JMX operations, which will take several minutes to complete. With version 2.6.7, calling these operations will return HTTP 500 but code will continue executing until complete. But with version 2.7.4, calling these operations will return HTTP 504 and then execution stops.

Below commit seems to be related. https://github.com/codecentric/spring-boot-admin/commit/73fadf6dbeb9c0559b1d938d56ab21bb8afb6710

clockrun avatar Aug 26 '22 07:08 clockrun

@clockrun, calls via the instanceProxy to managed operations of MBeans are handled like any other request. So a workaround would be to increase the general timeout configuration of the underlying instanceWebProxy inside the client (spring.boot.admin.client.read-timeout - the default is 5000ms).

ulischulte avatar Aug 26 '22 11:08 ulischulte

Hi Ulishulte, Unfortunately this doesn't work. After set this value manually, I cannot see its key in property list from spring boot admin. Besides, it didn't take effect after calling JMX operation again.

BTW, I'm using spring-boot-admin with kubernetes. Spring-boot-admin-client is not included. I'm also using config map as property source. Will this be relavent?

Regards, Clockrun

clockrun avatar Aug 27 '22 15:08 clockrun

@clockrun, the property has to be configured inside a client application's configuration (the client applications which register themself to SBA), not inside SBA itself. Regards Uli

ulischulte avatar Aug 28 '22 09:08 ulischulte

@clockrun , Did you apply the previous suggestion and did it solve the problem? Regards, Uli

ulischulte avatar Sep 02 '22 06:09 ulischulte

Hi Ulischulte, We are using spring cloud discovery together with kubernetes to locate spring boot application (via annotation on service). We are not using spring boot admin client. Then how can I apply such configuration in our case?

Regards, Clockrun

clockrun avatar Oct 03 '22 13:10 clockrun

Hi @clockrun,

OK, I see. You're still able to configure timeouts for specific endpoints on the admin side (@see https://codecentric.github.io/spring-boot-admin/current/#_configuration_options): For instance spring.boot.admin.monitor.timeout.jolokia=10000

Regards, Uli

ulischulte avatar Oct 07 '22 07:10 ulischulte

Seems not working for me. I set environment variable SPRING_BOOT_ADMIN_MONITOR_TIMEOUT_JOLOKIA to value 1800000, but it still get time out immediately.

clockrun avatar Oct 08 '22 16:10 clockrun

Hi @clockrun,

we have tried to reproduce the issue but weren’t successful. We have implemented a long running JMX operation (2min) that fails using the default timeouts throwing an error 504 as you stated. When we increase the server-side time out as written by @ulischulte, the operation is executed correctly and works like a charm.

Is there any other actor that might close the request due to a timeout (e.g. the target server, tomcat or something else)? Please check if they have configured a connectionTimeout or similar that blocks the request.

Regards, SteKoe

SteKoe avatar Oct 21 '22 07:10 SteKoe

Closed due to missing feedback. If your problem still occurs, please open a new issue.

ulischulte avatar Dec 09 '22 07:12 ulischulte