spring-cloud
Broken error handling when using DefaultRestClient that respons with error status code
Describe the bug
With the changes of https://github.com/spring-cloud/spring-cloud-gateway/issues/3405 (introduced in version 4.1.4) the error handling does not work properly any more when using DefaultRestClient.
When the client returns any error status code (e.g. 400), the DefaultRestClient raises an org.springframework.web.client.ResourceAccessException when trying to read the response body which happens RestClientProxyExchange.
However the ResourceAccessException does not provide any accessible information about the status code other than the message text. Therefore we cannot handle it properly and only respond with 500 Internal Server Error by default.
A workaround is to use a different http client. But actually I would expect Spring Cloud Gateway to work out of the box with Spring's default client. However, you may have other opinions on this.
Sample
I'm about half an hour before my three week holiday starts, so unfortunately I'm not able to provide a reproducable example any more. I can do that when I get back, if that helps. All I can do for now is to provide the error stack trace:
2024-07-11T16:00:53.232+02:00 WARN 249503 --- [api-outages-proxy] [omcat-handler-0] d.e.a.p.e.ErrorEntityExceptionHandler : I/O error on POST request for "http://localhost:11112/api/outages": Server returned HTTP response code: 400 for URL: http://localhost:11112/api/outages
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:11112/api/outages": Server returned HTTP response code: 400 for URL: http://localhost:11112/api/outages
at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.createResourceAccessException(DefaultRestClient.java:575) ~[spring-web-6.1.10.jar:6.1.10]
at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchangeInternal(DefaultRestClient.java:498) ~[spring-web-6.1.10.jar:6.1.10]
at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchange(DefaultRestClient.java:465) ~[spring-web-6.1.10.jar:6.1.10]
at org.springframework.cloud.gateway.server.mvc.handler.RestClientProxyExchange.exchange(RestClientProxyExchange.java:42) ~[spring-cloud-gateway-server-mvc-4.1.4.jar:4.1.4]
at org.springframework.cloud.gateway.server.mvc.handler.ProxyExchangeHandlerFunction.handle(ProxyExchangeHandlerFunction.java:120) ~[spring-cloud-gateway-server-mvc-4.1.4.jar:4.1.4]
at org.springframework.cloud.gateway.server.mvc.handler.HandlerFunctions$LookupProxyExchangeHandlerFunction.handle(HandlerFunctions.java:107) ~[spring-cloud-gateway-server-mvc-4.1.4.jar:4.1.4]
We have hit this bug also - we have a backend we expect to be returning a 404, but after upgrading to 4.1.4 that 404 produces a 500 error to the client instead.
Following the flow of the code it looks like HttpURLConnection throws a FileNotFound exception on 404, which due to this line of code is now throwing in a different place than it previously would have, and the exception isn't handled from here correctly:
https://github.com/spring-cloud/spring-cloud-gateway/blame/656ab5aba4a25738e2e7af1a2bb7846667667e3b/spring-cloud-gateway-server-mvc/src/main/java/org/springframework/cloud/gateway/server/mvc/handler/RestClientProxyExchange.java#L51
This is a blocker for upgrading for us, and I hope it can be addressed soon. For now we'll stay on the older version.
Here is a minimal reproducable example:
application.properties:
spring.cloud.gateway.mvc.http-client.type=autodetect
Gateway Application:
@SpringBootApplication
public class GatewayDemoApplication {
public static void main(String[] args) {
SpringApplication.run(GatewayDemoApplication.class, args);
}
@Bean
public RouterFunction<ServerResponse> getRoute() {
return route().GET("/status/*", http("https://httpbin.org/")).build();
}
}
Example request resulting in http status code 500 instead of correct response code:
curl -i http://localhost:8080/status/400
Stacktrace:
2024-07-30T14:33:35.442+02:00 ERROR 201951 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://httpbin.org/status/400": Server returned HTTP response code: 400 for URL: https://httpbin.org/status/400] with root cause
java.io.IOException: Server returned HTTP response code: 400 for URL: https://httpbin.org/status/400
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1998) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[na:na]
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[na:na]
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[na:na]
at org.springframework.http.client.SimpleClientHttpRequest.executeInternal(SimpleClientHttpRequest.java:88) ~[spring-web-6.1.11.jar:6.1.11]
at org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70) ~[spring-web-6.1.11.jar:6.1.11]
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-6.1.11.jar:6.1.11]
at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchangeInternal(DefaultRestClient.java:492) ~[spring-web-6.1.11.jar:6.1.11]
at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchange(DefaultRestClient.java:465) ~[spring-web-6.1.11.jar:6.1.11]
at org.springframework.cloud.gateway.server.mvc.handler.RestClientProxyExchange.exchange(RestClientProxyExchange.java:42) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
at org.springframework.cloud.gateway.server.mvc.handler.ProxyExchangeHandlerFunction.handle(ProxyExchangeHandlerFunction.java:120) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
at org.springframework.cloud.gateway.server.mvc.handler.HandlerFunctions$LookupProxyExchangeHandlerFunction.handle(HandlerFunctions.java:107) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
at org.springframework.web.servlet.function.support.HandlerFunctionAdapter.handle(HandlerFunctionAdapter.java:108) ~[spring-webmvc-6.1.11.jar:6.1.11]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.1.11.jar:6.1.11]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.1.11.jar:6.1.11]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.1.11.jar:6.1.11]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.1.11.jar:6.1.11]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.1.11.jar:6.1.11]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.26.jar:10.1.26]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.springframework.cloud.gateway.server.mvc.filter.WeightCalculatorFilter.doFilter(WeightCalculatorFilter.java:229) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.springframework.cloud.gateway.server.mvc.filter.FormFilter.doFilter(FormFilter.java:93) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.1.11.jar:6.1.11]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.11.jar:6.1.11]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.1.11.jar:6.1.11]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.11.jar:6.1.11]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.1.11.jar:6.1.11]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.11.jar:6.1.11]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Any update on this? Am stuck here for some time, maybe downgrading to an older version?
But actually I would expect Spring Cloud Gateway to work out of the box with Spring's default client.
The default client in Spring Cloud Gateway WebMVC is the jdk HttpClient, not HttpURLConnection
@spencergibb I've forked the latest version of branch 4.1.x to test the changes so that I can continue testing my gateway.
However, seems like the issue isn't quite fixed yet. The ResourceAccessException gets thrown after an IO error.
Which occurs here:
try {
InputStream body = clientResponse.getBody(); // ---------HERE
// put the body input stream in a request attribute so filters can read it.
MvcUtils.putAttribute(request.getServerRequest(), MvcUtils.CLIENT_RESPONSE_INPUT_STREAM_ATTR, body);
}
catch (FileNotFoundException e) {
// if using SimpleClientHttpRequestFactory
return ServerResponse.notFound().build();
}
So the catch doesn't get called.
Which causes a ResourceAccessException later and returning the default TomCat error HTML page in the API response.
Maybe the following might help:
Before the program executes clientResponse.getBody() I do it manually in the debugger, which causes:
When executing it again in the debugger console:
It does work
I can even parse the error message:
Is there any alternative till this issue is resolved?
@mendiCap I don't get a ResourceAccessException, I get a FileNotFoundException. Can you tell me how to recreate your specific situation?
Currently I'm testing a 409.
Do you know of a previous version this would work?
@spencergibb I managed to fix it with the following added catch method:
try {
InputStream body = clientResponse.getBody();
// put the body input stream in a request attribute so filters can read it.
MvcUtils.putAttribute(request.getServerRequest(), MvcUtils.CLIENT_RESPONSE_INPUT_STREAM_ATTR, body);
}
catch (FileNotFoundException e) {
// if using SimpleClientHttpRequestFactory
return ServerResponse.notFound().build();
}
catch (IOException e) {
return ServerResponse.status(clientResponse.getStatusCode())
.body(new String(clientResponse.getBody().readAllBytes(), StandardCharsets.UTF_8));
}
This returns the correct status code and error body for the downstream backend.
However, I'm not sure it's a good fix because I couldn't use an after-filter, the headers become read-only.
P.s: I used spring-cloud-starter-gateway-mvc, the initial changes for this bug were on spring-cloud-server-gateway-mvc.
Is there a difference or should it be changed on both?
I don't think we can arbitrarily catch IOException here. Other filters, such as circuitbreaker and retry rely on exceptions to function correctly. I'm worried that this is a rabbit hole I don't want to go down and URLConnection is not a very good http client for a proxy. Is there a reason you are not using the jdk http client?
This is the default config, I haven't explicity used that. How can I use the jdk one?
@Mendistern it is not the default. The jdk http client is the default unless you set spring.cloud.gateway.mvc.http-client.type=autodetect
@spencergibb Some context: I've an Angular frontend, Spring Backend, Oauth, and the gateway. The gateway handles the Oauth flow and routes requests to the frontend or backend.
I've removed the autodetect line. But without this, it doesn't forward requests and the browser stays on loading state.
Even though the filter was found:
Predicate "/**" matches against "HTTP GET /my-deposits/new-modification-deposit"
2024-10-30T10:18:22.999+01:00 TRACE 17312 --- [edepot-api-gateway] [nio-7081-exec-6] o.s.w.s.f.support.RouterFunctionMapping : Mapped to org.springframework.web.servlet.function.HandlerFilterFunction$$Lambda/0x000002331da8ce48@7f60d7ac
2024-10-30T10:18:22.999+01:00 DEBUG 17312 --- [edepot-api-gateway] [nio-7081-exec-6] o.j.s.OpenEntityManagerInViewInterceptor : Opening JPA EntityManager in
I tried setting http-client.type=jdk, but this also doesn't help.
I tried manually setting an HttpClient bean:
@Bean
public HttpClient httpClient() {
return HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_2)
.build();
}
Also doesn't work.
Here's my application.properties:
spring.application.name=api-gateway
server.port=${SERVER_PORT}
#### FILTERS
#BFF
spring.cloud.gateway.mvc.routes[0].id=api
spring.cloud.gateway.mvc.routes[0].uri=${backend-uri}
spring.cloud.gateway.mvc.routes[0].predicates[0]=Path=/api/**
spring.cloud.gateway.mvc.routes[0].filters[0].name=DedupeResponseHeader
spring.cloud.gateway.mvc.routes[0].filters[0].args[name]=Access-Control-Allow-Credentials Access-Control-Allow-Origin
spring.cloud.gateway.mvc.routes[0].filters[1].name=TokenRelay
#spring.cloud.gateway.mvc.routes[0].filters[2]=AddResponseHeader=Content-Type, application/json
#Back-channel logout
spring.cloud.gateway.mvc.routes[1].id=auth-route
spring.cloud.gateway.mvc.routes[1].uri=${scheme}://${hostname}:${bff-port}
spring.cloud.gateway.mvc.routes[1].predicates[0]=Path=/bff/**
#spring.cloud.gateway.mvc.routes[1].filters[0]=StripPrefix=1
#Frontend routes
spring.cloud.gateway.mvc.routes[2].id=frontend
spring.cloud.gateway.mvc.routes[2].uri=${frontend-uri}
spring.cloud.gateway.mvc.routes[2].predicates=Path=/**
# Forwarding support
#spring.cloud.gateway.mvc.http-client.type=jdk
#spring.cloud.gateway.mvc.http-client.connect-timeout=60s
#spring.cloud.gateway.mvc.http-client.read-timeout=60s
#spring.cloud.mvc.discovery.enabled=true
# Spring JDBC Session
spring.session.store-type=jdbc
server.servlet.session.timeout=600
spring.session.jdbc.initialize-schema=never
spring.session.jdbc.table-name=AUTH_SESSION
spring.datasource.url=jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?currentSchema=${DB_SCHEMA}
spring.datasource.username=${DB_USER}
spring.datasource.password=${DB_PASSWORD}
# Flyway
spring.flyway.enabled=true
spring.flyway.locations=classpath:db/migration
spring.flyway.table=flyway_spring_session
spring.flyway.baselineOnMigrate=true
spring.flyway.baselineVersion=0
#spring.cloud.gateway.httpserver.wiretap=true
#spring.cloud.gateway.httpclient.wiretap=true
Other than that I haven't customized anything besides the Oauth
I did manage to make it work with the apache http 5 module. Is this compatible with the gateway?
After discussion with the team, I've reverted the original change for the FileNotFoundException 7a41f6ad1bf3a3e01e6d86aeca905803326c84f4 and won't be adding any other workarounds for SimpleClientHttpRequestFactory. We can use this issue to document that SimpleClientHttpRequestFactory (and URLConnection which it uses) is not suitable for the WebMVC gateway server.
Okay, thank you very much for clarifying this matter.
I misunderstood the actual purpose and impact of spring.cloud.gateway.mvc.http-client.type=autodetect.
I think with https://github.com/spring-cloud/spring-cloud-gateway/issues/3571 this gets much more intuitive :+1: