Apache HTTP Client: Connection pool shut down

Open CoKueb opened this issue 8 months ago • 0 comments

Describe the Bug

In the last two weeks it occurred twice, that all calls to a saml-bearer destination were stopped by the circuit breaker. Looking into the cause, that opened the circuit breakers I was able to observe the following:

It seems, that the pooled apache http client that is used by cloud-sdk/xsuaa integration to retrieve access token during saml bearer flow shuts down at some point. Once it is shutdown it will not recover. Therefore the circuit breakers will open shortly after. The only way to recover this situation is to restart the application.

For more information please take a look at the attached stacktrace.

BR, Corvin

Steps to Reproduce

Run Spring Boot App with CloudSDK
Retrieve Saml Bearer Destination
Issue occures (after some time, no clue what causes the connection pool to shutdown)

Expected Behavior

Connection Pool does not shut down.
Connection Pool can shut down, but is recovered if the logic notices it has been shut down. Also it should add some information in the log why the pool was shut down.

Used Versions

Java v21 Spring-Boot v3.4.4 CloudSDK v5.17

Stack Trace

Caused by: com.sap.cloud.sdk.cloudplatform.resilience.ResilienceRuntimeException: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Header provider 'OAuth2HeaderProvider' threw an exception: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$3(Resilience4jDecorationStrategy.java:191)
	at io.vavr.control.Try.onFailure(Try.java:658)
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$4(Resilience4jDecorationStrategy.java:190)
	at io.vavr.control.Try.of(Try.java:74)
	at io.vavr.control.Try.ofCallable(Try.java:104)
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateSupplier$1(Resilience4jDecorationStrategy.java:156)
	at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorationStrategy.executeSupplier(ResilienceDecorationStrategy.java:99)
	at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorator.executeSupplier(ResilienceDecorator.java:197)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.resilientCall(DestinationService.java:367)
	... 240 more
Caused by: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Header provider 'OAuth2HeaderProvider' threw an exception: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
	at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:242)
	at com.sap.cloud.sdk.cloudplatform.thread.DefaultThreadContextExecutorService.lambda$decorate$0(DefaultThreadContextExecutorService.java:68)
	at com.sap.cloud.sdk.cloudplatform.security.SecurityContextThreadContextDecorator.lambda$decorateCallable$0(SecurityContextThreadContextDecorator.java:60)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	... 1 more
Caused by: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Header provider 'OAuth2HeaderProvider' threw an exception: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
	at com.sap.cloud.sdk.cloudplatform.connectivity.DefaultHttpDestination.getHeadersFromHeaderProviders(DefaultHttpDestination.java:203)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DefaultHttpDestination.getHeaders(DefaultHttpDestination.java:172)
	at com.sap.cloud.sdk.cloudplatform.connectivity.HttpClientWrapper.wrapRequest(HttpClientWrapper.java:121)
	at com.sap.cloud.sdk.cloudplatform.connectivity.HttpClientWrapper.execute(HttpClientWrapper.java:139)
	at com.sap.cloud.sdk.cloudplatform.connectivity.HttpClientWrapper.execute(HttpClientWrapper.java:33)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationServiceAdapter.getConfigurationAsJson(DestinationServiceAdapter.java:134)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.retrieveDestination(DestinationService.java:148)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.lambda$loadAndParseDestination$0(DestinationService.java:133)
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$2(Resilience4jDecorationStrategy.java:175)
	at io.github.resilience4j.bulkhead.Bulkhead.lambda$decorateCallable$5(Bulkhead.java:173)
	at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.call(ThreadContextExecutor.java:293)
	at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:236)
	... 6 more
Caused by: com.sap.cloud.sdk.cloudplatform.resilience.ResilienceRuntimeException: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$3(Resilience4jDecorationStrategy.java:191)
	at io.vavr.control.Try.onFailure(Try.java:658)
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$4(Resilience4jDecorationStrategy.java:190)
	at io.vavr.control.Try.of(Try.java:74)
	at io.vavr.control.Try.ofCallable(Try.java:104)
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateSupplier$1(Resilience4jDecorationStrategy.java:156)
	at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorationStrategy.executeSupplier(ResilienceDecorationStrategy.java:99)
	at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorator.executeSupplier(ResilienceDecorator.java:197)
	at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.retrieveAccessToken(OAuth2Service.java:136)
	at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2HeaderProvider.getHeaders(OAuth2HeaderProvider.java:30)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DefaultHttpDestination.getHeadersFromHeaderProviders(DefaultHttpDestination.java:197)
	... 17 more
Caused by: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
	at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:242)
	... 6 more
Caused by: com.sap.cloud.sdk.cloudplatform.security.exception.TokenRequestFailedException: Failed to resolve access token.
	at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.lambda$executeUserExchangeFlow$2(OAuth2Service.java:299)
	at io.vavr.control.Try.getOrElseThrow(Try.java:747)
	at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.executeUserExchangeFlow(OAuth2Service.java:299)
	at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.lambda$retrieveAccessToken$0(OAuth2Service.java:144)
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$2(Resilience4jDecorationStrategy.java:175)
	at io.github.resilience4j.bulkhead.Bulkhead.lambda$decorateCallable$5(Bulkhead.java:173)
	at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.call(ThreadContextExecutor.java:293)
	at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:236)
	... 6 more
Caused by: java.lang.IllegalStateException: Connection pool shut down
	at org.apache.http.util.Asserts.check(Asserts.java:34)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
	at com.sap.cloud.security.xsuaa.client.DefaultOAuth2TokenService.executeRequest(DefaultOAuth2TokenService.java:83)
	at com.sap.cloud.security.xsuaa.client.DefaultOAuth2TokenService.requestAccessToken(DefaultOAuth2TokenService.java:63)
	at com.sap.cloud.security.xsuaa.client.AbstractOAuth2TokenService.getAndCacheToken(AbstractOAuth2TokenService.java:289)
	at com.sap.cloud.security.xsuaa.client.AbstractOAuth2TokenService.getOrRequestAccessToken(AbstractOAuth2TokenService.java:248)
	at com.sap.cloud.security.xsuaa.client.AbstractOAuth2TokenService.retrieveAccessTokenViaJwtBearerTokenGrant(AbstractOAuth2TokenService.java:205)
	at com.sap.cloud.sdk.cloudplatform.connectivity.OAuth2Service.lambda$executeUserExchangeFlow$5b6622b9$1(OAuth2Service.java:279)
	at io.vavr.control.Try.of(Try.java:74)
	... 12 more

Affected Development Phase

Production

Impact

Downtime for internal customers

Timeline

ASAP

May 19 '25 07:05 CoKueb