The retry mechanism for internal/data/v1 calls is not functioning.

[cbabey]

Description

Hi Team,

The retry mechanism is not functioning for the internal/data/v1 calls. After the very first attempt fails, it is immediately marked as failed without any retries being performed.

[2024-09-25 18:04:37,254] ERROR - APIUtil Failed to retrieve /internal/data/v1/subscriptions from remote endpoint. Maximum retry count exceeded.Connect to localhost:9443 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

[2024-09-25 18:04:37,264] ERROR - SubscriptionDataLoaderImpl Error while executing the http client /subscriptions
org.apache.http.conn.HttpHostConnectException: Connect to localhost:9443 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.wso2.carbon.apimgt.impl.utils.APIUtil.executeHTTPRequestWithRetries_aroundBody8(APIUtil.java:649) ~[org.wso2.carbon.apimgt.impl_9.0.174.425.jar:?]
	at org.wso2.carbon.apimgt.impl.utils.APIUtil.executeHTTPRequestWithRetries(APIUtil.java:639) ~[org.wso2.carbon.apimgt.impl_9.0.174.425.jar:?]
	at org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataLoaderImpl.invokeService_aroundBody32(SubscriptionDataLoaderImpl.java:473) ~[org.wso2.carbon.apimgt.keymgt_9.0.174.410.jar:?]
	at org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataLoaderImpl.invokeService(SubscriptionDataLoaderImpl.java:455) ~[org.wso2.carbon.apimgt.keymgt_9.0.174.410.jar:?]
	at org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataLoaderImpl.loadAllSubscriptions_aroundBody0(SubscriptionDataLoaderImpl.java:88) [org.wso2.carbon.apimgt.keymgt_9.0.174.410.jar:?]
	at org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataLoaderImpl.loadAllSubscriptions(SubscriptionDataLoaderImpl.java:82) [org.wso2.carbon.apimgt.keymgt_9.0.174.410.jar:?]
	at org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataStoreImpl.lambda$1_aroundBody104(SubscriptionDataStoreImpl.java:332) [org.wso2.carbon.apimgt.keymgt_9.0.174.410.jar:?]
	at org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataStoreImpl.lambda$1(SubscriptionDataStoreImpl.java:330) [org.wso2.carbon.apimgt.keymgt_9.0.174.410.jar:?]
	at org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataStoreImpl$PopulateTask.run_aroundBody0(SubscriptionDataStoreImpl.java:830) [org.wso2.carbon.apimgt.keymgt_9.0.174.410.jar:?]
	at org.wso2.carbon.apimgt.keymgt.model.impl.SubscriptionDataStoreImpl$PopulateTask.run(SubscriptionDataStoreImpl.java:828) [org.wso2.carbon.apimgt.keymgt_9.0.174.410.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
	at java.net.Socket.connect(Socket.java:609) ~[?:?]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) ~[httpclient_4.5.13.wso2v1.jar:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient_4.5.13.wso2v1.jar:?]
	... 26 more

Steps to Reproduce

1. Configure the APIM server then create multiple tenants and deploy APIs on those tenants
2. Enable the Eger loading for all the tenants [The initial internal calls for tenant loading will fail.]
3. Start the APIM server

Affected Component

APIM

Version

4.0.0.295

Environment Details (with versions)

No response

Relevant Log Output

No response

Related Issues

No response

Suggested Labels

No response